The generative part of language models can make for really fun "single-player" games where you're really competing with the inventiveness of the language model, so there's some sense that you're playing a game with infinite hidden complexity.
Thanks for sharing Simon! I will note that by training an adapter layer between this autoencoder's embedding space and OpenAI's, it's possible to recover a significant amount of detail from text-embedding-ada-002's embeddings with this model too[0]. But as the paper author's reply in a different thread points out, their iterative refinement approach is able to recover much more detail in their research with a smaller model.
This is definitely an interesting idea I've also pondered before. In my experience (just speaking from intuition) what's "easy" for LMs to predict often doesn't line up with our human expectations for what's "obvious". Often LLMs will learn seemingly "low information content" statistical correlations that just helps it lower its training loss.
For the many Obsidian users here, wanted to share an Obsidian demo/plugin that I saw recently by Justin Smith[0] that I think faithfully carries over a lot of what I liked about this idea into the Obsidian land, complete with a semantic index w/ language models.
If you're an Obsidian user, do check out the demo. I can't take credit for any part of building it, but it's really cool to see the idea in action :)
Hey HN! Fun to see this project from my college years on HN again. To clear my name, I have actually written other languages that are arguably more useful, like Oak [0] which I used to build a bunch of personal software infrastructure [1] including my current daily-driver apps for notes, longform writing, a Twitter client, a personal search engine, and a few other knickknacks.
I find building little toy interpreters and programming languages fantastically fulfilling, creatively energizing, and a really fun way to learn how such a common element of our work that's often a black box functions inside. I've written before about how to get started [2] with these kinds of little toy programming language projects if you're curious to dive in.
I wonder if a hacky solution may be to have some kind of intermediate model to serialize the text (whether from an image of it or the raw PDF data) into LaTeX? I imagine the LM has seen enough formulas in TeX to understand it, but in most PDFs formulas are just jumbles of letters.
That's interesting! I think GPS falls in the category of "works so reliably that I don't perceive it as technology" for me. But yes, GPS an underrated piece of infrastructure, even more impressive as it's globally available (edit: as in, US tech infra made available beyond US).
Thanks for the repo/issues link! Didn't know that resources existed.
Looks like the immediate issues got taken care of thanks to the HF team, but I'll probably pop over there in the future if I have ideas or notice things that can be improved.
The code generated is most definitely related to the names/descriptions! To do this, I have to first generate the description then generate the code _from the descriptions_. The downside of this is that I can't parallelize text generation, but the upside is the code feels much more realistic. Here's the prompt I used (from that same file):
The idea here was to give the model a prompt that felt like a tutorial or some kind, and try to minimize non-Python non-ML-y code.
---
$MODEL_DESCRIPTION_FROM_EARLIER
Let\'s use this model. The basic use case takes only a few lines of Python to run the inference. Here are the first few lines.
Sorry about that. I don't think I'm leaning on any super new browser/JS features, but if you share your FF version string (or an error in the console) I can try to troubleshoot what's missing!
Proceedings of Deep Learning Advancements Conference, list of accepted deep learning models
1. [StyleGAN] StyleGAN is a generative adversarial network for style transfer between artworks. It uses a traditional GAN architecture and is trained on a dataset of 150,000 traditional and modern art. StyleGAN shows improved style transfer performance while reducing computational complexity.
2. [GPT-2] GPT-2 is a decoder-only transformer model trained on WebText, OpenAI\'s proprietary clean text corpus based on Wikipedia, Google News, Reddit, and others comprising a 2TB dataset for autoregressive training. GPT-2 demonstrates state-of-the-art performance on several language modeling and conversational tasks.
Yep, that's spot on. The overall performance is comparable to Curie, but depending on the particular task GPT-J performs better or worse (I believe empirically it's slightly better at chat and code, worse at some others).
Yep. I didn't want to have to host user-generated data (for all the perils that carries), so the sharable links work by embedding all the generated data in the link itself.
There's a pre-generated set to (1) spare my server some work and (2) showcase some output I liked. But as sibling comment noted, you can (or could) generate your own — I'm working on bringing that side back up...
The pre-generated set is hand-curated, but they are still 100% generated by the GPT-J model behind the scenes. More info -> https://github.com/thesephist/modelexicon
This point should get more visibility — embeddings are not made in the abstract; they reflect the lexicon of their training sets, and I strongly suspect word embeddings used here reflect the lexicon and word frequencies (and their meaning/usage context) in modern literature. Using them for text in the past (nearly a century back!) warrants some skepticism.
Interesting perspective! It had actually not occurred to me to take a more general non-anthropocentric perspective and I'm grateful to know there's non-human sides to semiotics/notation to think about. TIL :)
I mention this in passing in the post, but I think the various notations that exist to express finite state machines help us greatly to understand complex software systems.
One way to express FSMs is the general node-and-arrow diagram. That makes sense for simple systems like a traffic light ("red" can only go to "green", etc. etc.) But for more complex state machines, we might want to express it in a regular expression, which is mathematically equivalent and lends itself to a different kind of mental model that might be better for, say, writing a search engine. Another "notation" to represent FSMs is matrices that model state transitions (if you can go from state 1 to state 2, X[1][2] = 1, otherwise it's 0, etc.) This is useful for certain other kinds of questions like "how many steps to go from state A to state B?.
One practical use case of state machines is in parsing theory. If you're trying to parse JSON correctly to the spec, you might care about the formal grammar of JSON, which is often written in a specialized notation called BNF that describes what symbols can come after another. This is useful for implementing a parser, but to understand the grammar itself I personally find railroad diagrams [0] more intuitive. Different notations for the same idea let us be smarter in different use cases.
On a completely different track of ideas: you might be interested in the work from Dion Systems [1] which are doing some interesting research into more dynamic ways to work with source code that I think reflects a lot of the ideas I wrote about. It touches on your specific comment more directly.
The generative part of language models can make for really fun "single-player" games where you're really competing with the inventiveness of the language model, so there's some sense that you're playing a game with infinite hidden complexity.