I’m currently working on a document parsing engine for a specific type of document. The inputs are usually PDFs. I’m able to get great structured output from both the latest Gemini Flash models and the latest Llama Scout models. The best latency I get with Gemini is about 5 seconds end to end. With llama hosted on groq it’s about 3 seconds.
My use case is latency constrained, so I’m exploring fine tuning / distilling to see if I can get latency sub second. I imagine these are the kinds of scenarios where it’s still worth it to fine-tune and distill.
My plan is to generate a lot of synthetic training data using more capable slower foundation models and use that to train the smaller model.
It’s really useful for generating synthetic data for search and recommendations that you can use to train a smaller / faster model. This is especially useful if you don’t have lots of click-through data or with cold start scenarios. There are some good articles that cover this, if you’re interested I’ll try to find them and share
- Conventional SGD: Fixed target (e.g. "make an exact replica of this butterfly image") and it follows greedy path to minimize the error
- Open Ended Search Process: No predetermined goal, explores based on what's "interesting" or novel. In Picbreeder, humans would see several generated images, pick the "interesting" ones, and the system would mutate/evolve from there. If you were evolving an image that looked like an egg and it mutated toward a teapot like shape, you could pivot and pursue that direction instead.
This is kinda the catch -- there is a human element here where individuals are choosing what's "interesting" to explore, it's not a pure algorithmic process. That said, yes, it does use a genetic algorithm (NEAT) under the hood, but I think what the authors are suggesting is that the key difference isn't whether it's genetic or gradient based optimization... they're getting at the difference in objective driven vs. open-ended search.
I think the main position / takeaway from the paper is that something about conventional SGD training produces these "fractured entangled representations" that work but are not well structured internally so they're hard to build on top of. They look at things like the curriculum / order things are learned in, objective search vs. open-ended search, etc...
Could a major opportunity to improve representation in deep learning be hiding in plain sight? Check out our new position paper: Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis. The idea stems from a little-known observation about networks trained to output a single image: when they are discovered through an unconventional open-ended search process, their representations are incredibly elegant and exhibit astonishing modular decomposition. In contrast, when SGD (successfully) learns to output the same image its underlying representation is fractured, entangled - an absolute mess!
This stark difference in the underlying representation of the same "good" output behavior carries deep lessons for deep learning. It shows you cannot judge a book by its cover - an LLM with all the right responses could similarly be a mess under the hood. But also, surprisingly, it shows us that it doesn't have to be this way! Without the unique examples in this paper that were discovered through open-ended search, we might assume neural representation has to be a mess. These results show that is clearly untrue. We can now imagine something better because we can actually see it is possible.
We give several reasons why this matters: generalization, creativity, and learning are all potentially impacted. The paper shows examples to back up these concerns, but in brief, there is a key insight: Representation is not only important for what you're able to do now, but for where you can go from there. The ability to imagine something new (and where your next step in weight space can bring you) depends entirely upon how you represent the world. Generalization, creativity, and learning itself depend upon this critical relationship. Notice the difference in appearance between the nearby images to the skull in weight space shown in the top-left and top-right image strips of the attached graphic. The difference in semantics is stark.
The insight that representation could be better opens up a lot of new paths and opportunities for investigation. It raises new urgency to understand the representation underlying foundation models and LLMs while exposing all kinds of novel avenues for potentially improving them, from making learning processes more open-ended to manipulating architectures and algorithms.
Don't mistake this paper as providing comfort for AI pessimists. By exposing a novel set of stark and explicit differences between conventional learning and something different, it can act as an accelerator of progress as opposed to a tool of pessimism. At the least, the discussion it provokes should be quite illuminating.
For anyone interested in NEAT, you’ll likely enjoy Ken’s later work on novelty search, open-endedness, etc.
His book “Why Greatness Cannot be Planned: The Myth of the Objectives” is one of the most perspective altering works I have ever read.
Very excited to see what he does next. He mentioned on Twitter a couple times his interest in representation learning and how objective based search affects this. Very interesting stuff
One downside of rolling it into an IRA is that you'll pay more in taxes to do backdoor roth IRA conversions every year. I think it's better to leave it in 401k if the fund options / fees are acceptable.
@Jeremy, have you ever encountered Ken Stanley’s book “Why Greatness Cannot be Planned: The Myth of the Objective”?
If you’re not familiar with him, he was the guy that invented the NEAT algorithm, Novelty Search, etc.
In his book he talks about stepping stones and following what’s “interesting” to individuals. It seems like Answer.AI is focused on applied engineering, but any thoughts on this type of approach from a Research perspective?
Why Greatness Cannot Be Planned: The Myth of the Objective by Ken Stanley and Joel Lehman. This book was a fascinating read for anyone with ambitious objectives (or an interest in optimization algorithms). Ken is such a deep thinker, I love when he's on podcasts or gets interviewed, and reading his book was a real treat.
Allegis Global Solutions is the biggest recruiting company in North America. I'm on an internal team that builds software for our 10,000+ recruiters. We help the recruiting teams manage their clients / open job requisitions, find and attract talent, and manage their recruiting teams.
We’re looking for a data engineer to join our team and help us mature our analytical capabilities by modelling our most important business processes and creating useful data models in our data warehouse.
You’ll be able to make a big impact almost immediately with some quick wins and you will have the opportunity to own solutions end-to-end with a lot of autonomy. We’ll support your growth and actively look to help you develop your technical skills, ensure you’re doing work you enjoy, and make sure you’re advancing your career.
We recently completed setting up our overall data infrastructure / stack for analytics, and now is a fantastic time to join the company in this role. We leverage Fivetran to ingest data from various data sources into Redshift which functions as our data lake. We then transform the loaded data using DBT and materialize our data models back into our Redshift cluster to create our data warehouse.
We are at the very beginning of a super exciting stage of our team’s journey, we expect to grow rapidly and for you to grow with us, and we can’t wait to talk to you about the job in more detail.
This is what I do, it has worked great for me. The only annoying thing is I can’t get tab completing to work in my shell (ZSH). Something about the alias I think.
I am so grateful the FastAI team exists. It wasn't until I discovered their "Machine Learning for Coders" course that I really started to grok ML. I was in grad school trying to pivot my career from finance to data science. I didn't come from a computer science / math background and things just weren't clicking for me. I remember feeling angry, embarrassed, dumb, and overall that I wasn't smart enough to learn this stuff -- I was incredibly discouraged and felt that I didn't belong there. I was lucky enough to stumble across one of the course videos on YouTube (thanks recommendation algorithm!), and the rest is history.
The amazing thing about these courses is how simple Jeremy (and team) are able to make machine learning. I didn't need to understand python dependency management in order to learn how to train an really good image classifier. Their approach helped me have lots of little wins, gave me confidence, and helped build the motivation to slog through the harder stuff when I needed to.
From the bottom of my heart, thank you @jph00. You changed my life immeasurably for the better. I learned that I AM good enough, I AM smart enough, and I CAN do hard things... I just had to find the right way to learn them. Your courses completely changed my perspective on what was possible for me and opened the door to some of my life's greatest passions.
I work as a machine learning engineer on search & recommender systems, mostly focused on relevancy ranking in the recruiting space. It's unbelievable the pressure I have encountered to give "diverse" candidates a boost across the board in our scoring algorithms. Almost every client's talent acquisition team has asked what we can do to help them meet their diversity goals (which they loosely define as hiring more women and non-white / Asian people). I keep up with what some of the major players are doing (LinkedIn, Indeed, etc.), and they come straight out and say they preferentially weight diverse candidates
> For a given search or recommendation task, our algorithms seek to
achieve a desired distribution of top ranked results with respect to
one or more protected attributes.
“Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis“
https://arxiv.org/abs/2505.11581