My point with the force field example wasn’t to argue against neural scaling as a valid strategy, it totally is effective and a lot of groups are doing it. But I feel like we might be talking past each other a bit.
What I’m pushing back on is what I think is a sort of one-dimensional view of Sutton’s bitter lesson. People seem to equate it with model scaling, but there are lots of general ways to leverage computation that don’t involve just scaling models and supervised training datasets up. For example Sutton’s first example is straight up search, no parameters at all.
The point of the force field example is that it seems you don’t need billions of parameters to represent the functions we’re interested in, but with small models it’s harder to find those functions by pushing harder on the standard training algorithms, and that maybe some different algorithm that leverages computation more effectively could do so.
I don’t think there’s a fundamental reason that performance has to be monotonic in model size or even training FLOPs. At least I don’t think it’s been proved to be so, so I think “misinformed” is a bit premature and sort of makes GP’s point.
There’s evidence that model size and representational capacity are not exactly the same, and that scale is maybe more important for learning than it is for representation (past a point). Consider the early work from the current neural scaling paradigm. The Chinchilla scaling study shows that smaller models can match the performance of larger models by training longer.
To GP’s point, if everyone is exploiting the scaling lever, few resources are being allocated to finding more efficient training algorithms that could let us work with right-sized models instead of pulling the scaling lever as hard as we can afford to.
I’ll end with a dramatic example from my field of materials science (which admittedly might not strictly generalize to LLMs). A lot of the field is pursuing the model scaling strategy, and it’s still paying off. But [0] recently reported competitive accuracy with much smaller models that run faster and can address much larger problems. The model architecture is pretty much the same, but they use a different training strategy and really focus on data quality
I don’t mean to pick on you in particular here, but this approach has been bothering me a lot lately, and it seems like it’s super common.
I get that it’s an early prototype and not all the design choices are made yet, but I struggle with “I can’t afford human-readable documentation yet”. Isn’t human readable documentation important for efficiently planning and deciding what you want to build? I feel like I can’t afford not to have human readable docs and plans while the project is taking shape
Related, if a user can prompt an agent to translate to a human-readable summary, wouldn’t it be better to just do this in place? Sure, models can deal with noisy LLM outputs but shouldn’t a document that’s easier for humans also be easier for bots?
I often start prompts with “please”, but I usually don’t thank the model. Framing a question or a request for help with “please” is in distribution for me, it’s a distraction from composing a thoughtful prompt about my actual question to go back and edit out politeness.
I don’t reply “thanks” like I would to a person though, I just close the chat if I have no more follow-ups
The DHS secretary seems to me to have the point of hosting international students backwards
> This final rule ensures that foreign students remain focused on their primary purpose: completing their studies and returning home.”
Especially at the PhD level, why would educating foreign students and sending them away be the primary motivation for granting visas? Historically it’s been about attracting the best and brightest in the world, training them to be excellent researchers and scholars, and giving them a path to becoming citizens and contributing to growing our economy and technology development and all that. Sending them away after investing in training them for years makes no sense! Especially given the administrations adversarial stance on technology development as a competition between nations more than as an opportunity for collaboration.
I mentor a postdoc who might be affected by this, he’s on a J visa and it’s already been a nightmare of paperwork. He’s really good. Not disputing that there’s some amount of people gaming the student visa system, but this just doesn’t seem well thought out to me :(
I think JEPA is super interesting, but I feel like this example highlights some of the challenges of long horizon planning. For one, chunking the planning stage into a bunch of intermediate goals seems really limiting, because a lot of what makes model based control interesting is that we don’t want to impose a solution strategy (because we want to solve problems we don’t know how to solve)
Another thing that has been bothering me is that you have to write the goal in input space. That doesn’t align with all problems, for some problems there could be many different states that satisfy a goal. For Mario maybe it’s ok, but there’s some weirdness still, like should the goal state be Mario at the finish line of the level with a specific timer state in the frame header? What about optimizing the number of points?
Also it’s interesting to think about how you would get Mario to reliably jump on koopas and goombas. IIUC JEPA models are usually trained with random rollouts, and then you’d handle this sort of intermediate goal in the planning optimizer? But that seems inefficient, and including some planning in the pretraining rollouts might be necessary to get enough relevant intermediate states. And then it starts feeling like reinforcement learning…
I’d be happy to have a check on my intuition here, or pointers to interesting writing on these topics
p.s. on topic, I liked the debugging strategies used in the blog post, that was my favorite part of the writeup
What aspect do you consider basic? I haven’t had a chance to read more than the abstract because of the paywall, but the really interesting thing here is the mechanically induced transformation that leads to a 3-phase nanocrystalline alloy. I haven’t understood the “fully coherent” part from just the abstract, but I think it’s a very novel report. It means they have three different crystal structures with seamless interfaces because all the atoms line up at where the crystals meet. To do that with three structures is remarkable. I’m not sure if it’s the first example, but I’m only aware of alloys with two coherent phases (some superalloys for example)
There’s some precedent for mechanical deformation to get nanocrystalline grain structure, and some precedent for mechanically induced phase transformations (see TrIP steels) but I consider both of those concepts pretty advanced
TrIP is probably the closest thing, but I’m not sure how widely known there are among the “metal forging” community? TrIP is usually targeting well known phase transformations to two-phase microstructures, here we have three nanostructured phases.
Finally high entropy alloys are absolutely not well understood, even if the idea of mixing a lot of elements and getting a disordered solution seems simple on its face.
The science daily article is just incorrect to call this a superalloy, which it is not. This is a high entropy refractory alloy (HfNbTaTiZr), superalloys are usually based on lighter metals and they usually have only one dominant element while HEAs have 4+ dominant elements
This is really cool metallurgy. They start with an alloy and deform it and because of elemental size mismatch they can cause the alloy to self assemble into nanoscale crystals with three different structures
As an aside, “super alloy” is not the best wording choice on the part of the author of this sciencealert article, superalloys are an established alloy family that follow a different design strategy and have a very different composition profile https://en.wikipedia.org/wiki/Superalloy
Terry Tao has actually been one of the more prominent voices in the math community exploring AI for cutting edge mathematical discovery. This particular post is a bit softer but he has also written a lot about using AI assistance for serious core research
But this is true for lots of solids, not just glass. Consider creep deformation [0] which is deformation mediated by diffusion of defects in solid materials. It’s a big problem in metal turbine blades, it limits the maximum usable temperature to well below the melting point.
The physical mechanism would be similar in glass flowing in this way, so I don’t think evidence of glass flowing like this should make us think of it as a liquid instead of an amorphous solid
Lab to product scale-up is a well known hard problem in materials (and chemistry), lots of public and private investment has been aimed at accelerating this for decades
I think the distinction between discovering a material and processing / scaling it up is a bit artificial. A lot of people think of a new material as just the crystal structure or something, but really all the defects and complex multiscale structure is just as much part of what defines a material, and controlling all that is why materials development is hard, and why you need so many different complementary measurements to understand what’s going on
I was a bit underwhelmed by this writeup because it’s a bit generic. I didn’t really see any specific new ideas on how to accelerate this process, or to differentiate from the main stream of materials discovery research which has been pretty dominantly AI forward for at least five years now
EDIT: I checked out some of their case studies and they’re pretty interesting and exploring some new materials characterizations territory. They’d be more impactful if they were more than just text IMO but much more concrete and less generic than the linked post
Maybe I have too optimistic a mindset, but “just be honest” in academia isn’t about being a rule-follower, it’s about not short-changing yourself by coasting though on autopilot instead of learning to think and solve problems for yourself.
Whether that really matters if your goal is to climb the social ladder and have power and influence, I don’t know.
> You allowed a take-home exam which means students are able to use any and all resources.
It was a closed-book exam. The professor shouldn’t have to hold students’ hands for them to act with integrity, they are all adults.
In this particular class, the professor made the final exam in-person, and didn’t count the take-home midterm because the score distribution wasn’t consistent between the two exams. I think that’s a reasonable approach, but it’s kind of sad that it was necessary
In my experience the input field lags on short chats too, sometimes in the middle of writing the second or third prompt. Are they running some kind of prospective evaluation or something?
If the explicit role-playing prompt is just to identify multi-valent terms, then revising the question to include more specific context without a role-play prompt should work just as well right? I’d be really interested if anyone has evaluated that hypothesis
A fun (frustrating) feature of language is that we get these name collisions even with a single domain. One that I have to remember to revise myself fairly often these days when chatting with other experts in my field is “diffusion model” which can either mean generative deep learning or a differential equation describing mass transport.
That’s the active research area GP mentioned. In startup land there are a few large outfits, Lila Sciences, Period Labs, Radical AI are all doing a mix of simulations, AI, and autonomous laboratory infrastructure specifically for materials science. (Lila does a lot of biotech but the have materials researchers too)
Also lots of interest and activity in this space in the national labs and academic research scene
It depends what you mean by commercially interesting. There’s loads of interest in aerospace (for high temp corrosion resistant structural components) and catalysis but these alloys are pretty much across the board at a relatively low level of technical readiness. It’s developed enough that there’s significant industry R&D and not just academic and government research, but I don’t think there’s really wide-scale deployment yet of alloys with 4+ principal elements
My research interests include applied machine learning for quantifying materials substructure, which we call microstructure.