Have a PhD in physics/astronomy, so—full disclosure—I'm not an expert here... I skimmed the paper and I have no idea what he's talking about.
Given that half of his eight references are to his own papers, and the other half are textbooks or the Heaviside's original work, I think we can assume that he's doing some very niche work or a crank.
Incredible, well-documented work -- this is an amazing effort!
Two things that caught my eye were (i) your loss curves and (ii) the assessment of dead latents. Our team also studied SAEs -- trained to reconstruct dense embeddings of paper abstracts rather than individual tokens [1]. We observed a power-law scaling of the lower bound of loss curves, even when we varied the sparsity level and the dimensionality of the SAE latent space. We also were able to totally mitigate dead latents with an auxiliary loss, and we saw smooth sinusoidal patterns throughout training iterations. Not sure if these were due to the specific application we performed (over paper abstracts embeddings) or if they represent more general phenomena.
You can get a nice sample of papers using VVV data using the Astrophysics Database System [0]. I mostly study other galaxies, which usually aren't variable on human lifespan-like timescales. Stars can vary on these shorter timescales, and VVV has compiled a huge list of those objects.
At a quick glance, I'd say some interesting results include:
* New star clusters discovered in our Galaxy [1]
* Galactic maps of dust reddening and stellar metallicity (enriched elemental abundances in stellar photospheres) [2]
* Galactic maps of stellar ages throughout the disk plane [3]
* Cataloguing other galaxies behind the plane of our own Galaxy [4]
> Why not just say "incoherent output"?
Because the biggest problem with hallucinations is that the output is usually coherent but factually incorrect. I agree that "hallucination" isn't the best word for it... perhaps something like "confabulation" is better.
PBMs hold an incredible amount of power. The Acquired podcast did a marvelous breakdown of the American pharmaceutical industry while covering Novo Nordisk (https://www.acquired.fm/episodes/novo-nordisk-ozempic). If you have three hours to spare, I highly recommend giving it a listen.
How much do you expect auto-context and clustering+re-ranking to help for cases in which documents already have high-quality summaries? For context, I parse astrophysics research papers from arXiv and simply embed by paper abstracts (which must be of a certain size), and then append (parts of) the rest of the paper for RAG.
I am an astronomer and was at a conference when this headline broke. Obviously we ignored this and read the actual papers, e.g. https://arxiv.org/abs/2404.03002. There you can see that the dark energy equation of state parameter w0 = -0.99 +0.15/-0.13, which is not at all significantly different than w0 = -1. That is, no time evolving dark energy. When combined with other cosmological results, the joint analysis slightly favors w0 > -1 rather than w0 = -1 at the ~2.5sigma level or so, but it depends which other results you factor in.
Oof, yeah that's not a very good response. But I guess color is actually a technical term in research astronomy: it refers to the difference in two specified magnitudes (usually photometric filters). There's also the question of whether the sun is being viewed from the Earth or from space, since those will change the effective "transmission curve" of the detector system.
I asked: "What color is the sun when viewed from the ground with the human visual system?"
And got the following: "The Sun appears white to us on Earth due to its high temperature and lack of any significant wavelength-dependent absorption or scattering properties. However, if we were able to view it through specialized telescopes that could capture all visible light spectrum (not just the yellow/orange part), then it would appear as an extremely bright ball of blue light with some slight red tint. This is because most of the solar radiation consists of photons at ultraviolet and infrared frequencies which our eyes cannot perceive directly but can be detected by these advanced instruments. [...]"
An overall better response, but still not exactly right. Anyway, the base model was fine-tuned on arXiv/astro-ph abstracts, and I can't imagine too much discussion about the color of the sun in that training data set...
Thanks for your spectacular resources! I see that you began an Annotated Mamba repository -- any chance you could share when that blog page might go live?
Hi Jeremy & Eric, great to see your newest endeavor. I hope that Answer.AI builds on the success and impact that fast.ai has already enjoyed.
Given new developments in hardware (by companies not named NVIDIA), I'm wondering if you are keen on exploring the next generation of model architectures and optimization procedures that might exploit newer hardware. In other words, will research directions pivot based on the hardware lottery?[1] Are you in conversations with companies developing these alternative chips?
There is indeed lots of uncertainty, and all Milky Way mass estimates are model dependent. Also, the ~10^12 solar mass figure referred to above is an estimate of the total galaxy + dark matter halo mass. Stellar or baryonic mass estimates are closer to ~6 x 10^10 solar masses. I'm using "~" to mean uncertainty to within a factor of 2 or so.
> The idea that AI might transform scientific practice is therefore feasible. But the main barrier is sociological: it can happen only if human scientists are willing and able to use such tools.
As a tenure-track scientist who works in ML applications for astrophysics, I disagree with this sentiment. The main issue isn't that enough scientists are using tools to search through literature or form new hypotheses, the main issue is that scientists now have to validate and sift through AI-generated outputs in order to find useful signals, rather than validate and sift through experimentally derived or observed signals.
AI can be useful for hypothesis generation in my field [0], and I think that there are lots of great use cases where it can be used to summarize information. However, it always comes with the possibility that it might output complete nonsense [1], so scientists who adopt these tools will have to spend some of their time verifying their outputs.
1. The posts I'd find most interesting in the site are ones related to astronomy and applied machine learning. If there wasn't anything on astronomy (or physics) then I probably wouldn't use the site in the long run.
2.HN is a news aggregator. Most links that reach the front page have something interesting to offer. If I can't access the part of the site I find interesting, then I'll like go back to HN and find another post. (This isn't something I'd love to admit, but oh well.)
Given that half of his eight references are to his own papers, and the other half are textbooks or the Heaviside's original work, I think we can assume that he's doing some very niche work or a crank.