On the discovery progress, we can't look at all 200M papers with the LLM, so we prioritize some of them (the first 100 most promising) for deep analysis. Within those, we find a few that are relevant. But the rate at which we discover these tells us roughly what will happen if we read the next 100 (if we're discovering new relevant papers all the time, we will likely continue). We need a better explanation on the website, but we can statistically model this to quantitatively predict how many papers we would find if we exhaustively searched the whole database.
While the time/cost of using GPT-4 is not ideal, GPT-4-level classification is absolutely crucial for the entire adaptation process to succeed. With 3.5 guiding the adaptation, we find that errors quickly accumulate. It can't identify complex ideas correctly.
3-6 minutes for results takes getting used to, but we've found most people don't complain if it solves a problem that is actually impossible to solve without hours of digging, ie if you use it on something truly hard. Low latency is more crucial for public search engines like Google (0.5s delay -> 20% traffic loss) where there are convenient, fast alternatives.
Preliminary search is a blend of semantic embeddings on 100M+ papers and keyword search, citation links, etc. Reasonably accurate, but full of noise for complex queries.
Appreciate the advice. Re: timing, it's bottlenecked by the sequential nature of the search. To be comprehensive, we discover a few papers, and use that info to choose where to look more closely next.
Here's an example report on: tokenization-free large language model architectures, which have been shown to achieve compute/accuracy tradeoffs comparable to or better than traditional token-based models
https://app.undermind.ai/query_app/display_one_search/05f0b8...
Why don't we track scientific authorship trends like financial markets?
We should track how experienced scientists switch fields, since this is probably a strong signal for important science.
I tracked author movement between fields from all ArXiv papers over the last two decades. The results reveal some obvious trends (ie. the growth of AI), and can more generally provide valuable context for career choices.
More info here: https://www.undermind.ai/static/Undermind_whitepaper.pdf