To add a bit for readers who may be unaware, https://www.continue.dev/ uses embeddings as a strategy for indexing codebases. When used together with keyword search, I find that it performs quite well, especially when used together with the reranking functionality.
I don't understand why the editors allowed the engineers' names to be made public. What did they hope to gain by doing this other than making them magnets for harassment and possibly threats?
Although voyage-3-m-exp is at the top of the leaderboard, I would not use it for production use cases unless your data is extremely similar to "in-domain" MTEB data. Check out this post from Nils (one of the authors of the original MTEB paper) for more info: https://x.com/Nils_Reimers/status/1870812625505849849
voyage-3-large will work better for almost all real-world production use cases.
> As with all LLM models and their subproducts, the only way to ensure good results is to test yourself, ideally with less subjective, real-world feedback metrics.
This is excellent advice. Sadly, very few people/organizations implement their own evaluation suites.
It doesn't make much sense to put data infrastructure in production without first evaluating its performance (IOPS, uptime, scalability, etc.) on internal workloads; it is no different for embedding models or models in general for that matter.
While the raw similarity score does matter, what typically matters more is the score relative to other documents. In the case of the examples in the notebook, those values were the highest in relative terms.
I can see why this may be unclear/confusing -- we will correct it. Thank you for the feedback!
Because LLMs such as Gemini -- and other causal language models more broadly -- are trained on next token prediction, the vectors that you get from pooling the output token embeddings aren't that useful for RAG or semantic search compared to what you get from actual embedding models.
One distinction to make here is that token embeddings and the embeddings/vectors that are output from embedding models are related but separate concepts. There are numerous token embeddings (one per token) which become contextualized as they propagate through the transformer, while there is a single vector/embedding that is output by embedding models (one per input data, such as long text, photo, or document screenshot).
No worries at all. That's great feedback and an area of improvement for us when it comes to future posts -- we'll be more explicit about multilingualism in blogs and in our docs.
We provide retrieval metrics for a variety of datasets and languages: https://blog.voyageai.com/2024/09/18/voyage-3/. I also personally encourage folks to either test on their own data or to find an open source dataset that closely resembles the documents they are trying to search (we provide a ton of free tokens for the evaluating our models).
One quick minor note is that the resulting embeddings for the same text string could be different, depending on what you specify the input type as for retrieval tasks (i.e. query or document) -- check out the `input_type` parameter here: https://docs.voyageai.com/reference/embeddings-api.
Many of the top-performing models that you see on the MTEB retrieval for English and Chinese tend to overfit to the benchmark nowadays. voyage-3 and voyage-3-lite are also pretty small in size compared to a lot of the 7B models that take the top spots, and we don't want to hurt performance on other real-world tasks just to do well on MTEB.
Agreed. ViTs are better if you're looking to go multimodal or use attention-specific mechanisms such as cross-attention. If not, there's evidence out there that ViTs are not better than convnets for small networks and at scale (https://frankzliu.com/blog/vision-transformers-are-overrated).
A lot of folks I've spoken with say that single-agent systems are still extremely limited, let alone multi-agent platforms. In general, it seems to boil down to:
- Agents need lots of manual tuning and guardrails to make them useful
- Agents with too many guardrails are not general-purpose enough to be worth the time and effort to build
I believe truly great agents will only come from models whose weights are dynamically updated. I hope I'm wrong.
> I think a few details (and perhaps a small amount of customization) would go a long way.
I hear you and agree 100% - I unfortunately haven't gotten around to writing better documentation nor solid code samples that utilize Radient yet.
Regarding molecule vectorization: that capability comes from RDKit (https://rdkit.org) - I just uploaded a sample to the /examples directory. You're right that molecule-to-audio and audio-to-molecule search is nonsensical from a semantic perspective, but I could see a third modality such as text or images that ties the two together, similar to what ImageBind is doing (https://arxiv.org/abs/2305.05665).
> You're double-dipping into the data. You look at the performance, then you tune some part of the network by hand, see if it helps, and then keep doing that. It's testing on the training data.
I purposely tried to avoid adding any niche network modifications that would help it overfit to in-1k. All three of the modifications are applicable to other networks and datasets.
> No one cares about ImageNet-1k. No one needs to classify ImageNet-1k in real life.
I completely agree with you; I just don't have the compute to train this on a massive dataset. With that being said, I'm not advocating for taking an in-1k model and putting it into production. I'm merely saying we can get ResNet to reach the same level of performance as ViTs. And that there's evidence that convnets reach that same level of performance at scale.
> What you want is a ViT that's seen massive amounts of data so that your embeddings don't become degenerate because they're far away from ImageNet-1k!
"Although the success of ViTs in computer vision is extremely impressive, in our view there is no strong evidence to suggest that pre-trained ViTs outperform pre-trained ConvNets when evaluated fairly."
Neural networks can be unpredictable, and there's evidence that questions how important transformers' lack of inductive bias (at scale) really is.
To add a bit for readers who may be unaware, https://www.continue.dev/ uses embeddings as a strategy for indexing codebases. When used together with keyword search, I find that it performs quite well, especially when used together with the reranking functionality.