It helps to comprehend research papers (and not only papers - any document on any language) faster.
The tool is free to use, because we have credits from GCP. I guess at some point we'll need to introduce some level of subscription fee to keep it alive and useful, as it uses LLMs and vector search quite a bit.
Leo Boytsov, my guest on the Vector Podcast, made an honest and an eye-opening claim about vector search being intellectually rewarding, but professionally undervalued. What picked my attention was how he gives credit to people who actually deserve it, and how he speaks modestly about his own achievements. When professionally as a researcher, he accumulated over 1800 citations by now and helped to create the famous HNSW vector search algorithm.
I'm working on the tool, that includes AI. My original target is to test it on my https://www.youtube.com/c/VectorPodcast by offering something that Lex Fridman does for his episodes.
Current features:
1. Download from YT
2. Transcribe using Vosk (output has time codes included)
3. Speaker diarization using pyannote - this isn't perfect and needs a bit more ironing out.
What needs to be done:
4. Store the transcription in a search engine (can include vectors)
5. Implement a webapp
If anyone here is interested to join forces, let me know.
I write about vector search, ANN algorithms, neural search frameworks, search engines and algorithms in general and publish episodes of the Vector Podcast.
It was great to discuss with Jo Kristian on these topics and more:
- History of Vespa
- Tensor data structure and its use cases
- Multi-stage ranking pipeline
- Game-changing vector search in Vespa
- Approximate vs exact nearest neighbor search tradeoffs
- Misconceptions in neural search
- Multimodal search is where vector search shines
- Power of building fully-fledged demos
- How to combine vector search with sparse search: Reciprocal Rank Fusion
- The question of WHY (my favourite)
I wonder what topics are interesting to HN community -- it would help me focus on these topics / embed into my questions in new episodes.
Thanks for the article, I've learnt new search engines despite spending a couple of years recently in web scale search. I think you may consider https://usearch.com/ as another dimension in web scale search, where query log is learnt from the data, making it quite unique.
Great project! Elasticsearch / OpenSearch / Solr have their own learning to rank plugins. Have you considered integrating Metarank with such systems? Or is your vision to provide a reranker layer, that can be independent of the underlying search engine architecture?
This is fantastic work, Max! I like where you're going with this unit economy.
It is increasingly more and more common that companies need ways of productizing their deep learning models with tight SLA constraints and tight budgets.
I think, one of the logical steps could be integration with vector databases / frameworks and measuring gains there in client setups.
Thanks! One practical target I imagine having is:
1. write the program in Python.
2. translate it into Ć ("interlingua")
3. compile Ć into Go -- for instance there is a belief around me, that Python is not suitable for web scale loads, while Go is more suitable.
Go source code / binary in this case are of less importance for code readability, because they are meant for production deployments. Something that happened in GWT: write in Java, compile into JavaScript.
Thanks for sharing! The practical value of such languages could be to build an interlingua and support Python->Ć->Go.
The question is: if Ć->Python works seamlessly, how easy it is to make the opposite translation? For instance, take some code that uses external library, like spacy.
If this becomes ever possible, it is going to be revolutionary.
Vector Podcast is here to bring you the depth and breadth of Search Engine Technology, Product, Marketing, Business. In the podcast we talk with engineers, entrepreneurs, thinkers and tinkerers, who put their soul into search.
Depending on your interest, you should find a matching topic for you -- whether it is deep algorithmic aspect of search engines and information retrieval field, or examples of products offering deep tech to its users.
"Vector" -- because it aims to cover an emerging field of vector similarity search, giving you the ability to search content beyond text: audio, video, images and more.
"Vector" also because it is all about vector in your profession, product, marketing and business.
It helps to comprehend research papers (and not only papers - any document on any language) faster.
The tool is free to use, because we have credits from GCP. I guess at some point we'll need to introduce some level of subscription fee to keep it alive and useful, as it uses LLMs and vector search quite a bit.
Feedback is welcome!