Because colbert is not an end-to-end solution. As seen for RAGatouille, it has integrated colbertv2 into this repo. However, it's not a database, we implement tensor within infinity, aim to make an end-to-end solution for late interaction based ranking models.
paradedb could also deliver three-way hybrid search through pg_vector, pg_sparse and pg_search. Compared with paradedb, infinity has following advantages:
1. Performance
The performance of pg_vector is far slower than vector search of Infinity due to the vector index design.
The performance of pg_sparse is also slower than sparse vector search of infinity.
The performance of pg_search is much slower than full text search of infinity. pg_search is based on Tantivy, which is much slower than the inverted index of infinity.
2. Infinity has all the builtin implementation of the above three search approaches. These indices could work smoothly together with the executor of infinity. The users could use any combination of the search approaches, together with the fused ranking algorithms, in a very efficient approach.
3. Infinity has also builtin support for tensor, which makes it possible to deliver an in-database colbert reranker compared with the cross encoder based reranker outside. The colbert reranker could bring much benefits for search qualities.
4. Infinity is much easier to use, it could be deployed as either a standalone server, or as an embedded python library just through pip install.
5. Infinity is designed start from scratch, it does not have the burden of postgresql, and is evolving fast. It will run on cloud in very near future which could save the cost a lot.
A noticeble work to demostrate the effectiveness of hybrid search is blended rag by IBM research (https://arxiv.org/abs/2404.07220), which has shown that 3-way hybrid search could achieve STOA over multiple evaluation datasets.
And also, we've reproduced the results of blended rag, as shown in this article. Additionally, blended rag + colbert based reranker could have a much better results.
The major challenges are how to implement and manage such many indices within single database. That's why we build this database start from scratch. Infinity is actually a kind of "indexing" database, based on a columnar store. The executor also requires refined design to fuse these hybrid search approaches effectively.
There are some vector databases that already include both dense vector search and sparse vector search, such as qdrant. A hybrid search of these 2 does not solve many problems well, such as exact queries. Moreover, according to our experiments, as seen in the article, the performance of dense vector + sparse vector, improves only a little bit. In addition to these 2 way recall, infinity offers bm25 as well as colbert reranker, which can make the ranking quality of the hybrid search much better.
Hi, I'm one of the creators of infinity, and the article has mentioned about the sparse vector vs bm25. While the sparse vector performs well under some evaluations, it is obtained by training a model, which means that it can't fully represent all of the user's keywords/tokens, and those that don't appear in the training set, are truncated. So this is a very big impact for many enterprise vertical scenarios. And bm25 doesn't have such a limitation
From the viewpoint of RAG 2.0, during the stage of indexing or pre-processing, such approaches as knowledge graph is a MUST to resolve such issues as multi hop question answering, long text question answering as well as semantic gap between question and answers. As a result, you could look on graphrag as a component of future RAGFlow. Given the graph based orchestration, integrating graphrag into ragflow is not difficult. As a result, RAGFlow will support graphrag in very near future.
From 0.8, RAGFlow(https://github.com/infiniflow/ragflow) will provide no code workflow orchestration. This article describes what kind of graph orchestration engine is needed, and how it can be used to implement Agentic RAG.
RRF is a simple and effective means of fused ranking for multiple recall.
Within our open source RAG product RAGFlow(https://github.com/infiniflow/ragflow), Elasticsearch is currently used instead of other general vector databases, because it can provide hybrid search right now. Under the default cases, embedding based reranker is not required, just RRF is enough, while even if reranker is used, keywords based retrieval is also a MUST to be hybridized with embedding based retrieval, that's just what RAGFlow's latest 0.7 release has provided.
From the next version(weeks later), Infinity will also provide more comprehensive hybrid search capabilities, what you have mentioned the 3-way recalls(dense vector, sparse vector, keyword search) could be provided within single request.
I read the paper and there are some similarities between ZenDB and RAGFlow, but also many differences.
The goal of RAGFlow is to use computer vision models to recognize the structure of a document, including diagrams and tables, and then to slice these structures into appropriate formats, such as table information combined with table definitions into text, which is then sent to the RAG system to be used for retrieval and answering questions.
ZenDB also makes use of computer vision models to understand documents, but it is mainly used to understand the semantic structure of documents, such as headings, phrases, etc., which also involves semantic-based text clustering. ZenDB also defines a query language specifically for querying these semantics. ZenDB is pretty useful to query and summarize long text.
I think some combination of RAGFlow and ZenDB for processing unstructured document data could be interesting to work on.
Actually we've tried almost all lof existing open source models for document processing, and none of them performs well for complex documents, especially those having complicated tables, such as tables cells without borders, cells need to be combined,...,etc. Although adopting LLMs to perform such document understanding tasks is more scalable, it requires much more data and computation power to achieve similar results. That's why we design such models start from scratch.
Thanks for your nice suggestion. We train the model using YOLO, but during inference, the model is converted into ONNX and we use ONNXRuntime for the model inference. As a result, YOLO itself is not included in the software package.
We will open the training code in the repo soon.
We've used YOLOv8 as the object detection model, and use some public datasets, such as PubTable, CDLA, together with some private data to train the model. The model on Huggingface is the one trained using public dataset, and we would open this work later. We use YOLOv8 just because we want to let the document parser run without GPU, I think you could also try any other object detection models such as Detectron, and use the public datasets to train the model as well. We've not used transfomers for this task, because given limited data, it could not outperform traditional CNN based models.
Hi, buddy, I'm sorry to make you feel like we don't take women's voices into account. We came up with this name because RAG is already a consensus, as an acronym for retrieval augmented generation, RAG is used in many places, and we think it would actually be a standard for LLM oriented B-side scenarios, so that's why we adopted this name.
Congratulations! Vespa is by far the best open source search engine even starts from more than twenty years before. The code base is complicated to support web scale search and right now it could provide both full text search as well as vector search. As a long term memory for LLMs, vector search alone is evidently not enough, while sparse retrieval is also a MUST to be added. That's why Vespa could attract such funding even under current situation.
fractal trees should have a faster read performance with leveldb with a comparable updating performance. There does not exist any open source solution for such structure, while we could hope that www.acunu.com might provide an open source solution for similary Stratified B-tree http://www.acunu.com/2011/04/log-file-systems-and-ssds-made-...
LevelDB should not be compared with the storage model of TC, because leveldb could be looked on a bigtable that run on single node, which is a model that is optimzied for updating instead of reading. The application situation for leveldb is different with that of TC