Might be interesting to add an optional embedded Weaviate [1] with a flat-index [2] to the project. It wouldn't use external services and is fully disk-based. Would allow you to search the whole filesystem (about 1.5kb per file (384 dimensions) which would be added to the metadata as well).
We also want to share details about how HammerBot works on the backend. Right now, our developers are using OpenAI’s text-davinci-003 model, otherwise known as GPT-3, trained on a custom dataset of our articles. The data is stored in a vector database from Weaviate and the bot is coded primarily with Python, using LangChain, a framework that makes it easy to customize AI output.
When you enter a prompt, the server queries against the dataset that’s stored in Weaviate to get the search results. Those are then sent to the LLM to help it develop a consistent response to your question.
A lot of chatbots will talk to you about anything on Earth. But don’t ask HammerBot for a knitting pattern! It’s designed to have limits on what it will talk about; it focuses on the expertise you can only get from Tom’s Hardware, so it may say it doesn’t know or can’t answer if prompts that fall outside of its training.
What you propose is certainly possible, you could do that with a vector database like Weaviate (https://weaviate.io). Import the data, vectorize it, and connect it to a person with cross-references
Well, there is is still a lot to do. Going from memory to disk, etc. But I see your point.
What might be a bit confusing is the mix of vector search libraries with vector search engines. It's a bit like comparing an inverted index library with SOLR :)
But we (I.e., the wider vector search ecosystem) are working on this
Agreed with your observations about the difference between the libraries and databases. Maybe for a follow-up article? It's nice to see some independent research on the topic, tho.
Agreed with fzliu, you can also use https://weaviate.io (disclaimer, I'm affiliated with Weaviate). You might also like this article which describes why one might want to use a vector search engine: https://db-engines.com/en/blog_post/87
Im affiliated with a company that makes one. We create a vector search engine called Weaviate [1] we also publish content on how it's done and the search engine itself is open source which might be helpful for you too.
Always when Ted Nelson shows up, this point is made one way or the other. I even once heard Vint Cerf make these points at a conference.
1. Xanadu being vaporware
2. His ego
What I really don't get is what Ted Nelson does to trigger this anger in people.
E.g., his book computer lib/dream machines inspired so many, and I find him super self-aware. His appearance in Werner Herzog’s movie about the internet was (IMHO) so refreshing.
I love Ted's work, and it's a huge inspiration to me in my daily work. This video just adds to that.
Interesting writeup. I'm of the opinion that the problem of the naming issue (how to call "things"?) sits in the idea that going from structured documents to structured data is one abstraction level too deep (i.e., people don't agree on how to call "things"). I believe this can be solved by similarity search; if we can approximate the data and represent the structure in embeddings. Hopefully, this might be a step in the 2nd try, as mentioned in the MD :)
> It would be like wikipedia, but even more all encompassing, and far more transformational.
MESHCode is a theory about how information might be stored in the brain. Very interesting and potentially has other implications for information storing as well. Check "The Mechanical Basis of Memory – the MeshCODE Theory": https://www.frontiersin.org/articles/10.3389/fnmol.2021.5929...
1. https://weaviate.io/developers/weaviate/installation/embedde... 2. https://weaviate.io/developers/academy/py/vector_index/flat