On-disk HNSW index for Postgres with pg_embedding(neon.tech)
neon.tech
On-disk HNSW index for Postgres with pg_embedding
https://neon.tech/blog/pg_embedding-on-disk-hnsw-index
7 comments
Thanks for sharing. How does this compare with DiskANN (https://zilliz.com/blog/diskann-a-disk-based-anns-solution-w...) or HNSW-IF (https://blog.vespa.ai/vespa-hybrid-billion-scale-vector-sear...)?
DiskANN construction is much much slower than HNSW, and the support around updates has been discussed in literature but has not been open sourced yet (They call this FreshDiskANN in the paper - DiskANN is from microsoft research btw).
HNSW-IF is an excellent extension to HNSW (that the vespa team has made easy to implement) that takes advantage of the speed/recall of HNSW in combination with the disk scalability of inverted indices - it is a hybrid approach.
HNSW-IF is an excellent extension to HNSW (that the vespa team has made easy to implement) that takes advantage of the speed/recall of HNSW in combination with the disk scalability of inverted indices - it is a hybrid approach.
I've done just a little reading on ANN indexes and data structures, but considering you're at Chroma, could you explain why closeness (inverted distance) would be used in HNSW-IF instead of distance?
This is comparing apples and oranges. You've pointed to some technologies from vector database vendors. Neon isn't a vector database vendor. They're a Postgres vendor. The objective of pg_embedding, pg_vector, and others is to offer teams who deploy Postgres the opportunity to use their existing infrastructure for vector search. The added and important benefit here is hybrid search: using existing DB data to filter semantic search results.
What the work done by Neon, the pgvector team, Supabase and others points to is that "speed" isn't the only factor in vector database selection. Developer experience and existing infrastructure investment are too.
What the work done by Neon, the pgvector team, Supabase and others points to is that "speed" isn't the only factor in vector database selection. Developer experience and existing infrastructure investment are too.
No one is discrediting (or even discussing really) the validity of the point that if you use postgres pgvector or pgembedding may makes sense for the reasons you mention. The question was about how the algorithms compare, I compared the algorithms. Comparing ANN algorithms is relatively apples-to-apples, but is still complicated and filled with nuance.
DiskANN has some great ideas that will likely make it into pg_embedding. The most important implication is how many vectors it can support. DiskANN claims it can manage 1Bln vectors and it's a huge deal. Index construction does take a long time, but we are researching how we can scale it out.
This looks very cool.
I'm interested in how many vectors are indexed/how large the index is that corresponds to the latency chart? If we have an in-memory HNSW index of 10M vectors at ~20GB (512 dim), say, what are the RAM requirements when using the disk-based version?
I'm interested in how many vectors are indexed/how large the index is that corresponds to the latency chart? If we have an in-memory HNSW index of 10M vectors at ~20GB (512 dim), say, what are the RAM requirements when using the disk-based version?
We used 1M vectors of 1536 dimensions (OpenAI). The index size was ~6GB.
Thanks - how much RAM is required to serve from the index in that case?
The GitHub project has no license file that I see. Does anyone know if this is going to be released under an OSS license of some kind?
It's under Apache license
Are there any plans to release binaries for this extension? E.g. TimescaleDB is really easy to install.
CEO of Neon here. After we built an in memory HNSW index for Postgres that allowed us to establish a baseline in performance and prove that it's the right approach to support vector search we now built it "the right way" and now it support restarts of Postgres, replication and the rest of the Postgres machinery.
Can you clarify:
Does the the initial construction (creation) of the index need to fit into RAM?
Also, should one expect the reading via Postgres's buffer cache to have better access patterns / need less memory / have other benefit vs having a non-Postgres HNSW index that's mmapped from disk?
Does the the initial construction (creation) of the index need to fit into RAM?
Also, should one expect the reading via Postgres's buffer cache to have better access patterns / need less memory / have other benefit vs having a non-Postgres HNSW index that's mmapped from disk?
The index doesn't need to fit in the shared buffers to be built, but you can speed the queries if you load it to memory
Forgive me I'm not super familiar with the vector indexes outside of the basic tsvector for text search.
What's the difference between pg_embedding, pg_vector, and tsvector? Are they compariable/interchangable? And how do you know which one to pick?
My understanding is pg_vector has poorer performance compared to some dedicated vector databases, does pg_embedding perform better?
Sorry if these are silly questions.
What's the difference between pg_embedding, pg_vector, and tsvector? Are they compariable/interchangable? And how do you know which one to pick?
My understanding is pg_vector has poorer performance compared to some dedicated vector databases, does pg_embedding perform better?
Sorry if these are silly questions.
Different vector indexing algorithms under the hood, with different tradeoffs.
In this case "hnsw" is the name of the algorithm. If you want to know more, just search "hnsw" and "ivfflat" to understand the differences.
tsvector is something else; it's not a numeric vector index, it's a different kind of data structure that stores the actual lexemes (keywords) with positions and weights to facilitate full text search (rather than using a text embedding model.)
In this case "hnsw" is the name of the algorithm. If you want to know more, just search "hnsw" and "ivfflat" to understand the differences.
tsvector is something else; it's not a numeric vector index, it's a different kind of data structure that stores the actual lexemes (keywords) with positions and weights to facilitate full text search (rather than using a text embedding model.)
pg_embedding is based on a different approximate nearest neighbors algorithm - HNSW, which is generally considered - and studied - to be faster as well as more accurate than pgvectors IVF (ignoring a lot of nuance).
However pg_embedding serves the index out of disk, whereas most vector databases opt to serve the index out of memory, or delegate to mmap'ed files. HNSW is a graph-based algorithm, thus access patterns are random and this does not lend itself to disk based access. I'd expect pg_embedding to be slower than memory-resident indices due to this fact. Also in general with a postgres index, my concern would be scalability and resource isolation. It's convenient to colocate these things but ANN indices have very different CPU/Memory/Disk usage patterns than what you may need for just your relational data.
For example
Chroma -> Serves HNSW out of memory, persists to disk with a WAL.
Weviate -> Serves HNSW out of memory, writes HNSW graph search to WAL and uses that for durability.
Milvus -> Serves HNSW out of memory, supports partial mmap, also supports another algorithm called DiskANN which is optimized for SSD. It uses cloud storage with a shared-everything architecture for durability (a lot of nuance here.)
QDrant -> Has a WAL, supports memory and mmap'ed indices.
However pg_embedding serves the index out of disk, whereas most vector databases opt to serve the index out of memory, or delegate to mmap'ed files. HNSW is a graph-based algorithm, thus access patterns are random and this does not lend itself to disk based access. I'd expect pg_embedding to be slower than memory-resident indices due to this fact. Also in general with a postgres index, my concern would be scalability and resource isolation. It's convenient to colocate these things but ANN indices have very different CPU/Memory/Disk usage patterns than what you may need for just your relational data.
For example
Chroma -> Serves HNSW out of memory, persists to disk with a WAL.
Weviate -> Serves HNSW out of memory, writes HNSW graph search to WAL and uses that for durability.
Milvus -> Serves HNSW out of memory, supports partial mmap, also supports another algorithm called DiskANN which is optimized for SSD. It uses cloud storage with a shared-everything architecture for durability (a lot of nuance here.)
QDrant -> Has a WAL, supports memory and mmap'ed indices.
As I mentioned elsewhere, there's more to vector database selection than raw performance: Developers leveraging their existing experience with Postgres, existing infrastructure investment (often in managed Postgres on AWS/GCP etc), and a single API into the vector store shared with other parts of their app (their ORM / DB layer).
Many teams can also get away with good performance vs the fastest performance, given smaller index sizes and the other tradeoffs I mentioned.
That said, I can imagine the pgvector folks precaching the new HNSW index support they're working on, as they do with their IVFFLAT index.
* edited for the grammar gremlins
Many teams can also get away with good performance vs the fastest performance, given smaller index sizes and the other tradeoffs I mentioned.
That said, I can imagine the pgvector folks precaching the new HNSW index support they're working on, as they do with their IVFFLAT index.
* edited for the grammar gremlins
Sure, I don't disagree that there is more to vector database selection than raw performance. Any technical decision is filled with many considerations.
The commenter - thewataccount - asked about performance and I shared my intuition.
The commenter - thewataccount - asked about performance and I shared my intuition.
pgvector is working HNSW too. We have a blog post about it: https://neon.tech/blog/pgvector-meets-hnsw-index
Well it doesn't serve it from disk. It's persisted to disk and Postgres buffer cache keeps the working set in memory.
Maybe I am misunderstanding but the postgres buffer cache is LRU and will evict pages right? At times data may go to disk and then during serving it will have to be loaded into memory? So it is quite dependent on the size of your buffer cache as well as contention for that buffer cache.
Also the cache access patterns will vary between these implementations and is worth considering.
Also the cache access patterns will vary between these implementations and is worth considering.
For sure.
I would appreciate a rough comparison with usearch:
https://unum-cloud.github.io/usearch/
Which was also recently on HN: https://news.ycombinator.com/item?id=36942993
https://unum-cloud.github.io/usearch/
Which was also recently on HN: https://news.ycombinator.com/item?id=36942993
I think it's good technology. Our team flagged it, we haven't tested it yet.