Basically- the same math as modern automated manufacturing. Super expensive and complex build-out - then a money printer once running and optimized.
I know there is lots of bearish sentiments here. Lots of people correctly point out that this is not the same math as FAANG products - then they make the jump that it must be bad.
But - my guess is these companies end up with margins better than Tesla (modern manufacturer), but less than 80%-90% of "pure" software. Somewhere in the middle, which is still pretty good.
Also - once the Nvidia monopoly gets broken, the initial build out becomes a lot cheaper as well.
We have been running this playbook for the last 2 years in healthcare, and we have been super successful. Doubling every quarter over the last year. 70%+ profitability, almost 7 figures of revenue. 100% bootstrapped.
People are still mentally locked in to the world where code was expensive. Code now is extremely cheap. And if it is cheap, then it makes sense that every customer gets their own.
Before - we built factories to give people heavy machinery. Now, we run a 3d printer.
Everyday I thank SV product-led growth cargo cults for telling, sometimes even forcing our competition to not go there.
One of the most pleasant experiences I had writing code, is early AI days when we did hyperscript SSE. Super locality of behavior, super interesting way of writing Server Sent Events code.
on message as string
put it into #div
end
on open
log "connection opened."
end
on close
log "connection closed."
end
on error
log "handle error here..."
end
Lots of YC companies copy each other process and selection criteria. Basically- they all have the same blind spots and look for the same type of engineer.
So, super easy to scam all of them with the same skillset and mannerism.
I send this article as part of onboarding for all new devs we hire. It is super great to keep a fast growing team from falling into the typical cycle of more people, more complexity.
The “equivalent” here would be Jina-Clip (architecture-wise), not necessarily performance.
The ColPali paper(1) does a good job explaining why you don’t really want to directly use vision embeddings; and how you are much better off optimizing for RAG with a ColPali like setup. Basically, it is not optimized for textual understanding, it works if you are searching for the word bird; and images of birds. But doesn’t work well to pull a document where it’s a paper about birds.
I would like to through our project in the ring. We use ColQwen2 over a ColPali implementation. Basically, search & extract pipeline: https://docs.colivara.com/guide/markdown
Here is a nice use-case. Put this in a pharmacy - have people hit a button, and ask questions about over-the-counter medications.
Really - any physical place where people are easily overwhelmed, have something like that would be really nice.
With some work - you can probably even run RAG on the questions and answer esoteric things like where the food court in an airport or the ATM in a hotel.
We benchmarked two ways to improve the latency in RAG workflows with a multi-vector setup. Hybrid search using Postgres native capabilities and a relatively new method of token pooling. Token pooling unlocked up to 70% faster latency with <1% performance cost.
So - the synthetic QAs datasets in the Vidore datasets are exactly like that 90% text, 10% charts/tables. OCR + BM25 is at ~90% NCDG@5 which is pretty decent. ColPali/Ours is at ~98%.
It is a small upgrade, but one nonetheless. The complexity, and the cost of multi-vectors *might* not make this worth it, really depends on how accuracy-critical the task is.
For example, one of our customers who does this over FDA monographs, which is like 95%+ text, and 5% tables - they misses were extremely painful - even though there weren't that many in text-based pipelines. So, the migrations made sense to them.
1. Uses half-vecs, so you cut down everything by half with no recall loss
2. Uses token pooling with hierarchial clustering at 3, so, you further cut down things by 2/3rd with <1% loss
3. Everything is on Postgres and pgvector, so you can do all the Postgres stuff and decrease corpus size by document metadata filtering
4. We have a 5000+ pages corpus in production with <3 seconds latency.
5. We benchmark against the Vidore leaderboard, and very near SOTA
I would check out vision models as a technique to go around OCR errors.
ColPali is the standard implementation & SOTA. Much better than OCR. We maintain a ready to go retrieval API that implements this: https://github.com/tjmlabs/ColiVara
We have traditionally used Django in all our projects. We believe it is one of the most underrated, beautifully designed, rock solid framework out there.
However, if we are to be honest, the history of async usage in Django wasn't very impressive. You could argue that for most products, you don’t really need async. It was just an extra layer of complexity without any significant practical benefit.
Over the last couple of years, AI use-cases have changed that perception. Many AI products have calling external APIs over the network as their bottleneck. This makes the complexity from async Python worth considering. FastAPI with its intuitive async usage and simplicity have risen to be the default API/web layer for AI projects.
tldr: Async django is ready! there is a couple of gotcha's here and there, but there should be no performance loss when using async Django instead of FastAPI for the same tasks. Django's built-in features greatly simplify and enhance the developer experience.
So - go ahead and use async Django in your next project. It should be a lot smoother that it was a year or even six months ago.
The main benchmark for this is the Vidore leaderboard. Where we would love to see where VoyageAI performs compared to the more open-source implementations.
This is great find, very juicy, and would kill a few hot startups.