Right now I am experimenting with various parsing and chunking strategies for our LLM application. We are using self hosted ChromaDB.
It feels odd to create a collection for each strategy to experiment with. It still works in dev but in prod creating a new collection every time with the relatively big amount of data (35k PDFs which translate into ~600K Docs of ~2000 character length) and then changing the collection name in env vars feels wonky at best.
Is there something I am missing? How do you experiment on parsing/chunking/embedding rapidly on docs at this scale and then point to the latest stable one or A/B test on them?
In an ideal scenario i'd want something like neon.tech where there are branches where we can experiment + we could merge the branch we are satisfied with into main and it takes effect without having to manually change config/env vars.
Please let me know if there is something similar that exists preferably for chromaDB (we don't have money for managed solutions)
It feels odd to create a collection for each strategy to experiment with. It still works in dev but in prod creating a new collection every time with the relatively big amount of data (35k PDFs which translate into ~600K Docs of ~2000 character length) and then changing the collection name in env vars feels wonky at best.
Is there something I am missing? How do you experiment on parsing/chunking/embedding rapidly on docs at this scale and then point to the latest stable one or A/B test on them?
In an ideal scenario i'd want something like neon.tech where there are branches where we can experiment + we could merge the branch we are satisfied with into main and it takes effect without having to manually change config/env vars.
Please let me know if there is something similar that exists preferably for chromaDB (we don't have money for managed solutions)