We used frameworks in the past, tried langchain, langgraph, and Openai's agents SDK pretty extensively. Now we roll our own, generally a much better and cleaner experience. We essentially built our own internal framework for our own use-case, we liked the graph approach of langgraph - so we took elements of that. We write everything async, and added nice handling for streaming.
You can see our framework [here](https://github.com/aurelio-labs/graphai). I don't necessarily recommend it as it's built for our use-cases and I make no guarantees for others, but it might be interesting to see what rolling your own might look like
Sharing my walkthrough on fine-tuning LLMs with LoRA using NVIDIA's NeMo microservices. The result is a llama-3.2-1b-instruct model fine-tuned to be really good at function-calling, making it ideal for agent-use.
It was a ton of fun to figure it out and it brought back some nostalgia from the days of training ML models, tweaking learning rates, dropout, and watching loss charts in W&B.
Final performance was way better than any 1-3B parameter LLM I tried with agentic workflows in the past.
Releasing this walkthrough on fine-tuning LLMs with LoRA using NVIDIA's NeMo Microservices (they sponsored the video, but with no reqs on what I do or say). We cover a ton on building prod AI applications, including:
- LoRA fine-tuning of Llama 3.2 1B
- Deploy of private AI agent systems
- Using NVIDIA NIMs to host our fine-tuned LLM
- Interacting with our LLM + streamed output
It was a ton of fun to figure this out, and it brought back some nostalgia from the good old days of training ML models, tweaking learning rates and dropout, and watching loss charts in W&B.
The result is a llama-3.2-1b-instruct fine-tuned to provide pretty good function-calling abilities (better than any other out-of-the-box 1-3B models I tried).
I made a course covering everything you'd need to know to start building with LangChain. It assumes no level of expertise, you can be a complete beginner and by the end of the course be building AI agents with chat memory, streaming, async, etc. There's no cost, it's all free!
I and my team have been building a python library for improving the steerability of AI agents, the library also allows us to add an essentially unlimited number of tools to agents, add safety guardrails, etc. We have a lot more coming and I'm just excited to share our first public release. Let us know your thoughts!
Not weighing on whether it is AGI-like or not, but my view on the search component is that it's like when we google info, we have a search term and we don't know whether that will return good results or not. So I suppose in order for RAG to become better the LLM needs to be able to review the information given, and decide whether to search more (which they can do with agents to be fair), and we also need to be building retrieval systems that do very well at returning the right information even with sub-optimal search queries.
I think LLMs are fairly good at deciding what information is relevant to a query or not though, it is rare that I have found an LLM to be distracted by RAG results (even when they're not relevant)
Yeah RAG has been around for some time, paper [2] being where I first stumbled onto it — I remember a 2-3 years ago building a RAG pipeline using either the model they trained in paper 2, or that was trained based on the same idea, with Deepset AI's Haystack library. It has been a little surreal to see that terminology become so popular again
I may be misunderstanding, but I'll try to answer — quantization typically means retrieval will be slower (if referring to techniques like product quantization), but that is the case whether you're at 10K vectors or 1B vectors, afaik it doesn't really make a difference because you're only quantizing the query vector at query time (it has been awhile since I read anything on quantization, so I could be mistaken).
Maybe your question is referring to the need to have quantization at larger index sizes? In which case, yes would typically be true because you're wanting to either (1) minimize the index size when quantizing it, or (2) optimizing the query space (to search through less). Whether you want (1) or (2) will impact on the type of quantization being performed (basically 1 == product quantization, and 2 == inverted index)
So once you get to the 1M+ size, you need to consider quantization in some form - or you can go with graph-based retrieval, if you don't mind using a lot of disk space.
For Huberman Podcast I imagine he pays someone to do the annotations manually, so they're accurate. But on most videos I've found Whisper's annotations to be more accurate than YouTube's default annotations - not to bash YouTube's, they're still great but occasionally you get some weird annotations
I should add, Riley used the ada embedding model (rather than sentence transformers). Performance wise they should be similar (in ability to encode meaning accurately) but the ada model can encode a much larger chunk of text. I don't know exact numbers but something like 1-2 pages of text in a typical corporate PDF. Whereas sentence transformers are typically limited to around a paragraph of text.
Typically you'd split the text in paragraph sized chunks to handle this requirement of sentence transformers, with GPT-3 embeddings you naturally have more flexibility there
you can return the chunks of text containing the answers, but not generate answers as that isn't what text-embedding-ada-002 is for. For that you need generation model (davinci in this case)
I built something similar using a variety of YouTube channels focused on NLP, AI, etc. The app is here https://huggingface.co/spaces/jamescalam/ask-youtube - you can ask things like "what is a transformer model?" or "what is semantic search?"
- Use sentence transformer to create embeddings of text
- Index embeddings (with transcribed text, timestamps, and video URL attached) in Pinecone's vector database
- Wrap up the querying functionality in a nice UI
(this is for the search functionality)
If wanting to replicate the Q&A part, I also built something similar and wrote about it (https://youtu.be/coaaSxys5so) - it's essentially the same process but we return text snippets to GPT-3 along with the original question and it generates an answer
They act as a "position signal" that modifies the patch embedding. The learned signals are similar to other neighbouring position signals, and the later layers of the model will use the "similarity" between signals to identify the proximity/order of different patches.
There is no explicit mechanism that tells the network to make neighboring position embeddings similar, it's just a result of the training that fortunately works and seems logical.
Faiss is great, but I see there being several strong reasons for using Pinecone:
1. The people who built the Pinecone vector index include some of the top experts in information retrieval/vector search in the world, the chances of someone like myself building something more efficient in Faiss is pretty slim, even after spending months learning the library and as much as I can about vector search
2. Faiss is only a vector index, it doesn't manage your data, so you need to personally build a framework around your Faiss index to handle that, speaking from experience this can be frustrating
3. Pinecone has one of (if not the) most advanced implementation of filtering available, they use 'single stage' filtering, which provides pre-filtering speeds with post-filtering accuracy
4. Scalability, want to search through an index of 2B vectors? Not an issue, getting that working with Faiss is not fun
5. Ease of use, all of the above takes an API key and ~5 lines of code to setup
I think the use case of chatbots is better solved with open domain Q&A (eg https://www.pinecone.io/learn/question-answering/). The focus of most chatbots seems to be on answering questions, but wrapping it up in a nice interface. That's fine but the chatbot can only answer (accurately) questions that have an answer somewhere (probably buried deep in some Q&A pages). It's much more user-friendly imo to have a Google type interface where you can answer a question, and return answers, or at least get an idea of where the answer is - and open domain Q&A does this fairly well (as proven by Google)
You can see our framework [here](https://github.com/aurelio-labs/graphai). I don't necessarily recommend it as it's built for our use-cases and I make no guarantees for others, but it might be interesting to see what rolling your own might look like