Every time I start a new project I have to collect the data and guide clients through the first few weeks before I get some decent results to show them. This is why I created a libary to quickly build classic data science pipelines with LLMs, which you can use to demo any pipeline and even use it in production for non-critical use cases.
I belive that LLMs will become better and better in the near future and pipelines will replace classic approaches with LLM-enriched pipelines will drastically simplify the ETL flows.
If your problem is compute, you are already optimizing. This is here for all the steps before you start thinking latency-compute. Not all use cases are made equal.
I see it as a gray area - long term there will be a need for both and you will have just one tool to choose from when presented with time-budget-quality constraints.
You are right! This is here to be used when your resources do not allow you to build full-blow solutions. Yes, I used LLMs to help create examples from my existing code, but they are based on things I have put in production when the client's resources were limited and wanted to move from point 0 to test out the potential of LLMs on their data.
Every time I wanted to use LLMs in my existing pipelines the integration was very bloated, complex, and too slow. This is why I created a lightweight library that works just like scikit-learn, the flow generally follows a pipeline-like structure where you “fit” (learn) a skill from sample data or an instruction set, then “predict” (apply the skill) to new data, returning structured results.
High-Level Concept Flow
Your Data --> Load Skill / Learn Skill --> Create Tasks --> Run Tasks --> Structured Results --> Downstream Steps
Each step results can be turned to JSON and reused later or elsewhere.
When you want AI to perform tasks independently, it's important to be able to control it and understand its actions. Additionally, you need a way to organize the results and manage the inputs efficiently. FlashLearn simplifies this process by using JSON definitions to structure everything clearly. This structured approach ensures that you can easily define tasks, process results, and handle inputs, making AI management more straightforward.
Yes, you can! The responses are always structured, you will have to read the files yourself (as img/text/audio or mixed) and then you can feed them to the FlashLearn agent to complete tasks with them.
Yes, you can find them on GitHub in the folder /examples. There you will find more samples and flows on how to use it. The examples are elementary, but they should give you an idea of how to build one.
I made a free ChatGPT bot for students with a bit of smart prompt engineering and data structuring(embeddings). There is a PRO version for longer responses etc. but you will not need it in 99% of the cases :)
If you have been playing around with OpenAIs APIs you know how annoying it is to actually integrate them to existing rpducts, since it's responses are not consistent and are really just generated text.
I have created an API that structures the responses for the most common NLP tasks to simply intgerate it into any existing app.