The open source AI ecosystem is progressing at a pace that is hard to keep up with. Our team is building KubeAI to make it easy to quickly operationalize OSS models and tools within Kubernetes. The goal is to serve as a drop-in replacement for proprietary platforms like OpenAI while also integrating well with existing cloud infrastructure (i.e. messaging systems, databases, etc). Design decisions are being made with an adherence to simplicity as the guiding principle (no dependency on Istio, etc). We currently have implemented support for scale-from-zero inferencing (built-in vLLM & Ollama operator). Next up, we are are going to focus on end-to-end training-through-serving support for LoRA adaptors next!
Would love to hear feedback and ideas!