Sorry I don't know how hackernews notifications work, didn't see this reply. But if you see this here's another reminder for you to clean this up into a public demo :)
This is really cool! Appreciate sharing this work and the explanation.
You mentioned that massaging the data into shape as one of the problems, which I think is possibly one of the best applications of LLMs in my opinion. Creating a pipeline of a data feed (still hard) into an LLM which outputs JSON with the fields of interest would be amazing.
I just found about this project from this comment, absolutely excited to try this out.
As someone who's never used any of the infrastructure tools, I'm thinking of pyinfra as a way to run shell commands + install dependencies on hosts (declaratively?) on a bunch of hosts via ssh.
Inventory is to sort of take a self-defined inventory on a bunch of hosts?
One final question on usage, would it be possible to sync or reference files from the machine running pyinfra on the remote hosts? Or would that have to be done indirectly via running shell commands to sync?
I think distillation in the original sense isn't being done anymore but finetuning on outputs from larger models like GPT-4 is a form of distillation (top-1 logit vs all logits and a curated synthetic data instead of the original dataset)
On quantization though its still weird how just the weights are quantized in methods like gptq / int8 while there are other methods which quantize the activations as well. There's also the matter of KV cache still being in original 16bit precision regardless which is also unsolved here. Do you have any thoughts or insights into this?
> This means it is way cheaper to look something up in a vector store than to ask an LLM to generate it. E.g. “What is the capital of Delaware?” when looked up in an neural information retrieval system costs about 5x4 less than if you asked GPT-3.5-Turbo. The cost difference compared to GPT-4 is a whopping 250x!
In a narrow use-case of a strict look-up. This seems to exaggerate the cost difference while having completely different trade-offs.
There are some unique assumptions being made in parts of the gist
> 10: Cost Ratio of OpenAI embedding to Self-Hosted embedding
> 1: Cost Ratio of Self-Hosted base vs fine-tuned model queries
I don't know how useful these numbers are if you take away the assumptions that self-hosted will work as well as API.
> 10x: Throughput improvement from batching LLM requests
I see that the write up mentions memory being a caveat to this, but it also depends on the card specs as well. Memory Bandwidth / TFLOPs offered by say 4090 is superior while having the same amount of VRAM as 3090. The caveat mentioned with token length in the gist itself makes the 10x claim not a useful rule of thumb.
I've been looking for a course like this! Especially great given how much of the recent progress in training large models is made possible with the aid of flash attention and fused kernels
That fact that OpenAI decided to not even reveal the parameter counts / training data composition in their "technical" report is surely a sign that their moat isn't as big as it would seem in fear of competition. This paired with the rise of LLaMa and other alternatives makes the monopoly even less likely.
Its just a matter of time till every company has an internal ChatGPT equivalent.
I think it is definitely a hard problem to solve on a large scale to address latency, quality and size of the index they plan to address. It definitely isn't as easy as spinning up an elastic search cluster.
I agree that getting something "mostly" good or a domain specific search engine isn't as hard with the newest advances in this space with vector similarity indices.
While I do agree with what you've said, I think they should be able to price these into a bill that they can send the app developers monthly. They have shown competency in running the pricing on AWS to such great margins and few people seem to complain about pricing.
Perhaps it is a combination of the sentiment that they are owed a percentage of your income and it being such a large number (20-30%) frustrates the developers.
I've taken Georgia Tech's OMSCS program. What really sets this apart is the active feedback on assignments / course interaction from Professors and TAs in Piazza.
Current online courses have really watered down content and zero/terrible ways of evaluating the material.
Things that I like too see in an online course:
- Keep assignments on a regular basis (on par with your campus schedule). Quizzes are great to keep them engaged but
do not use them to replace regular assignments.
- Design them to be auto-graded as much as possible. Bonus points if there was some way the students can evaluate their solution locally.
- Something like Piazza/Forum setup to answer student questions and have an appropriate number of TAs.
- Post all the resources used in the videos to be accessible for later reference (slides and links).
If you're recording content for future offerings, ensure that the content is consistently updated to stay relevant.