Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLC(bentoml.com)
bentoml.com
Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLC
https://bentoml.com/blog/benchmarking-llm-inference-backends
4 comments
How does Exllama rank among these? Heard good things about it.
Seems interesting!
https://github.com/turboderp/exllama
"A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights."
4-bit quantization tends to come at the cost of output quality losses.
https://github.com/ggerganov/llama.cpp/issues/9
Quality loss with quantization is expected. It seems like with GPTQ the loss is within acceptable range based on the perplexity score shown.
Why aren't there more of these benchmark studies? How did TGI make the cut?
Why was onnx not part of the tested runtimes? Seems like an oversight
Personally, I never seen onnx used for LLM.
onnx is not a good option for LLM type of autoregressive generation
[deleted]