Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLC(bentoml.com)

16 pointsby helloericsf2 anni fa8 comments

Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLC

https://bentoml.com/blog/benchmarking-llm-inference-backends

4 comments

ssheng2 anni fa

How does Exllama rank among these? Heard good things about it.

Seems interesting! https://github.com/turboderp/exllama "A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights."

helloericsf2 anni fa

4-bit quantization tends to come at the cost of output quality losses. https://github.com/ggerganov/llama.cpp/issues/9

ssheng2 anni fa

Quality loss with quantization is expected. It seems like with GPTQ the loss is within acceptable range based on the perplexity score shown.

ShawnBasquiat2 anni fa

Why aren't there more of these benchmark studies? How did TGI make the cut?

timliu92 anni fa

Why was onnx not part of the tested runtimes? Seems like an oversight

helloericsf2 anni fa

Personally, I never seen onnx used for LLM.

chaoyu2 anni fa

onnx is not a good option for LLM type of autoregressive generation

[deleted]2 anni fa