What is Azure thinking with this VM size NVadsA10 v5-serieslearn.microsoft.com1 points·by ssheng·vor 2 Jahren·1 comments
ssheng·vor 2 Jahren·discussTo get a full A10, one must also get 36 vCPU and 440 GB of memory. I must be missing something.
ssheng·vor 2 Jahren·discussQuality loss with quantization is expected. It seems like with GPTQ the loss is within acceptable range based on the perplexity score shown.
ssheng·vor 2 Jahren·discussCreative idea. Any data on how much it takes to load the LoRAs and how much latency it adds to the generation speed?
ssheng·vor 2 Jahren·discussHello! We're the authors of this blog post. Please let us know if there are other models and inference backends you'd like us to benchmark next.
ssheng·vor 4 Jahren·discussThe user feedback seems really encouraging. https://www.linkedin.com/posts/eric-riddoch_its-early-to-say...
ssheng·vor 4 Jahren·discussFastAPI is great building block but can't expect it to work for model serving out of box.