Towards Optimal LLM Quantization(picovoice.ai)

18 pointsby bejagervor 2 Jahren8 comments

https://picovoice.ai/blog/picollm-towards-optimal-llm-quantization/

6 comments

aa6864aavor 2 Jahren

How does it compare with AWQ, SqueezeLLM, or newer quantization methods?

abcd98vor 2 Jahren

How do you integrate with vLLM?

dynamixvor 2 Jahren

Is there a way for me to compress a custom fine-tuned model of my own?

bejagervor 2 Jahren

not yet but it's something we have in mind as a future feature.

eonlavvor 2 Jahren

Decent platform support - any plans for a Rust SDK?

bejagervor 2 Jahren

We continuously work on expanding SDK support, Rust is also on the list.

avielvor 2 Jahren

Any benchmarks with Falcon 2?

bejagervor 2 Jahren

we don't support Falcon 2 yet but new models are always on our radar to be added to the platform.

[deleted]vor 2 Jahren