> As an early proof of concept, Kimi K3 designed a chip to serve a nano model built on its own architecture. In a single 48-hour autonomous run, K3 built, optimized, and verified the chip using open-source EDA tools on the Nangate 45nm library. Within 4 mm², the chip closes timing at 100 MHz and sustains over 8,700 tokens/s decode throughput in simulation, packing 1.46M standard cells, 0.277 MB of SRAM, and an INT4 MAC array with fused dequantization. A chip built by a model, for a model, reflects K3's long-horizon agentic capabilities.
This is a fair critique! We plan to use our system to generate many more inference libraries of this nature, and I'll make it a point to release better, broader correctness measures when we do so.
Unfortunately, not at present; we went for FP8 because we believed it was generally the best tradeoff of quality and speed. Allowed faster iteration as well.
We believe our improvements would hold on BF16, but let me check.
Yes, speculative decoding will make both us and VLLM faster, but we believe it would be a relatively even bump on both sides, so we didn't include it in this comparison. Worth another test!
We validate with MMLU and Hellaswag presently, and are getting this independently verified by a 3rd party.
We have considered open-sourcing some of our optimized inference libraries in the future, but have not yet come to a decision on this.
Also if you need a rough intuition as to why this is possible: it's because this entire inference stack was built for exactly one model, and thus we can really tune the entire framework accordingly.
Say hi : luke (at) lukebechtel.com
Current: - Founding Engineer @ Infinity AI Institute - AI/ML Consulting @ Positive Sum Products (https://positivesumproducts.com)
Former: - Founder @ Reasonote (https://reasonote.com) - Principal AI/ML Eng. @ Regscale (https://regscale.com) - Dir. of Eng. @ Revaly (now Cadchat) (Acq. in 2023) - Cofounder @ Collider Inc. (Acq. in 2022)