I see. That's slightly unfortunate. In principle, increasing temperature flattens out the distribution but the ordering between different tokens' probabilities remain the same, so setting a top-k shouldn't break my test. Can't say the same for top-p though. And all of this is probably too deep into the provider's implementation details for me to make assumptions on.
I didn't know this! I've always thought speculative decoding was "if p(draft_token) > threshold, use it". You made me go read how it actually works and it's pretty neat!
That said, I still think some providers are cheating. Please correct me if the test below is flawed.
I generated texts at temperature = 0 vs temperature = 2. At high temperature, the distributions effectively become flatter, meaning the difference between real and draft effective distributions (the D_LK used in theorem 3.5 of 2211.17192) becomes smaller. When T=2, the model speaks complete gibberish, so the effective distribution must be pretty flat. This should mean fewer rejections --> a lot faster speculative decoding. Yet, I see no increase in throughput at all...
Fair point. If Moonshot is holding back the true weights or inference techniques that affect correctness, then providers including Together should call them out on that. I for one would stop using Kimi if that is the case.
Anyway, Novita is doing significantly better on the vendor verifier chart than Together, so the low quality must be partially Together's fault at least.
Inference is impressively fast. But what about quality? In the Kimi vendor verifier (https://github.com/MoonshotAI/K2-Vendor-Verifier/), Together has one of the highest tool call failure rates (>300 failures over the benchmark, compared to 0-2 for the official API, groq, SiliconFlow, and Infinigence).
The instruct model is trained on the Baize dataset, which is generated from ChatGPT. So it is no surprise that Falcon censors the topics that ChatGPT censors.
If you have an uncensored instruct dataset, you can use that to fine tune your own uncensored version from the base Falcon.
Async UI is similar to Sycamore in term of not diffing VDOM.
The API is different in that in Sycamore, you tell the framework what to render (by using sycamore::render) and the framework will handle it from there. In Async UI, you await what you want to render yourself. Async UI's API is more transparent in this way, and this brings benefits including
- making async control flow (like the control flow example in the blog post) possible
- making component simply async function or anything that implements IntoFuture<Output = ()>
Sycamore's reactivity is pretty painless (a little too magical for my taste, but that's probably just me), so it's something Async UI can learn from.
I must admit I've never seen Concur before!.The idea of using generator/async is pretty similar. The difference, as I understand, is that in Concur, you yield the widget and let the framework mount it, while in Async UI, you await the widget yourself.
The Todo example is 2.5 MB non-gzipped. About 600KB gzipped.
Still large, but it's because we're shipping a lot of code that we don't need. Mostly the APIs exposed by web_sys. Proper dead code elimination would bring it down a lot.
That example is indeed pretty noisy, but partly due to my premature optimization. The operations of adding, editing, and toggling Todo items are all O(log N)[1]. Things could be simpler if I’d just take the O(N).
[1] O(log N) for “our” code. I don’t know what the browser’s rendering/layouting engine is doing.
In general Async UI will still be noisier than SwiftUI or React, but I hope only in the way that Rust is more explicit/verbose than other languages.
RUI is really cool, but it is solving a different problem. It is immediate-mode, and it is focused on building a new toolkit. Async UI is more about re-exposing existing retained-mode toolkits in nice-to-use, Rust-friendly APIs.
The cool thing is this: with some effort, it is probably possible to adapt the two library together to have a UI with RUI's rendering system and Async UI's APIs.
This uses the executor used by async-std rather than the one used by Tokio. So if you're using something Tokio-specific, you would probably need async-compat[1].
But can you share more on what you need Tokio for? I'd completely understand if we're working with servers. But when it come to clients - UI applications - I feel like async-std and smol are pretty competitive.
The async control flow is not directly inspired by anything. It is a cool side effect of using async for everything that I myself only discovered once I started writing examples.
Async UI as a whole is inspired by the simple fact that UI is an effect system[1], and async is also an effect system.
Re error handling:
There's no real support yet. For now when I hit an error I just render nothing. I might add support for components returning Result<_, _> in the future.
You're right that components should intuitively pend forever. But with Async UI, the line between a component and a normal async function is blurry; our login_form eventually returns, so it is a normal async function, but it also renders something, so it is kind of a component...
Concretely the login_form function works by racing a render (a true, never-completing component) with a "listener" future that completes when the user submits their login. Once the listener completes, the race is over and the render future gets dropped. We can then return from login_form.
Hi HN! I've been working on Async UI for half a year now, and I think it is time I share it with everyone. The project is not even really usable yet (due to lack of implemented components and documentation), but I think the proof-of-concept is enough to demonstrate this new way of doing UI — a way that, I believe, is more suitable to Rust than existing UI framework designs.
P.S. My website is new, so if you find any readability or accessibility issue, please let me know!