Show HN: Making AI chat sessions durable to network failures(github.com)

5 pointsby infiniteregretsvor 12 Monaten0 comments

Show HN: Making AI chat sessions durable to network failures

https://github.com/s2-streamstore/resumable-stream

The experience with AI chat apps can be a bit fragile — there's no resilience to network failures, and responses aren't deterministic like traditional search. As Garry Tan put it (https://x.com/garrytan/status/1927038513108701662), "It feels like catastrophic data loss when a given response comes back and fails halfway, and then the next retry is not as good."

Vercel recently released a "resumable-stream" package (https://github.com/vercel/resumable-stream) to solve this problem using Redis as a backing store.

I felt that we could simplify the approach using s2.dev (I'm an engineer on the team), and created a fork which ended up borrowing the API but otherwise completely rewritten: https://github.com/s2-streamstore/resumable-stream

- S2 streams are cheap, so instead of optimizing to flush only when there is another reader, the stream is always created and written to.

- Unlike the typically ephemeral nature of Redis, tokens are always durable, and there are no memory constraints around number of streams or how much data is on them.

- Relies on S2’s built-in concurrency control. A unique fencing token ensures a single writer, and a sequence number is optimistically supplied to match the one expected to be assigned to the next record, to deduplicate in case of retries.

It integrates nicely with Chat SDK, and you can try this out here: https://ai-chatbot-s2.vercel.app by reloading the page during a larger response!

There are some interesting future possibilities, like building a multiplayer chat experience around shared streams.

0 comments

—