This is such an interesting peek into the engineering of these sort of systems with very constrained resources at the time. Like any random file you open contains little pragmatic design decisions like this
# THE POSITION OF THE MOON IS STORED IN THE COMPUTER IN THE FORM OF
# A NINTH DEGREE POLYNOMIAL APPROXIMATION WHICH IS VALID OVER A 15
# DAY INTERVAL BEGINNING SHORTLY BEFORE LAUNCH. THEREFORE THE TIME
# INPUT BY THE USER SHOULD FALL WITHIN THIS 15 DAY INTERVAL.
We're building a new multiplayer IDE but for docs/tasks [1]. Local-first, real-time collaborative and end-to-end-encrypted sync. Not open source but self-hostable with a single binary and hackable with plugins (custom properties, views, code, etc).
Drop the “chat” and just call it “control”. The current proposal is so vague it would cover anything with online sharing/syncing people can sign up for. Any SaaS, any app, any service. Chat, email, file syncing, todo lists, doesn’t matter.
This was such a cool demo back then. There were many systems that could boot from a floppy, of course, but booting into a GUI with TCP/IP stack showing a real internet browser was really something!
Combining the feel of plain text with real structure is also exactly why we're building an "IDE but for tasks/notes" [1].
With structured apps (task managers, outliners) you lose the illusion of editing plain text, but plain text alone lacks things like structure, links, dates, and collaboration. We've spent the last few years building an editor completely from scratch to keep the ease of text editing while adding planning and structure.
Funny how reasonable performance is now treated as some impossible lost art on the web sometimes.
I posted a little clip [1] of development on a multiplayer IDE for tasks/notes (local-first+e2ee), and a lot of people asked if it was native, rust, GPU rendered or similar. But it's just web tech.
The only "secret ingredients" here are using plain ES6 (no frameworks/libs), having data local-first with background sync, and using a worker for off-UI-thread tasks. Fast web apps are totally doable on the modern web, and sync engines are a big part of it.
This is awesome! The demo is really fun, nostalgic 90s vibes, it even lets me play chiptunes :). Keeping all dependencies optional and just making it a single file is great too!
A backend can be part of the functionality though, such as for real-time collaboration and syncing. But you can have ownership and longevity guarantees for both the data and the service as long as you can eject [1] from the cloud and switch to self-host or back at any time, which is what we do for our notes/tasks IDE
Very cool! This is so clever and makes deploying it so simple. I just wasn't able to use it yet because we'd have (many) thousands of SQLite DBs to backup. I quickly hacked something together using fanotify + SQLite's Backup API to have some copies at least, but I'm going to try to switch to Litestream if this amount of files would be supported by the wildcard replication.
Depending on the threat model you want to protect against, yes. For example, although a revoked user doesn't have direct access to data anymore, they could technically collude with us to get future encrypted messages and then decrypt them. At the moment rekeying is only possible by exporting a workspace, changing its key (with a command line tool), and then reimporting it.
The workspace data is encrypted with a (symmetric) workspace key. Each user gets hteir own encrypted copy of that key, using the user's public key. By wrapping the data-encryption key for each user the content is only encrypted once but each authorized user can decrypt it.
We're building a collaborative IDE for tasks and notes [1] from scratch without frameworks/dependencies.
Not saying frameworks are never the right answer of course, but it's as much a trade-off for complex apps as it is for blogs. Things like performance, bundle size, tooling complexity, easy of debugging and call stack depth, API stability, risk of hitting hard-to-work-around constraints all matter at scale too.
Looks impressive! Using the VFS is such a fun "hack" :)
We developed our own sync engine for an offline-first IDE for notes/tasks [1] we're building, where the data structure is a tree (or graph actually) to support outlining operations. Conflict resolution is always the challenge, and especially with trees multiple offline players can optimistically commit local changes which would result in an invalid tree state once globally merged.
The way we handle this is by rolling back tree inserts when a client comes online and receives other tree insert deltas. From what I understand from the description of SQLSync, the best way to handle this would be to pull in a latest snapshot and then replay. Pulling in a full snapshot sounds rather expensive though. We have some sort of heuristic where we can do this if the number of incoming deltas would be very large, but for most offline syncing we never need it. Just curious how SQLSync defines these snapshots? Sticking with the note-taking example, in our case we can't really have a snapshot of a single "note" because of graph features like transclusions. Does SQLSync have some clever way to avoid having to send all state in order to "reset and replay"?
We've built a sync engine from scratch. Our app is a multiplayer "IDE" but for tasks/notes [1], so it's important to have a fast local first/office experience like other editors, and have changes sync in the background.
I definitely believe sync engines are the future as they make it so much easier to enable things like no-spinners browsing your data, optimistic rendering, offline use, real-time collaboration and so on.
I'm also not entirely convinced yet though that it's possible to get away with something that's not custom-built, or at least large parts of it. There were so many micro decisions and trade-offs going into the engine: what is the granularity of updates (characters, rows?) that we need and how does that affect the performance. Do we need a central server for things like permissions and real-time collaboration? If so do we want just deltas or also state snapshots for speedup. How much versioning do we need, what are implications of that? Is there end-to-end-encryption, how does that affect what the server can do. What kind of data structure is being synced, a simple list/map, or a graph with potential cycles? What kind of conflict resolution business logic do we need, where does that live?
It would be cool to have something general purpose so you don’t need to build any of this, but I wonder how much time it will save in practice. Maybe the answer really is to have all kinds of different sync engines to pick from and then you can decide whether it's worth the trade-off not having everything custom-built.
The car industry in Germany is being Nokia'd. The iPhone didn't win because the US had central planning.
I'm not convinced at all free market economics are holding us back. Rather other parts of the world are leapfrogging us despite not adopting it.
It was always obvious EVs would be the future. I remember being at a friend's party in Germany more than a decade ago, talking to some guests who worked in the auto industry. Told them about this new upcoming car company called Tesla and how it would transform the entire transportation sector and eat their lunch. The entire group laughed me out of the room and talked about the amazing research they were doing on "clean diesel" or something, I kid you not.
It's simply a combination of innovator's dilemma, complacency and no appetite for risk taking.
Yep love CRDTs but I think we are in agreement that it's a matter of trade-offs whether or not to use a backend (beyond a shared fs service). The idea with ejectable is that if you've decided the trade-off of a backend is worth it to enable certain features/UX, then it's a good way to ensure longevity of the entire app for users.
Co-founder Papyrs. Wrote hobby OS, TriangleOS.
Feel free to contact me, wim at stunf.com or https://x.com/wcools/