Logseq – adding settings for self-hosted sync(github.com)
github.com
Logseq – adding settings for self-hosted sync
https://github.com/logseq/logseq/discussions/10733
4 comments
Oh frell. As a user who loves shell scripting, version control, and alternative options, i hate hate hate DB-based datastores. I won't have git auto-commits, showing me over time how I edit logseq. I check in work-notes, which are markdown files that can be viewed easily in GitHub right nkw. I also use file sync tools to keep tablet, phone, and laptop in sync: these can handle resync after online if I'm not working on the same files. Going to a database will blow all this sync up, will go from an open conventional malleable extropic system to something esoteric & cloaked, turn us away from the external & push us internal. This will be awful, I'm so striken to hear this.
DBs are just so awful, such a thing programmers love but which abridges the shareability of the system with users. We already have a great key value system, and it's one that has amazing universal tools around it: the file system. DBs have advantages for querying, so it's fine if you want to project your filesystem's contents into a database, but man, files are just so much more user-friendly, user-empowering. The 9p dream is real, everything is a (simple) file is still the untried win that can bridge the dev-user computing gulf. There was a recent submission, File As Metadata Format, that talked to this all, was so good to see, https://prma.dev/posts/files-as-metadata-format https://news.ycombinator.com/item?id=40890902
I love logseq but this is an app in trouble for sure. It's a huge sprawling Clojure project. That's enormously powerful, but trying to wrestle with that internal complexity is so hard. There's such a huge backlog of open issues. The community loves the product, but there are seemingly so few who have deep knowledge, who can answer questions or improve things or help.
Regarding the Android version, if you tap away from a line, tap back on it, tap away again, and tap on it again the "dot" shows up. It seems to be a visual bug only for me; I can use the "next line" toolbar option to get a new line as usual, navigate around as usual, but yeah it's way harder to navigate & browse since the dots all disappeared two or three months ago.
DBs are just so awful, such a thing programmers love but which abridges the shareability of the system with users. We already have a great key value system, and it's one that has amazing universal tools around it: the file system. DBs have advantages for querying, so it's fine if you want to project your filesystem's contents into a database, but man, files are just so much more user-friendly, user-empowering. The 9p dream is real, everything is a (simple) file is still the untried win that can bridge the dev-user computing gulf. There was a recent submission, File As Metadata Format, that talked to this all, was so good to see, https://prma.dev/posts/files-as-metadata-format https://news.ycombinator.com/item?id=40890902
I love logseq but this is an app in trouble for sure. It's a huge sprawling Clojure project. That's enormously powerful, but trying to wrestle with that internal complexity is so hard. There's such a huge backlog of open issues. The community loves the product, but there are seemingly so few who have deep knowledge, who can answer questions or improve things or help.
Regarding the Android version, if you tap away from a line, tap back on it, tap away again, and tap on it again the "dot" shows up. It seems to be a visual bug only for me; I can use the "next line" toolbar option to get a new line as usual, navigate around as usual, but yeah it's way harder to navigate & browse since the dots all disappeared two or three months ago.
I was originally also a pretty big anti-fan of the whole db idea.
But it makes sense here. What I want is a block-oriented outliner with a lot of referencing around. This is not a great use-case for raw files (the referencing everywhere, specifically), while it's a pretty trivial one for let's say SQLite (which they're using here).
Do note that I believe the current plan is to have two-way sync between files and the db, so presumably custom modification of md files should still work fine. But we'll see how that evolves.
This is exacerbated by e.g. block IDs, which right now have to be shoehorned into the markdown format, while being hidden by the editor - which can be error prone as those IDs can move around. With a DB that's a solved problem. In Obsidian when you're doing block-level linking you end up with explicit "^jnflds" ID markers all over the place at line ends, which is quite awful if that's your primary use-case (to be clear, Obsidian overall is great, just not what I'm looking for - but if you care about the "raw files" approach, they seem to be embracing it).
But it makes sense here. What I want is a block-oriented outliner with a lot of referencing around. This is not a great use-case for raw files (the referencing everywhere, specifically), while it's a pretty trivial one for let's say SQLite (which they're using here).
Do note that I believe the current plan is to have two-way sync between files and the db, so presumably custom modification of md files should still work fine. But we'll see how that evolves.
This is exacerbated by e.g. block IDs, which right now have to be shoehorned into the markdown format, while being hidden by the editor - which can be error prone as those IDs can move around. With a DB that's a solved problem. In Obsidian when you're doing block-level linking you end up with explicit "^jnflds" ID markers all over the place at line ends, which is quite awful if that's your primary use-case (to be clear, Obsidian overall is great, just not what I'm looking for - but if you care about the "raw files" approach, they seem to be embracing it).
Oh thank the stars!
> Do note that I believe the current plan is to have two-way sync between files and the db, so presumably custom modification of md files should still work fine. But we'll see how that evolves.
Perfect, fingers crossed. There was a submission Trying TinaCMS a couple months ago that mentioned what sounds similar & ideal to me, Keeping flat files as a source of truth then having a projection of those files in a DB for faster querying. You can just throw the whole DB away & TinaCMS will rebuild it as needed. I love this approach as a way to give everyone what they need, fingers crossed we see similar here with by beloved Logseq.
https://tina.io/ https://cassidoo.co/post/trying-tinacms/ https://news.ycombinator.com/item?id=37988585
> Do note that I believe the current plan is to have two-way sync between files and the db, so presumably custom modification of md files should still work fine. But we'll see how that evolves.
Perfect, fingers crossed. There was a submission Trying TinaCMS a couple months ago that mentioned what sounds similar & ideal to me, Keeping flat files as a source of truth then having a projection of those files in a DB for faster querying. You can just throw the whole DB away & TinaCMS will rebuild it as needed. I love this approach as a way to give everyone what they need, fingers crossed we see similar here with by beloved Logseq.
https://tina.io/ https://cassidoo.co/post/trying-tinacms/ https://news.ycombinator.com/item?id=37988585
Didn't know "FAMF"
> The impact of a high number of inodes and the limitations of the number of inodes in Unix-like systems. A very valid point. For which I created 4 million directories, under ...
"FAMF" is horrible for large volumes of writing, it might work well for KPM, but I wouldn't trade DB (even if it's SQLite) for FAMF, probably because I'm a software engineer (as you mentioned in your text)
> The impact of a high number of inodes and the limitations of the number of inodes in Unix-like systems. A very valid point. For which I created 4 million directories, under ...
"FAMF" is horrible for large volumes of writing, it might work well for KPM, but I wouldn't trade DB (even if it's SQLite) for FAMF, probably because I'm a software engineer (as you mentioned in your text)
The technical reasoning for moving to a database-backed store is twofold, they're running into performance issues with large graphs and, more importantly, want to support live collaboration. There are a wealth of other note applications that are just plaintext and operate fully locally. IMO it's worth the tradeoff, especially if they continue to support exporting to EDN and markdown (which I expect them to). It's not like they're going with something poorly supported, they're using SQLite compiled to WASM.
Doing any sort of structured querying over files sucks bad and performs even worse.
Doing any sort of structured querying over files sucks bad and performs even worse.
The biggest problem IMO is that querying is too complex. Without that, it's a pretty limited app, given what it's used for.
I completely agree. Logseq is definitely the note-taking app that has fit my brain best.
I especially like the workflow of writing basically everything in the journal, in an outline like:
With that said, I’ve also gotten nervous after Logseq’s quiet period, especially since they use a non-standard Markdown dialect that doesn’t work well in other apps. I also find Obsidian lacking in the outliner department. Recently, I’ve therefore gone back to Emacs Org-mode. I can use it in a similar way to Logseq (with Org Roam), and feel safer that it will continue to be developed for the next decade or more. As an added benefit, I now have CDLaTeX and Evil available.
I especially like the workflow of writing basically everything in the journal, in an outline like:
- [[Topic1]] [[Topic2]] Headline
- Blah blah blah
- Blah
- Blah blah
- [[Topic3]] [[Topic4]] Headline
- Blah blah blah
- Blah blah blah
With that said, I’ve also gotten nervous after Logseq’s quiet period, especially since they use a non-standard Markdown dialect that doesn’t work well in other apps. I also find Obsidian lacking in the outliner department. Recently, I’ve therefore gone back to Emacs Org-mode. I can use it in a similar way to Logseq (with Org Roam), and feel safer that it will continue to be developed for the next decade or more. As an added benefit, I now have CDLaTeX and Evil available.
I have the formatting issue on mobile sometimes as well. Reinstalling the app fixes it. Annoying but it works.
That did in fact work! Thank you!
Been using Syncthing with graphs forever now. Probably will continue using that.
Edit: Not sure why I posted this as a comment on someone else's post, but I'll leave it since it's here.
I haven't followed Logseq much since they announced the move to a database. I liked the idea when it was a regular open source project. They've taken a considerable amount of money from outside, they're moving from plain text files to a database, and they're planning to monetize by selling sync features.
We've been down this road, I don't know, maybe a few thousand times before. We know how this story ends. Heck, we could write the blog posts and the defensive comments about sustainability, running a business, and all that jazz right now.
(This comment is not a request for a lecture on how business models work. I understand how they work, and that's the point of my comment.)
I haven't followed Logseq much since they announced the move to a database. I liked the idea when it was a regular open source project. They've taken a considerable amount of money from outside, they're moving from plain text files to a database, and they're planning to monetize by selling sync features.
We've been down this road, I don't know, maybe a few thousand times before. We know how this story ends. Heck, we could write the blog posts and the defensive comments about sustainability, running a business, and all that jazz right now.
(This comment is not a request for a lecture on how business models work. I understand how they work, and that's the point of my comment.)
> they're moving from plain text files to a database
I wonder if a database is required for CRDTs. I don't know if logseq use that, but if we had a FOSS note-taking app with CRDTs, we might have to give up the text format.
I wonder if a database is required for CRDTs. I don't know if logseq use that, but if we had a FOSS note-taking app with CRDTs, we might have to give up the text format.
You can store a text-editing CRDT's state in a standalone file instead of in a database. However, it is hard to make the file human-legible or updateable. The best you could do is probably something like a piece table (https://en.wikipedia.org/wiki/Piece_table), or the plain text followed by some opaque CRDT metadata.
If you want to preserve the ability to edit plain text files using an arbitrary program, your sync engine could be a file watcher that computes a diff each time the file is saved - like git does, but character-based instead of line-based.
If you want to preserve the ability to edit plain text files using an arbitrary program, your sync engine could be a file watcher that computes a diff each time the file is saved - like git does, but character-based instead of line-based.
To me this both sounds perfectly reasonable for the Logseq developers to ignore, and for a fork to be made to add it.
100% agree. If users want to remove a feature that funds maintenance and development of the project then they should really just fork it.
In this case they would be adding a feature that complements the feature they depend on for funding with one that would be free to use, rather than removing anything.
[deleted]
I'm really looking forward to their db-oriented version which is supposed to be merged into main (here's the long-lived branch[0]) this month. Presumably that will bring the project back up to speed, since that branch is currently almost 4k (!!!) commits ahead of main.
At the same time, I'm a bit worried about how the company is gonna sustain itself. After all they raised quite a bit of money ($4M 2y ago), while at the same time I'm not sure how large a market there is for commercialisation of an open-source PKM app like this. Esp. since it looks like its market-share is maybe ~1/10th that of Obsidian (based on most popular plugin download counts). TFA is kind of related to this.
While Obsidian is great from a sustainability perspective (it seems to me), it unfortunately comes short of being a good outliner.
[0]: https://github.com/logseq/logseq/tree/feat/db