When it comes to web archiving, I've found that Markdown has some real limitations. Sure, it's great for basic text, but it struggles with things like embedded content and non-standard layouts. Try archiving a Twitter thread or an app-style webpage in Markdown, and you'll see what I mean. It just doesn't capture the full picture.
That's why I've come to prefer formats like webarchive, mhtml, or single HTML files for archiving. They're incredibly faithful to the original content - you get almost perfect rendering of the original page, complete with styling and layout. Plus, they can capture stuff behind paywalls or on logged-in pages, which is a huge plus.
The real challenge, though, isn't just about saving the content. It's about making that saved content useful. These archive formats are great for preservation, but they can quickly become a mess of unorganized files that are hard to search through or make sense of.
I think the key is finding ways to organize and interact with these archives more effectively. Things like full-text search across all your saved pages, the ability to add notes or highlights directly on the archived content, and smart tagging systems could go a long way. And it'd be really powerful if we could integrate these archives with other knowledge management tools we use.
I develop a tool called HamsterBase that seems to address a lot of these issues we've been discussing. t's a local-first app. That means all your data stays on your own device - no need to worry about your personal archives being stored on someone else's servers. There's no sign-up or registration required, which is refreshing in today's cloud-centric world.
When it comes to web archiving, I've found that Markdown has some real limitations. Sure, it's great for basic text, but it struggles with things like embedded content and non-standard layouts. Try archiving a Twitter thread or an app-style webpage in Markdown, and you'll see what I mean. It just doesn't capture the full picture.
That's why I've come to prefer formats like webarchive, mhtml, or single HTML files for archiving. They're incredibly faithful to the original content - you get almost perfect rendering of the original page, complete with styling and layout. Plus, they can capture stuff behind paywalls or on logged-in pages, which is a huge plus.
The real challenge, though, isn't just about saving the content. It's about making that saved content useful. These archive formats are great for preservation, but they can quickly become a mess of unorganized files that are hard to search through or make sense of.
I think the key is finding ways to organize and interact with these archives more effectively. Things like full-text search across all your saved pages, the ability to add notes or highlights directly on the archived content, and smart tagging systems could go a long way. And it'd be really powerful if we could integrate these archives with other knowledge management tools we use.
It's an interesting problem space, and I think there's a lot of room for innovation in how we approach personal web archiving and knowledge management.
I have a read-it-later tool (https://hamsterbase.com/) that I've been maintaining as a hobby for 2 years, supporting Mac, Linux, Windows, iOS, Android, and self-hosted Docker.
All these versions share the same codebase.
VSCF: I extracted part of the source code from VS Code and developed a framework called VSCF. It includes commands, themes, dependency injection, key bindings, IPC.
Local server: On top of VSCF, I developed the underlying business logic, and file IO, SQLite, and the logic related to synchronization will be placed here.
Frontend: On top of VSCF, I used TypeScript and React for front-end development.
nodejs-mobile: This is an open-source project that allows me to use Node.js on mobile devices.
Self-hosted version = local server + frontend, using WebSocket for communication.
Desktop version = Electron + local server + frontend, using Electron's IPC for communication.
Mobile version = nodejs-mobile + self-hosted version. Users use webview to access the UI. It looks just like a native application. They can even use their phones as servers to access data on their phones from a computer.
I suggest trying out HamsterBase (HamsterBase is not open source).
1. It supports direct binding with SingleFile, enabling one-click web page saving. Because it saves in the browser, all other plugins will take effect.
2. It provides an open-source plugin https://github.com/hamsterbase/hamsterbase-highlighter, allowing you to annotate directly in the browser, and it automatically saves a snapshot of the web page when you annotate. When you visit the page again, it automatically displays the previous snapshots.
3. All data is stored on your local device, with both a docker version and a desktop version available. Different versions support P2P synchronization.
4. Provide full-text search function, which can search all the articles on the webpage.
Even though I have been developing for over a year, I still feel that it's not perfect. Currently, no account system has been developed in the software, only a donation website address is left on the official website. Users can leave their email after paying, and I promise to send them activation codes in the future.
The software price is 60, and the income in the recent 30 days is 300 dollars. The total income is 420 dollars.
In terms of PKM applications, believe CRDT is the future。
In response to this, I've developed a "read later" app based on CRDT. The following are my expectations for knowledge management software, as well as the goals I had in mind when designing this application:
1. I want my data to be stored locally so that I can access it anytime and anywhere, thus eliminating the need for expensive hard drives.
2. I hope the software can be purchased for a one-time fee, rather than by subscription. Since all the data is stored locally, this indicates that there are no additional costs for the developers. Therefore, I believe a one-time payment would suffice. At the same time, I'm open to the game DLC model, where enhanced features can be obtained through additional payments.
3. I hope the software can support synchronization so that I can access the most recent data on any device.
You can try the desktop version first, and then consider buying it after the mobile version is released. Until iOS and Android are developed, there will be no charge.
Currently hamsterbase provides rss subscription export, you can take any cell phone rss client subscription.
"Thank you for your reminder. May I ask if you are accessing the website on your mobile phone? The pricing page is located at the top of the official website for desktop version. If you are using the mobile version, you will need to click on the hamburger button.
I have updated the wording to: “Free during the beta period, Provide Believer plan, one-time purchase (60$) with lifetime updates.”
Most "read it later" services save web pages on their servers, which cannot preserve paywalls, ad-free content, pages that require login or registration, and pages on a local network.
Sharing why I decided to develop my own "read it later" software.
1. I have a habit of saving web pages I like, most of which are in MHTML format, some are saved as single-file HTML, and others are web archives saved on iOS. Altogether, I have accumulated thousands of them.
2. On my computer, I can preview them one by one, but I cannot search through them. So, I developed a Node.js service that parses web pages locally and stores them in an SQLite FTS for full-text search. I deployed the service using Docker on my NAS.
3. To enhance my learning experience, I also developed an annotation feature that allows me to make notes and annotations directly on the offline HTML. For good articles, I save them to read slowly over time.
4. Gradually, the app gained some users. Since they were not familiar with Docker, I wrapped it with Electron and developed a standalone desktop version. The desktop version and Docker version use CRDT for peer-to-peer synchronization.
5. Some users provided feedback that it was inconvenient to annotate after saving, so I developed an open-source browser extension. Users can now annotate web pages directly in the browser, and the annotations and snapshots are saved automatically. When visiting the page again in the future, previous annotations can be restored.
Supports self-hosting via Docker.
https://hamsterbase.com/docs/install/install-with-docker.htm...