The Sad State of Personal Knowledgebases(marcusvorwaller.com)
marcusvorwaller.com
The Sad State of Personal Knowledgebases
http://marcusvorwaller.com/blog/2015/12/14/personal-knowledgebases/
108 comments
This. As a near 40 person, I now need to search in 5-10 places for that thing I know I wrote down in Markdown, Vim Notes, Workflowy, Tiddlywiki, 1Password, Confluence, or email at some point in the last 20 years. But maybe we just need to give up on finding one tool to store them all. Instead we need to build more federated search across all these platforms. That way I can have one searcher that returns results all my various old "brains."
I have had very similar thoughts, but I have to wonder if it turned out to be wrong. Jerry's Brain:
http://jerrysbrain.com/
Apparently started in TheBrain in 1997 and now has more than 250K "thoughts". That's breathtaking for both the longevity and the scale. I'm guessing that most mind mapping programs wouldn't handle that well.
Gathering the legacy stuff is not as important as getting started with something, I think. The legacy will make it into a new system as it's needed.
I've been using Evernote for a number of years. I don't particularly like it (cumbersome UI without great navigation), but the search has worked well enough for me to find things and its ubiquity has been great.
http://jerrysbrain.com/
Apparently started in TheBrain in 1997 and now has more than 250K "thoughts". That's breathtaking for both the longevity and the scale. I'm guessing that most mind mapping programs wouldn't handle that well.
Gathering the legacy stuff is not as important as getting started with something, I think. The legacy will make it into a new system as it's needed.
I've been using Evernote for a number of years. I don't particularly like it (cumbersome UI without great navigation), but the search has worked well enough for me to find things and its ubiquity has been great.
Following up to my own point: products come and go all the time and it seems like luck that Jerry happened to pick a product that stuck around. It does seem unfortunate to have to risk your outboard brain on luck!
Totally agree about Evernote. Man did they drop the ball on staying ahead with their UI. Certain things that have been solved in text editors for years are still major issues in Evernote.
That said, I do use it still, and will continue doing so until there is a solid alternative that I don't think will go poof in 3 years.
That said, I do use it still, and will continue doing so until there is a solid alternative that I don't think will go poof in 3 years.
I've been working with own "PK" for about a year now. It's been an interesting experiment so far. My concept is based around random re-exposure to information. For whatever reason, the thought of "Let me look through my old bookmarks" never occurred to me and wasn't part of my daily workflow. So one aspect of my system is that it randomly selects bookmarks from different categories every day and shows them to me. It also has spaced repetition for facts and miscellany, or things I'm trying to learn about (like the US Presidents or Shakespeares plays), vocabulary, etc.
There were a few interesting things I've noticed by randomly viewing old bookmarks. The first is that linkrot is a very real thing and the web is decaying at a measurable rate. Second, it's amazing how limited my memory is. Half the time I don't remember something I bookmarked from years or even months ago at all. It's new to me, all over again. I'm going to bookmark this discussion and then completely forget it existed within a few months at most. Regularly reviewing old web pages and notes makes me feel like my mind is a sieve, as well as highlighting how my interest and thought processes have changed over the years.
I'm really interested in this topic, feel free to contact me to discuss.
There were a few interesting things I've noticed by randomly viewing old bookmarks. The first is that linkrot is a very real thing and the web is decaying at a measurable rate. Second, it's amazing how limited my memory is. Half the time I don't remember something I bookmarked from years or even months ago at all. It's new to me, all over again. I'm going to bookmark this discussion and then completely forget it existed within a few months at most. Regularly reviewing old web pages and notes makes me feel like my mind is a sieve, as well as highlighting how my interest and thought processes have changed over the years.
I'm really interested in this topic, feel free to contact me to discuss.
A friend showed me http://gethibou.com/ yesterday. Basically the idea seems to be that you bookmark a page, highlight what you want to remember and they use spaced-repetition to help you remember it. It might go along well with the PK that you created yourself.
As a Workflowy user, I needed a way to do this and found it in the Colt[1] add-on for firefox. I use the following custom formats to insert tabs and newlines and it produces pretty slick bookmarks in Workflowy's bullet style for me.
Standard bookmark with optional quote via highlighting:
Standard bookmark with optional quote via highlighting:
%T %N %B %U %N %L %N %?[S{"%S"}]
via %U%N%L%N%N%?[S{"%S"}]
#note via %U%N%L%N%N%?[S{"%S"}]%N----%N
The Sad State of Personal Knowledgebases | Hacker News
https://news.ycombinator.com/item?id=10739227
12/18/2015, 10:04:11 AM
"A friend showed me http://gethibou.com/ yesterday. Basically the idea seems to be that you bookmark a page, highlight what you want to remember and they use spaced-repetition to help you remember it. It might go along well with the PK that you created yourself."
Sounds like what I have been doing with Evernote.
I described the system in a previous discussion: https://news.ycombinator.com/item?id=10301205
What you say about review resonates with me. Several years ago I got excited about Spaced repetition software (like Anki, discussed recently on HN) and tried writing an Evernote app that would periodically show me interesting things. Their API didn't work very well for this, so I stuck to doing this manually, and it's some of my most creative and intellectually productive time.
I am contemplating moving to Emacs org-mode for future things, with some sort of conversion process for the accumulated things. That should make a more intelligent system possible.
I described the system in a previous discussion: https://news.ycombinator.com/item?id=10301205
What you say about review resonates with me. Several years ago I got excited about Spaced repetition software (like Anki, discussed recently on HN) and tried writing an Evernote app that would periodically show me interesting things. Their API didn't work very well for this, so I stuck to doing this manually, and it's some of my most creative and intellectually productive time.
I am contemplating moving to Emacs org-mode for future things, with some sort of conversion process for the accumulated things. That should make a more intelligent system possible.
That's an interesting concept. Is the selection of random bookmarks completely random, or does it use a more complex algorithm to choose what to display?
There are simple rules, like "Don't show the same bookmark again within X months", where X becomes progressively longer the more times the bookmark has been shown. I can also flag the best stuff so it shows up more often.
I did my best to group the text of the urls, using first kmeans in php (which is a very stupid idea) and finally settling on LDA. I'd like to use hierarchical LDA to automatically discover topics within my bookmarks, but that's "Future Work" for now.
I did my best to group the text of the urls, using first kmeans in php (which is a very stupid idea) and finally settling on LDA. I'd like to use hierarchical LDA to automatically discover topics within my bookmarks, but that's "Future Work" for now.
Are you planning on releasing/revealing the code at all?
Org-mode. A benevolent beast. Once one gets to learn it, it can be made to do anything. Also having a programming language shared with the runtime and other programmes for that runtime (Elisp and Emacs) is just priceless, it just lends itself to be customized in whichever way one would like. A major con for me, though, is the lack of decent iOS-Android integration.
I've been wanting to learn org-mode for years but I never get around to it. I use something similar, but I only worry that if I were to move to something else I would want it to be in the cloud so I could access it from devices and other laptops. Do you have a solution for this?
It's just plain text files, so you can sync it using whatever you want. Many people use Dropbox.
I have a bunch of org-mode files that I save into a Dropbox folder.
That gives me backups, and access from other computers / laptops.
Search is via ack or grep.
It theoretically also gives me access from my phone using MobileOrg[1], but I have never actually set that up.
[1]: http://orgmode.org/manual/MobileOrg.html
That gives me backups, and access from other computers / laptops.
Search is via ack or grep.
It theoretically also gives me access from my phone using MobileOrg[1], but I have never actually set that up.
[1]: http://orgmode.org/manual/MobileOrg.html
@spatten and @kristiandupont - orgzly is a nice android client. You can sync the org files with dropbox or with git. You could also run a linux locally and install emacs. A keyboard is essential though when you're using emacs, orgzly works ok with touch but is less full-featured. Just as with vim i only use a small subset of what's possible with org-mode: clock in/out, tables, nested outlines, tags and priorities, links to other files and images.
This. Best organizer I've ever used!
I'm a Vim user, but thanks to SpaceMacs/evil-mode I feel right at home. Actually, it works so well that I've come to prefer evil-mode over the real Vim!
http://spacemacs.org/
I'm a Vim user, but thanks to SpaceMacs/evil-mode I feel right at home. Actually, it works so well that I've come to prefer evil-mode over the real Vim!
http://spacemacs.org/
Hey, I found spacemacs a bit intimidating. How do you get started with it? I have plenty of text to write in the coming weeks, so I already have the substrate to practice on. I just need a good introduction to evil-mode+org-mode.
Spacemacs has pretty good docs: http://spacemacs.org/#learning-spacemacs
org-mode and evil-mode alone, without Spacemacs, work pretty well already. It's just additional glue to make it more seamless.
org-mode and evil-mode alone, without Spacemacs, work pretty well already. It's just additional glue to make it more seamless.
Thanks! For some reason I never saw this page before. I wasn't satisfied with the README on Github. Off to read!
My main problem with org-mode is the lack of good display for math notations as well as multimedia (picture/video etc). Did I miss something that I can use with it?
You can display images in-line if you're using the graphical emacs.
AFAIK it's possible to use $$ -- $$ for math notation, it exports to LaTeX/MathJax etc, and there's a setting for making it to preview within the buffer (the function is "org-latex-preview-fragment"). And there is very good latex integration, support for latex blocks etc. I do not know the details tho, I'm not a maths guy.
If you set
If you set
#+OPTIONS: inlineimages
AucTeX is what you're looking for. More ideas: http://kieranhealy.org/resources/emacs-starter-kit/
[deleted]
I haven't gotten around Emacs yet. I'm on a MBP and the Emacs controls are bothersome (to say the least) compared to Vim. I tried using Spacemacs, but it kind of assumes one is already familiar with Emacs and seems a bit too bloated for a newcomer.
So, how do you suggest I acquaint myself with Org-mode? Vanilla Emacs + evil-mode?
So, how do you suggest I acquaint myself with Org-mode? Vanilla Emacs + evil-mode?
These are my suggestions:
Take Emacs as a lisp machine, a working environment, a framework. Then start building it up to your liking. A bit of Emacs lisp knowledge is essential, and it is an easy language, though with rough edges. With org, be experimental and incremental, don't try to create a perfect system from day one, start small and tame it to your habits and tendencies. But get used to the capture+refile flow (see info).
I can't comment about evil, but I do not really feed the need for vi-style editing. Typing speed is not the bottleneck for me. When I type prose, I use Emacs standard bindings + visual line mode. When I program, and nowadays I program exclusively Emacs lisp, and exclusively for my needs, I use paredit. I feed good with these, and I do not feel a need for more speed. But everybody is a bit unique in their needs, so if you'll feel better with evil, go for it.
My last suggestion will be that of avoiding things like Prelude and SpaceMacs. Steal from them, but build your own config yourself. Apart from this, I always use a stable release of Emacs and always check in to git the packages I use, so that I will not have to depend on 3rd parties for my setup. My .emacs.d is simply vital to me.
Edit: about controls, mapping caps lock to ctrl is a life saver, though I do not know well the mac keyboard.
Take Emacs as a lisp machine, a working environment, a framework. Then start building it up to your liking. A bit of Emacs lisp knowledge is essential, and it is an easy language, though with rough edges. With org, be experimental and incremental, don't try to create a perfect system from day one, start small and tame it to your habits and tendencies. But get used to the capture+refile flow (see info).
I can't comment about evil, but I do not really feed the need for vi-style editing. Typing speed is not the bottleneck for me. When I type prose, I use Emacs standard bindings + visual line mode. When I program, and nowadays I program exclusively Emacs lisp, and exclusively for my needs, I use paredit. I feed good with these, and I do not feel a need for more speed. But everybody is a bit unique in their needs, so if you'll feel better with evil, go for it.
My last suggestion will be that of avoiding things like Prelude and SpaceMacs. Steal from them, but build your own config yourself. Apart from this, I always use a stable release of Emacs and always check in to git the packages I use, so that I will not have to depend on 3rd parties for my setup. My .emacs.d is simply vital to me.
Edit: about controls, mapping caps lock to ctrl is a life saver, though I do not know well the mac keyboard.
In my opinion, this is not a "problem" that can be solved. What makes something good varies a lot from one person to the next. In my case, I'd need a way to hold equations, computer code, PDF versions of papers, handwritten notes that someone else has given me, and a lot of other things. I want it stored in a Git repo. I want to view it in a browser. I want it to be convenient and I want it automated. The "answer" is a feature-filled scripting language that allows you to build a customized solution.
What about it would you want automated?
What kind of scripts would you add to your personal knowledgebase?
What kind of scripts would you add to your personal knowledgebase?
It's not so much what I would want automated as what I have already automated. As an example, consider how I handle notes that I leave for myself as I'm working. These are things like paragraphs to be added to a paper, ideas for a research topic, a list of things I have to do around the yard, or anything not directly related to the task at hand.
I have written a script in D (my programming language of choice). I have a web interface to all of the notes I've jotted down in the past. I can click a topic and see all notes related to that topic. I click the new note button, and it opens Emacs with the file that will hold the note. I type in a note in markdown, and as soon as I close Emacs, the note is converted to html. If I want to change it, I hit the edit button, Emacs opens up, and I make my edits. Upon closing Emacs, the html is rebuilt. Anytime I create a new note or edit an existing note, there is a commit to the Git repo.
That's what I mean by automated. I write scripts that handle everything except the content of the notes. I have written or am writing scripts for just about everything I do.
I have written a script in D (my programming language of choice). I have a web interface to all of the notes I've jotted down in the past. I can click a topic and see all notes related to that topic. I click the new note button, and it opens Emacs with the file that will hold the note. I type in a note in markdown, and as soon as I close Emacs, the note is converted to html. If I want to change it, I hit the edit button, Emacs opens up, and I make my edits. Upon closing Emacs, the html is rebuilt. Anytime I create a new note or edit an existing note, there is a commit to the Git repo.
That's what I mean by automated. I write scripts that handle everything except the content of the notes. I have written or am writing scripts for just about everything I do.
How about Camlistore by Brad Fitzpatrick [1]:
"Camlistore is a set of open source formats, protocols, and software for modeling, storing, searching, sharing and synchronizing data in the post-PC era. Data may be files or objects, tweets or 5TB videos, and you can access it via a phone, browser or FUSE filesystem."
[1]: http://camlistore.org/
"Camlistore is a set of open source formats, protocols, and software for modeling, storing, searching, sharing and synchronizing data in the post-PC era. Data may be files or objects, tweets or 5TB videos, and you can access it via a phone, browser or FUSE filesystem."
[1]: http://camlistore.org/
Camlistore is where I'm expecting to go. They've got some basic things modeled in their world, and the rest I'll fill in using inspiration from schema.org.
My plan is to land PDFs of research papers, web bookmarks (and archives of those pages), ebooks, book scans, textfiles/markdown documents, imports of social networks (favorites from Twitter, mostly) and scans of written notes I make to myself in Camlistore. From there, sync to devices and publish to the web as appropriate.
I don't expect that I'll ever find the time to build this all myself, so I'm hopeful that either my data will be in a format that other Camlistore apps can read, or if I get the format wrong I'll be able to write a translator to whatever the Camlistore community decides to model for that specific data type.
My plan is to land PDFs of research papers, web bookmarks (and archives of those pages), ebooks, book scans, textfiles/markdown documents, imports of social networks (favorites from Twitter, mostly) and scans of written notes I make to myself in Camlistore. From there, sync to devices and publish to the web as appropriate.
I don't expect that I'll ever find the time to build this all myself, so I'm hopeful that either my data will be in a format that other Camlistore apps can read, or if I get the format wrong I'll be able to write a translator to whatever the Camlistore community decides to model for that specific data type.
Camlistore is a nice append-orientated sync system, but I wasn't aware it included any data modelling or search?
My saving grace at work has been the tidily wiki (http://www.tiddlywiki.com). It's the easiest place to put one-off shell command line snippets, or instructions or whatever. It's search is fast, and the formatting is nice. The only downside is I have to remember to save and copy the downloaded file to google drive every time I edit anything.
Yeah tiddlywiki is by far my preferred way of storing this type of information. Coupled with a FreeNAS box I can't really think of a better way to store this type information.
I never managed to get tiddlywiki working right - pity, because I really wanted it to work for me, but couldn't seem to get to a stage where I'd trust it to have saved something when it said it had.
This was five years ago at least, mind - I've no idea what the problem might've been, and even if it wasn't user error, I suppose it might've evaporated by now.
(note to self: try tiddlywiki again soon)
This was five years ago at least, mind - I've no idea what the problem might've been, and even if it wasn't user error, I suppose it might've evaporated by now.
(note to self: try tiddlywiki again soon)
I love love love tiddlywiki. I've been using it for probably 6 or 7 years now. It's simply amazing.
Have you tried the firefox save plugin? You could use it to save to a synched google drive so you don't have to think about that.
I have a git repo with notes on all the things I'm interested in, a few larger files (some books I always reference, some images etc.).
Over time I've built up some scripts to help with general tasks, eg list TODOs etc.. It's perfectly moulded to my needs. For example, when I find a new utility I run mk_notes.sh [thing], which sets up three files:
I'm not sure I could buy such a thing, and if I did, I'd get irritated with some aspect of it sooner or later.
Search is easy (grep et al), git means it's available everywhere (for me).
It's literally changed my life, since I now don't 'lose' any research, and can pick up threads later and build.
Over time I've built up some scripts to help with general tasks, eg list TODOs etc.. It's perfectly moulded to my needs. For example, when I find a new utility I run mk_notes.sh [thing], which sets up three files:
- links (for relevant web links)
- thing.txt (which has basic info about it)
- cheat_sheet.txt (which has quick useful things I want to keep)
I'm not sure I could buy such a thing, and if I did, I'd get irritated with some aspect of it sooner or later.
Search is easy (grep et al), git means it's available everywhere (for me).
It's literally changed my life, since I now don't 'lose' any research, and can pick up threads later and build.
That sounds a lot like the org-mode concept.
Interesting; a quick glance at the home page, and it looks like it's for planning. I'm happy using JIRA for that, where I have customized workflows. But again, this is just because I'm familiar with JIRA, not because it's necessarily the best tool.
Also, I'm a vi user :)
Also, I'm a vi user :)
That's not mutually exclusive - see https://news.ycombinator.com/item?id=10739693
Most people use it for planning, but it's also a very nice personal wiki. The great thing is that you can define your tasks inline, if you want.
Most people use it for planning, but it's also a very nice personal wiki. The great thing is that you can define your tasks inline, if you want.
I don't think he's talking about a personal knowledgebase, I think he's talking about something else we are very familiar with. Think about it, he says a PK is "any system that you use to store and retrieve general information Unlimited size... just about everything you want to save for your whole life."
That's a computer, full stop. Or, in software terms, an operating system. In storage terms, a filesystem. These high level tools are designed precisely for the tasks he outlined: Handle data of ever increasing size reliably; make it easy to navigate and search; be simple to use, convenient and fast, and structured.
Where operating systems and filesystems fall down is in searching heterogenous data in a powerful way, and that's where people turn to PKs. Evernote is/was good at searching text and even pictures of text and for tagging; "the Brain" is apparently good at cataloging structured to-do and project type data; Kumu seems to do social graphs; inforapid seems to specialize in process descriptions.
There is no reason operating systems can't get better at processing this data natively. Apple has slouched in that direction in fits and starts with projects like Spotlight and Quick Look. Microsoft once fantasized about building a filesystem from a relational database and has developed an interesting system of "search contracts" for Windows.
That's a computer, full stop. Or, in software terms, an operating system. In storage terms, a filesystem. These high level tools are designed precisely for the tasks he outlined: Handle data of ever increasing size reliably; make it easy to navigate and search; be simple to use, convenient and fast, and structured.
Where operating systems and filesystems fall down is in searching heterogenous data in a powerful way, and that's where people turn to PKs. Evernote is/was good at searching text and even pictures of text and for tagging; "the Brain" is apparently good at cataloging structured to-do and project type data; Kumu seems to do social graphs; inforapid seems to specialize in process descriptions.
There is no reason operating systems can't get better at processing this data natively. Apple has slouched in that direction in fits and starts with projects like Spotlight and Quick Look. Microsoft once fantasized about building a filesystem from a relational database and has developed an interesting system of "search contracts" for Windows.
OP here. I mostly agree with you. If the filesystem was better at a few things then it'd be the ideal PK. Right now though it isn't enough.
It's too slow or too inconvenient for most things you'd want in a PK. Creating files isn't fast or easy. Usually you need to open some other app to do it, even if it's just a terminal. Linking one file to another and showing the connection with metadata like the direction of the link is possible, but not easy, especially if you want multiple links to and from a file. You can sync with things like Dropbox, but search is painfully slow and viewing/editing files means opening a different app. It's just cumbersome.
The reason I like TheBrain is actually because it's good for much more than todos and project data. I have one "file" that has over 10 thousand notes in it. It's organized arbitrarily by my preference, but definitely structured, searchable and "infinitely" expandable. The best part though is that it's really easy to visualize related data in TheBrain. In fact, as I've read comments today, I've come to realize that that is the one crucial thing most PK's are missing. There aren't good visualizations for Evernote, OneNote, most filesystems, org mode, etc.
I"m with you, there's no reason the filesystem can't become the best PK, but it's far from that now.
It's too slow or too inconvenient for most things you'd want in a PK. Creating files isn't fast or easy. Usually you need to open some other app to do it, even if it's just a terminal. Linking one file to another and showing the connection with metadata like the direction of the link is possible, but not easy, especially if you want multiple links to and from a file. You can sync with things like Dropbox, but search is painfully slow and viewing/editing files means opening a different app. It's just cumbersome.
The reason I like TheBrain is actually because it's good for much more than todos and project data. I have one "file" that has over 10 thousand notes in it. It's organized arbitrarily by my preference, but definitely structured, searchable and "infinitely" expandable. The best part though is that it's really easy to visualize related data in TheBrain. In fact, as I've read comments today, I've come to realize that that is the one crucial thing most PK's are missing. There aren't good visualizations for Evernote, OneNote, most filesystems, org mode, etc.
I"m with you, there's no reason the filesystem can't become the best PK, but it's far from that now.
I like Evernote for a lot of personal stuff, though that's one of the "worser" systems OP lists. Atlassian's Confluence does very well at being a wiki and though you're hostage perhaps to your original layout / organization, at $10/year for self-hosted it's a very good deal. It can store and index PDF, Word, Excel, etc., documents very well. I don't think it does handwriting recognition.
If I were to evolve the product category I'd use Confluence as a basis.
In fact I'd want to toy with using Confluence (+ Postgres backend) as the basis for any I.T.-related product -- write plugins for user input, generate all reports to Confluence. I'm not sure what licensing restrictions might be but Confluence provides a lot of useful UI for $10/year (structured + unstructured) for any customer, and building on top of that seems like a no-brainer. Perhaps this is what others do with Drupal and other CMS products, though ... I'm not familiar with that model.
If I were to evolve the product category I'd use Confluence as a basis.
In fact I'd want to toy with using Confluence (+ Postgres backend) as the basis for any I.T.-related product -- write plugins for user input, generate all reports to Confluence. I'm not sure what licensing restrictions might be but Confluence provides a lot of useful UI for $10/year (structured + unstructured) for any customer, and building on top of that seems like a no-brainer. Perhaps this is what others do with Drupal and other CMS products, though ... I'm not familiar with that model.
I'm surprised to read this. Confluence is the most unpleasant "wiki" product (its lack of support for redlinks prevents me from really considering it a wiki) I have ever used. Its built-in hierarchical structure for pages is excessively intrusive and undercooked at the same time, and the search is so bad that I have had it fail to pull up pages when I typed in the literal, complete title.
Like most Atlassian products, its only apparent notable quality to me is that it integrates with other Atlassian products, as well as being a semi-functional platform on which to pile plugins and customization. It's possible that there's a good Confluence install out there, probably administered by the same people who run that really nice JIRA instance that people claim to have used once.
Like most Atlassian products, its only apparent notable quality to me is that it integrates with other Atlassian products, as well as being a semi-functional platform on which to pile plugins and customization. It's possible that there's a good Confluence install out there, probably administered by the same people who run that really nice JIRA instance that people claim to have used once.
Confluence is... okay. I think Atlassian have successfully managed to sand down a lot of the sharp corners on older wiki-type packages that tend to act as a barrier to the non-technical types - from my perspective, that seems to mean that my non-technical colleagues can be persuaded to contribute to documents held in Confluence markedly more easily than to contribute to a MediaWiki instance that we previously had.
So, while Confluence itself as a piece of software makes me grumble more often than not, I'm more than happy to trade that off against the easier sell to others.
So, while Confluence itself as a piece of software makes me grumble more often than not, I'm more than happy to trade that off against the easier sell to others.
Jira is awful too
I've found that using google docs works well if you frame the document title in the form of a question or statement:
- What is a JavaScript constructor?
- making rvm permanent in rvspec
- postgresql listing, creating and dropping a db
Some of the "answers" are just a single line. Then all I have to remember is either something in the title or the gist of the thing I need to know and google docs search provides me with a list of related stuff.
Some of the "answers" are just a single line. Then all I have to remember is either something in the title or the gist of the thing I need to know and google docs search provides me with a list of related stuff.
In my opinion, FreeMind is the best solution currently out there because of forces mindmapping conventions and places a strong emphasis on folding. A lot of other solutions I've seen are more like concept maps, which are great for project management, but not great for knowledge bases.
http://freemind.sourceforge.net/wiki/index.php/Main_Page
(The one thing of note is that there is a bug in the default version of OS X Java, so it frequently crashes unless you update to Java 8 and edit the config to use that version of Java.)
http://freemind.sourceforge.net/wiki/index.php/Main_Page
(The one thing of note is that there is a bug in the default version of OS X Java, so it frequently crashes unless you update to Java 8 and edit the config to use that version of Java.)
Freemind is great for mindmapping in the context of a specific topic or project, but isn't a general purpose knowledgebase.
A mindmap starts at a central node and moves outward. This makes it limited in the practical number of nodes you can have--anything over a couple hundred is hard to manage. It also makes it difficult for one node to appear in multiple places within the mindmap.
A mindmap starts at a central node and moves outward. This makes it limited in the practical number of nodes you can have--anything over a couple hundred is hard to manage. It also makes it difficult for one node to appear in multiple places within the mindmap.
>A lot of other solutions I've seen are more like concept maps, which are great for project management, but not great for knowledge bases.
I've never understood this preference; would you be willing to explain it? It seems almost exactly backward to me: I model tasks as trees of sub-tasks, and knowledge as a general graph of connected concepts.
I've never understood this preference; would you be willing to explain it? It seems almost exactly backward to me: I model tasks as trees of sub-tasks, and knowledge as a general graph of connected concepts.
So all knowledge can be seen as connected, but all knowledge can also be put into hierarchies. Concepts and how those concepts are related is something that people come up with, rather than being an inherent property of the universe.
The idea of forcing yourself to put knowledge into hierarchies is that it makes your thinking more clear, makes it easier to remember where things are, and perhaps paradoxically makes it easier to see how things are related. (Or at least highlights certain types of relationships that are only visible in the context of hierarchies.)
I'll admit it's pretty weird at first to only get one root node, I think it took me a couple years to get used to that. But once I embraced it the benefits quickly became clear.
The idea of forcing yourself to put knowledge into hierarchies is that it makes your thinking more clear, makes it easier to remember where things are, and perhaps paradoxically makes it easier to see how things are related. (Or at least highlights certain types of relationships that are only visible in the context of hierarchies.)
I'll admit it's pretty weird at first to only get one root node, I think it took me a couple years to get used to that. But once I embraced it the benefits quickly became clear.
I'm surprised he does not mention email. This is my personal knowledgebase. It keeps boatloads of receipts and travel reservations from years ago. It is available on all my computing devices. It's Gmail so it's easy to search and I don't have to spend time sorting it. I write notes and send them to myself just so I will have them in the email. Other people are also obsessive about keeping their old email so I can't possibly be alone in this regard.
Gmail nails most of the features on his list.
Gmail nails most of the features on his list.
I set up a filter on my Gmail that automatically tags emails from myself with "Notes". Extremely easy to add a note anywhere -- just email myself from my phone. Easy to search -- just add "in:Notes" to the criteria.
Only thing it doesn't really cover is linking between notes. Though I wonder exactly how valuable that is anyway, when you have a powerful search available.
Only thing it doesn't really cover is linking between notes. Though I wonder exactly how valuable that is anyway, when you have a powerful search available.
Last I checked, Gmail couldn't search the contents of attachments.
That is a pretty show-stopping limitation for a "knowledge base"... have they fixed that?
That is a pretty show-stopping limitation for a "knowledge base"... have they fixed that?
Do you know how to go from a message to the message's "parent" (the message that the message is a reply to)?
I can't figure out how to do that in gmail.
I can't figure out how to do that in gmail.
I was once a warrior on the PK front. I built a PK Chrome extension called Deeper History http://lifehacker.com/deeper-history-searches-the-contents-o.... It compressed and stored pages you visted in IndexDB. You could then search your history by keyword form your url bar.
An example use case was trying to recall a comment of an article you liked. If you could remember a few of the words DH would pull it back up.
I stopped developing it once I realized it was a security nightmare. I built a version that encrypted the data before storing it but that ballooned the DB size and I also didn't have a user-friendly way for users to obtain a key set.
I stopped developing it once I realized it was a security nightmare. I built a version that encrypted the data before storing it but that ballooned the DB size and I also didn't have a user-friendly way for users to obtain a key set.
This is the problem that Dropbox Paper is solving—both for teams and for personal use. I use it now to write all of my blog posts, store research about everyday life things, and keep track of things like doctors, lawyers, etc. I think it solves the problem very well.
Early on they hadn't quite nailed "findability", but with the new (more generic) folders ported from the desktop metaphor and "starred" items this is largely solved. And the search is very fast.
Not only that, but it makes it incredibly easy to share with friends and even have them comment or contribute to individual files, without having to expose everything you write. Learning curve is near-zero.
All in all, super impressed.
Early on they hadn't quite nailed "findability", but with the new (more generic) folders ported from the desktop metaphor and "starred" items this is largely solved. And the search is very fast.
Not only that, but it makes it incredibly easy to share with friends and even have them comment or contribute to individual files, without having to expose everything you write. Learning curve is near-zero.
All in all, super impressed.
Evernote. Free and accessible across many devices. For me it has been best of the worst. It has been my one place to store all information.
Evernote serves well as a general knowledge base, but fails for storing programming snippets and ideas.
Co-founder of Faqt, mentioned in the blog post. The lack of good options is what drove us to create our own solution, with our primary/religious focus being on simplicity. At the risk of self-promotion, we have been quietly beta testing with several organizations a collaborative/team version. If you'd like to try it, there's a free signup at https://app.faqt.co/team/signup
Let me know if anyone has any question: [email protected]
Let me know if anyone has any question: [email protected]
I commented on Discus the site, as well. I've been a long time user of DEVONthink. It can be used for many purposes, including document organizer, email, GTD, tagging, etc. My favorite feature has always been the Magic Hat, which will suggest under what folders a document should be filed. I clip most interesting web pages, and it is surprisingly accurate once you have established a few documents.
http://www.devontechnologies.com
http://www.devontechnologies.com
I agree, it's great. I just wish DEVONthink had a better iOS client.
I've been using nvAlt for a while. Very fast search and ease of use. Not sure how well it scales in the long run but thus far I've had no problems.
Edit to add: Some other interesting features. You can sync to Simplenote automatically, and notes can be saved as text files for easy import/export. There is some markdown support (though I see a number of bugs in the preview). The main downsides appear to be: limited color schemes, no syntax highlighting for markdown, development appears to have stalled.
Edit to add: Some other interesting features. You can sync to Simplenote automatically, and notes can be saved as text files for easy import/export. There is some markdown support (though I see a number of bugs in the preview). The main downsides appear to be: limited color schemes, no syntax highlighting for markdown, development appears to have stalled.
I use Markdown notepads like Quiver [1] and Ulysses [2] to get a "pretty view" on my existing plaintext hoard and directory structure.
This lets my data remain portable and still gives me (almost) all the niceties of proprietary formats/editors.
[1]: http://happenapps.com/#quiver
[2]: http://ulyssesapp.com/mac/
This lets my data remain portable and still gives me (almost) all the niceties of proprietary formats/editors.
[1]: http://happenapps.com/#quiver
[2]: http://ulyssesapp.com/mac/
I just came across this a few days ago on HN:
https://youtu.be/VZQoAKJPbh8?t=2818
https://news.ycombinator.com/item?id=10726536
Eve, the new "programming" tool developed by Chris Granger of Light Table fame, looks absolutely amazing for this use case. I personally can't wait to try it out.
https://youtu.be/VZQoAKJPbh8?t=2818
https://news.ycombinator.com/item?id=10726536
Eve, the new "programming" tool developed by Chris Granger of Light Table fame, looks absolutely amazing for this use case. I personally can't wait to try it out.
I'm inclined not to see this as surprising. There are as many ways in which people conceptualise their 'offline' knowledge as there are people, and there is never going to be one size to fit all.
Org-mode is probably the closest contender, if only for its sheer flexibility - but the fact that it's wed to something (i.e. emacs) that the gen-pop would immediately recoil from is only ever going to limit its adoption to a tiny subset.
Org-mode is probably the closest contender, if only for its sheer flexibility - but the fact that it's wed to something (i.e. emacs) that the gen-pop would immediately recoil from is only ever going to limit its adoption to a tiny subset.
I'm using:
- markdown
- + folders (synced to my mobile)
- + a flat-file cms to wrap it up in the browser (in my case: yellow cms, but there are a few based on markdown and I could switch any time)
Still problematic:
- fast and easy linking
- suggested tags (no body likes tagging manually)
- rich media (especially as you'd ideally want a local mirror, like some research tools do it)
and the general problem:
- text is linear
- folders mostly hierarchic
- information is ribosomal
- ribosomal connections are rather associative and therefore manifold
- manifold structures are a pain in the ass to do manually by linking and tagging
An even better google would be the solution:
- you just dump information (+ meta data to give some more context and spare you explaining the current event surrounding the note)
- and it will be found on demand by AI
maybe in a few years I can feed something my wiki and it will make sense of it
- markdown
- + folders (synced to my mobile)
- + a flat-file cms to wrap it up in the browser (in my case: yellow cms, but there are a few based on markdown and I could switch any time)
Still problematic:
- fast and easy linking
- suggested tags (no body likes tagging manually)
- rich media (especially as you'd ideally want a local mirror, like some research tools do it)
and the general problem:
- text is linear
- folders mostly hierarchic
- information is ribosomal
- ribosomal connections are rather associative and therefore manifold
- manifold structures are a pain in the ass to do manually by linking and tagging
An even better google would be the solution:
- you just dump information (+ meta data to give some more context and spare you explaining the current event surrounding the note)
- and it will be found on demand by AI
maybe in a few years I can feed something my wiki and it will make sense of it
A couple years ago, I decided to setup a MediaWiki installation, and found it works well enough for storing thoughts/ideas/things. I added another sort of 'navigation' layer that makes sense in my head, and wrote a desktop app in python to use my navigation layer instead of the links. Eventually I would like to set it up to scan for links in the various pages, download the content then add it to a job queue to be parsed into a more concise format for the wiki itself.
I don't use it nearly as much as I should... it's one of those things that's 90% what I want, and just missing the last 10%.
I don't use it nearly as much as I should... it's one of those things that's 90% what I want, and just missing the last 10%.
Any reason a bunch of text files in directories wont work?
Any "advanced" text editor that can search directories should work: Notepad++,Vim,grep,... anything really.
I've been using a bunch of synced markdown files. Works for me, I just have a Markdown editor and a "grep-like" app for android. I'm not sure what you can do for iPhone though. When I've talked to people they couldn't seem to "browse for files/folders" which I thought was strange.
The one thing I'm missing is I'd like to "tag" my notes. I haven't decided how/if I'm going to do that yet.
Any "advanced" text editor that can search directories should work: Notepad++,Vim,grep,... anything really.
I've been using a bunch of synced markdown files. Works for me, I just have a Markdown editor and a "grep-like" app for android. I'm not sure what you can do for iPhone though. When I've talked to people they couldn't seem to "browse for files/folders" which I thought was strange.
The one thing I'm missing is I'd like to "tag" my notes. I haven't decided how/if I'm going to do that yet.
One of the main reasons for using a personal knowledgebase is surfacing patterns in what you've saved. If there is no way of visualizing the relationship between notes/nodes outside of a flat list, pattern finding is limited to what you can remember to search for.
Basically the core problem isn't saving and searching for data, it's having great ways to structure and visualize it.
Basically the core problem isn't saving and searching for data, it's having great ways to structure and visualize it.
Good point, unless I know a search term it doesn't work out great.
I've been thinking of switching to some kind of wiki format, like maybe creole. Then finding an app that can understand links between files or something.
I've been thinking of switching to some kind of wiki format, like maybe creole. Then finding an app that can understand links between files or something.
I use this [1] to do a simple full-text search of my personal knowledge base.
[1] https://github.com/slashdotdash/jekyll-lunr-js-search
[1] https://github.com/slashdotdash/jekyll-lunr-js-search
Amen. I currently use txt files & Tree 2 (mac app).
I’ve tried Quiver, Quip, & Workflowy and cannot find the perfect app to suit this space for my usage patterns. I’m almost 100% back to text files & sublime text.
I’ve tried Quiver, Quip, & Workflowy and cannot find the perfect app to suit this space for my usage patterns. I’m almost 100% back to text files & sublime text.
The problem is really that every time something new and better comes around, it becomes a mess to port everything over, especially if it is conceptually different (e.g. tag-based or something else). Dying products also seldom give you a convenient means to export your content. So ultimately the only thing that can last for decades, be cross-platform, and not be tied to one company's product, is text files and directories. For now, throw it all on Dropbox, Github, or whatever you like. When Dropbox and Github die, the file-directory structure will be easy to port to whatever is next.
This has been my current approach for a while, after exporting everything out of Tiddlywiki when it looked like that was dying for a while (Browser support drying up, etc).
Now that it's been revived with Tiddlywiki5 and development's picked back up, I'm somewhat tempted to get back into it. Main advantage: Better crosslinking, rich content, etc. Main disadvantage: Less simple to edit, some tedium to export.
I'd love to have a Tiddlywiki-like frontend to a set of Markdown text documents backing it. (This is something that's been gnawing at the back of my mind long enough that I might have to buckle down and make it myself) Maybe Markdown + JSON for rich content data...
Now that it's been revived with Tiddlywiki5 and development's picked back up, I'm somewhat tempted to get back into it. Main advantage: Better crosslinking, rich content, etc. Main disadvantage: Less simple to edit, some tedium to export.
I'd love to have a Tiddlywiki-like frontend to a set of Markdown text documents backing it. (This is something that's been gnawing at the back of my mind long enough that I might have to buckle down and make it myself) Maybe Markdown + JSON for rich content data...
Where I need markup, I've been sticking to HTML or LaTeX syntax for the most part.
Partly because I hate Markdown due to its inconsistent syntax (that's my personal opinion), but also because it feels like a fad of recent years, much like a successor of the MediaWiki fad and the BBcode fad. I feel like in another 2 or 3 years someone will invent Markleft or Markright or Visual Markdown#.NET 2.0 or Dustindown that's 10 times better than Markdown, and then I'll have to switch everything over yet again.
HTML and LaTeX on the other hand, aren't dying any time soon due to their extensive deployment, and I'm confident that my syntax will be renderable for at least another 10 years or more.
Partly because I hate Markdown due to its inconsistent syntax (that's my personal opinion), but also because it feels like a fad of recent years, much like a successor of the MediaWiki fad and the BBcode fad. I feel like in another 2 or 3 years someone will invent Markleft or Markright or Visual Markdown#.NET 2.0 or Dustindown that's 10 times better than Markdown, and then I'll have to switch everything over yet again.
HTML and LaTeX on the other hand, aren't dying any time soon due to their extensive deployment, and I'm confident that my syntax will be renderable for at least another 10 years or more.
i'm not sure you'd consider it better markdown (i do), but do you know about reStructuredText? it seems to even predate markdown. http://docutils.sourceforge.net/docs/user/rst/quickstart.htm...
Emacs and orgmode are probably your next step :)
orgmode is really a superset of markdown with PIM markup added.
It exports to markdown and a ton of other formats.
Cheers.
orgmode is really a superset of markdown with PIM markup added.
It exports to markdown and a ton of other formats.
Cheers.
Anyone Remember Neonem on Windows? Well, that was really good for organising personal info in a structured way. Discussion here seems too broad: comparing Evernote (which is basically losely structured), wiki's etc to real classic outliners that once were thriving on Windows in he old days but kind of have of have been forgotten the past years. If you search for "outliner" you'll find some good all abandoned Windows projects.
I, too, am an org-mode fan, but there are a few good ones missing:
- Zim (http://zim-wiki.org/, Open Source)
- WikidPad (http://wikidpad.sourceforge.net/, Open Source)
- ConnectedText (http://www.connectedtext.com/, Windows-only, $$$)
- Zim (http://zim-wiki.org/, Open Source)
- WikidPad (http://wikidpad.sourceforge.net/, Open Source)
- ConnectedText (http://www.connectedtext.com/, Windows-only, $$$)
Quiver in my iCloud is the best thing that I've found so far. One of the biggest things for me is support for code snippets /highlighting, without having to use something to convert to RTF in-between copy and paste. Sadly, I stopped using it since I now have to log in to my company iCloud and Apple accounts at work, and it's just too much of a hassle to switch back and forth. I mainly use OSX notes and Textmate now.
[deleted]
Chris Granger's latest experiments with Eve (github.com/witheve) look surprisingly like a semi-structured personal knowledge system.
Take a look at the last 15 minutes or so of this video for a demo:
https://www.youtube.com/watch?v=VZQoAKJPbh8
(the whole video is interesting for the evolution of it)
Take a look at the last 15 minutes or so of this video for a demo:
https://www.youtube.com/watch?v=VZQoAKJPbh8
(the whole video is interesting for the evolution of it)
An automated tool for capturing and sharing knowledge will truly make the world a better place. Learning can be faster , reducing the cost of education. People across the globe can connect based on topics of interest and knowledge, yielding more economic and job opportunities.
A personal knowledge base is far more valuable than what it might seem at first glance. Very happy to see you have articulated clearly and made people realize it. A PKB has great potential for knowledge transfer within organization or even at research labs for new people to familiarize with literature. Many people want to capture their thoughts for writing a book someday, but current tools aren't good enough for this. Sadly, a person's social graph is well captured today rather than their knowledge graph.
With these thoughts, I started implementing Hyperbook - www.getbook.co few months back. Currently it is in beta and really working hard to improve it every day. We got good feedback and reviews from thought leaders in content curation and knowledge management industry. Thanks for bringing the problem for discussion. It gives me motivation that is very much a worthy problem to solve.
A personal knowledge base is far more valuable than what it might seem at first glance. Very happy to see you have articulated clearly and made people realize it. A PKB has great potential for knowledge transfer within organization or even at research labs for new people to familiarize with literature. Many people want to capture their thoughts for writing a book someday, but current tools aren't good enough for this. Sadly, a person's social graph is well captured today rather than their knowledge graph.
With these thoughts, I started implementing Hyperbook - www.getbook.co few months back. Currently it is in beta and really working hard to improve it every day. We got good feedback and reviews from thought leaders in content curation and knowledge management industry. Thanks for bringing the problem for discussion. It gives me motivation that is very much a worthy problem to solve.
I've gotta say that org-mode and a git repo do pretty well by me for this kind of thing.
I use Microsoft's OneNote for this. The search works amazingly well and is fast.
I've been tempted by this, but I have some reservations:
1. Does it work under less-used OSs? Linux/BSD..?
2. Is it a proprietary format? Will you be able to open the file in a decade? My text files will be able to be opened, I'm not sure anyone can still open their MSWorks files.
Maybe I'm crazy to worry about opening these files after a decade since using them probably isn't worth while unless you keep up with it. However, I've got files from the 90's that I've been transitioning from machine-to-machine since storage costs keeps getting cheaper.
1. Does it work under less-used OSs? Linux/BSD..?
2. Is it a proprietary format? Will you be able to open the file in a decade? My text files will be able to be opened, I'm not sure anyone can still open their MSWorks files.
Maybe I'm crazy to worry about opening these files after a decade since using them probably isn't worth while unless you keep up with it. However, I've got files from the 90's that I've been transitioning from machine-to-machine since storage costs keeps getting cheaper.
No, and yes. It's proprietary and Windows/Android/iOS-only.
That's false, you can download documentation for the OneNote file format here
https://msdn.microsoft.com/en-us/library/dd924743(v=office.1...
it is only about 5% of the size of the Adobe PDF specification and I am sure there are corner cases, but it is not too bad.
https://msdn.microsoft.com/en-us/library/dd924743(v=office.1...
it is only about 5% of the size of the Adobe PDF specification and I am sure there are corner cases, but it is not too bad.
I admittedly did not know that they published a specification . It's still a proprietary file format though. Proprietary does not mean "binary and undocumented", it means that it's not an open standard.
Toss Mac in there as well:
https://itunes.apple.com/us/app/microsoft-onenote/id78480155...
https://itunes.apple.com/us/app/microsoft-onenote/id78480155...
Word can still open Works files to this day.
I've been using Zim (http://zim-wiki.org) for several years, backed up to Dropbox. It's a significant improvement over plain text files.
Does anyone have a system for managing their relationships? Kind of like CRM but on a personal level. Keeping track of people's details - like their family, their passions, their history etc.
Knowledge can be captured in wikis, notes, documents, videos, audio, and even photos. Our smartphone/smartwatch/web apps allow you to access any of the knowledge using a single, easy to use interface - voice. Basically, you tag content with a voice command (e.g. "to do list" or "show me the video on how to do xyz") then, whenever you need to retrieve the knowledge, just speak the command. Voice commands can be shared and the apps are all free.
https://youtube.com/treycent/search?query=notes
https://youtube.com/treycent/search?query=notes
I used to use Delicious but that didn't solve the "search the content of my bookmarks" problem for me.
I've been using Pocket for now and it seems to work fine.
I've been using Pocket for now and it seems to work fine.
imho the best one ever made is called getguru.com. It's brilliant actually but the team there has priced themselves out of the market charging $7/head as a b2b app because they're from and enterprise sales background and therefor don't realize guru's monster potential in consumer-web. They're happy taking the low hanging fruit $$ from call-centers.
sigh
sigh
I like trunknotes (http://www.trunknotes.com/)
Confluence for the simple win
several instances of PmWiki + Vimperator, being using that for nearly 10 years and thousands of edits so far
I've tried a few and at least twice tried to write my own (and gave up due to scope creep). Right now I just have a bunch of Markdown files and a giant messy DropBox folder and an almost-as-messy set of private GitHub repos. And photos. And videos. And e-mail. And backups. (What does one do with a Zip disk these days?)
I bet a lot of people my age (mid-40's) or older have the same problem: choosing or building a system we think would work long-term, and then gathering and organizing all the "legacy" stuff (which you definitely want!) is already an enormous undertaking, and so instead we live with a bunch of ad-hoc systems that aren't in any way cross-referenced.
Nobody wants to be the sysadmin of their own junk drawer, but I for one don't want years' worth of my stuff locked into somebody's Cloud du Jour.
(Then again, I have this same problem with actual paper documents, so maybe it's just me.)