Please Solve the Information Redundancy Problem(codeismightier.com)
codeismightier.com
Please Solve the Information Redundancy Problem
http://codeismightier.com/2009/02/solving-the-information-redundancy-problem/
3 comments
I have implemented such an algorithm in the past. The problem is that it actually makes things even more redundant since it gives you stuff you have already read and liked. The results returned are usually articles on the exact same topic from a different source.
I don't see redundancy as much of a problem for a couple reasons:
#1 If you teach yourself to scan text well, your brain becomes very good at parsing the data that is new. I can get the gist of a one-page article in 10-15 seconds if it contains only a few paragraphs of truly "New" information.
#2 Redundancy cements important knowledge in your mind. I am willing to bet that if you examined any 'designed' course of study such as you would find in college, you will see a great deal of redundancy. Studies have shown that spaced repetition and review can greatly impact how well you retain a certain body of knowledge. If you are reading articles on the same subject in an attempt to learn a body of knowledge there would be some benefit to the overlap, especially considering that any new nuggets of info would be learned in context and cemented in place next to already known facts.
#1 If you teach yourself to scan text well, your brain becomes very good at parsing the data that is new. I can get the gist of a one-page article in 10-15 seconds if it contains only a few paragraphs of truly "New" information.
#2 Redundancy cements important knowledge in your mind. I am willing to bet that if you examined any 'designed' course of study such as you would find in college, you will see a great deal of redundancy. Studies have shown that spaced repetition and review can greatly impact how well you retain a certain body of knowledge. If you are reading articles on the same subject in an attempt to learn a body of knowledge there would be some benefit to the overlap, especially considering that any new nuggets of info would be learned in context and cemented in place next to already known facts.
It wouldn't be that difficult to implement this. Run your RSS feeds through a program that will look in the articles for words you are already familiar with.
If you read an article, you can rate it good or bad and it can look at which terms are /in/ the article. If you rated it good, then the new terms are added to a list of terms to look for, terms that the algorithm will think are part of articles that will interest you.
If you rate it bad, it'll look at terms in common with other articles you've rated bad and see know what to look for in articles it should not bring to your attention.
It would be nice if such an algorithm would allow you to rate the articles on different scales for entertainment, technology, philosophy, etc, so when you are in one of those moods, you just click on the category and it'll hunt for things that may interest you.
I think an algorithm like this would be much more suited to an individual than a social group type filtering system like most groups use. At some point, the individual wants to break off from the group and explore independently and to do that effectively, they need a teacher.
In a way, this algorithm would be a teacher constantly showing you new things in the world and keeping your interest up.
The problem with social sites now is that there are many large communities who all at once look for a new venue for their point seeking. It promotes a king of the hill mentality where individuals within the group compete with others for points rather than personal excellence, education, and betterment.