Ask HN: Could you suggest a fast library for converting documents into a sparse matrix representation (e.g., COO or CSR) in any programming language? I'm guessing C beats most of the implementation? But there is also the issue of efficient n-gram hashing/indexing.
Many points raised here are valid concerns and should be addressed by the data scientists when designing the system.
Let's just not forget about the positives here: there is a company that openly uses term "A.I." and raised a lot of money. Am I the only one here waiting for the AI Winter to be over [1]? Maybe it's not the best A.I., yet, but they got the funding. Personally, I think that's great and I hope for others to come out of the closet.
Brendangregg, do you know if anyone tried to use this platform to predict failures or DDOS attacks ahead of time? Or it is not feasible with this API? Thanks.
To Ryan Kirkman and Thomas Davis: could you make some of the data freely available to the Natural Language Processing community?
If this is something that interests people, then it will be worth the time to automate this process. I suspect that there are blog entries, twitts, facebook updates that already answer these questions. Now we have to mine them. "Gold standard" set of Q/A would make it easier.
EDIT: But I hope there will be a way of getting Q/A in a bulk, right? Thanks
NOTE TO SELF: This could be a data mining or text mining web app. Check if the interest grows in time.
I think no one is like Ed. There maybe be 25% of programmers that have a long standing experience, even with the new tools, but they don't share wisdom like Ed does.
Ed why can't you have your own company? Why can't you be the one who is hiring or at least teach others how to hire. I bet you can smell a talent from 1000 miles away - even if someone did not finish a good school; the world of software engineering would make so much more sense.
[EDIT:] I imagine a job interview; an interviewee says: "I didn't program this but I did program something more difficult"; interviewer makes a frowny face; Ed comes round the corner, puts his hand on the interviewer shoulder and says: "He did not call you stupid; he just said he is ready for the challenge - he has the experience".
S.A. is talking about general-purpose AI (position 2 in the RFS). This means processing natural language. There is a lot of progress but it's just slow so it's almost invisible.
Also it's a very difficult field of science. Now you need to be proficient in AI, machine learning, computational linguistics, linguistic corpora research, cognitive sciences, statistics, and sometimes physics if the text changes over time. Of course, you also need to be a good programmer. This combination of skills is very rare. Thus, very slow progress.
I suggest to start with well defined practical problems. For example, no one seems to do much with user generated reviews. There is some sentiment analysis but that is just a binary text categorization problem - not even close to general purpose AI.
It would be much more interesting to show a seller a time ordered stream of clustered reviews that depict only the most representative review for each cluster. This way a seller can see how his/her fixes/changes impact user reviews. Also it would be a great source for features and bug fixes requests. This is an ideal testing bed for clustering, novelty detection, categorization and mild inference. The inference is required because of sparseness of data.
This would create a good data set for a more general purpose AI. We would have reviews and text documenting changes and improvements of a new version of a product. Now the computer could start learning the dialog between users and product developers. Then, we are just one more step from statistical inference based question-answering system. Not a brute force system like "Watson" or a hand crafted rule base system like "Siri".
[EDIT:] I was thinking more about a decision support system that can recommend product changes. But in a way that maximizes customer satisfaction and minimizes the cost of implementation. The dialogue between past changes and customer reaction would give us the surface that needs to be optimized. This would generalize well to other domains where there is a text for request and a text for response - just to name one: clinical text in healthcare (position 5 in the RFS).
Maybe it's also good idea to revisit the reason why these systems are not widely in use. They all operate under so called "closed-world" assumptions. It means that the knowledge about the world is very limited.
Once the limitation is lifted the system has to deal with non-monotonic reasoning (http://en.wikipedia.org/wiki/Non-monotonic_logic) and that leads to multiple inheritance problem (http://en.wikipedia.org/wiki/Multiple_inheritance). Unfortunately, the multiple inheritance problem has, so far, only NP-hard solutions (http://ijcai.org/Past%20Proceedings/IJCAI-89-VOL-2/PDF/047.p...). [EDIT: think of a command "eggs are in a fridge"; to find eggs in a fridge you need to know that they are inside a container; you need to know its shape and how to open it; if your software knows that and inherits this knowledge each time when asked for eggs, then it will brake when the eggs are not in the box.]
We are in need of software that deals with large networks of human knowledge. Then we can take SHRDLU to the next level. Otherwise we are stuck with Will Smith movie (http://www.imdb.com/title/tt0343818/).
[EDIT2: there are many theoretical "suggestions" but very little software or practical internet applications]
This should help you keep your strengths intact regardless of your current situations ... who is to say that next endeavor will be better than the previous one.
Here is a suicide prevention lifeline for those in US:
1-800-273-8255
[EDIT: suicide is a very complex disease that is far from being understood. I'm sure that prof. Yoshiki Sasai had to deal with much, much more than just two retracted articles.]
I would say that this is true for unstructured data that does not have availability of experts who participated in the creation of that data.
I noticed that by talking with the domain experts, for example radiologists that read head MRIs, you can greatly simplify the problem. It usually comes at the cost of experts annotating the data, which might be too expensive for some businesses. But then, I think, the data is analyzed just by scholars until they built-in the expert knowledge into their algorithms.
That's at least the pattern in healthcare data. The first one to notice a paper or source code like that dominates the market - this is just my conjecture.
I agree. What the internet needs is semantics not regular expressions. If there is a need for regular expression, it's only because semantics is not working, for example, the search engine does not have all the synonyms so I might write "do|does|done". We should be focusing more on textual query expansion (if the query hides some background knowledge), auto-tagging of web pages (if the web pages assume some background knowledge) and clever disambiguation of queries (if we don't know which background knowledge tap into).
You might be onto something. Could this be easily converted into an annotation tool for machine learning purposes? It looks like you could use it to annotate images and text. You need an interface to create annotation schema. Then, you need to have two or more annotators per text/image. And finally, you calculate inter-annotator agreement and export the data.
I know that there are couple startups that got seed money for "annotating the web". For example "Rap Genius".
Justin, I like your idea behind "circl.es": people already are generating a lot of data, there is no need to ask for more.
If you need help with your data just email me.
Note to self: data mining web app, it seems to use simple a similarity measure, same idea can be used for recommendation engine [Twitter + Facebook + Pinterest, etc ... to get good vector representation for each person, then clustering (probably non-convex variable density or not), and assigning recommended items from a cluster to all people in that cluster]) - product recommendation is not much different than dating
It means that our ranking algorithms have good recall but very poor precision. We value web page connectivity more than its content. We don't know how to teach machines to evaluate web page for its merit so we hope that a large number of Twitting, Liking and Plusing non-experts will approximate single expert. Millionshort shows that this model isn't good enough.
You are welcome. I've studied what makes a story be picked up. It's mostly time of the day and the name of the submitter. Here are my partial findings:
Note to self: Leanpub.com disrupts book publishing industry, it took them just ~ 11 months (EDIT: 2 yrs - look below) to get traction. Surprisingly, their original post got just TWO karma points:
