The Easy Way to Extract Useful Text from Arbitrary HTML(ai-depot.com)
ai-depot.com
The Easy Way to Extract Useful Text from Arbitrary HTML
http://ai-depot.com/articles/the-easy-way-to-extract-useful-text-from-arbitrary-html/
5 comments
Anyone else getting an error page about this site being reported to be an "attack website"?
Only Google has it shit-listed, therefore Firefox does too by extension.
Opera 9.5 and IE 7 (both w/ phishing detection as well) don't flag it. Google's malware detection engine has been known to be off plenty of times before - lost a startup I know a month's income because some bad sites were linking to them, before they apologized and undid their blacklisting.
Opera 9.5 and IE 7 (both w/ phishing detection as well) don't flag it. Google's malware detection engine has been known to be off plenty of times before - lost a startup I know a month's income because some bad sites were linking to them, before they apologized and undid their blacklisting.
Of course it's an "attack website." It'll strip away your HTML, and then what will you have?
Google SafeBrowsing has it flagged: http://safebrowsing.clients.google.com/safebrowsing/diagnost...
I think it's blacklisted by google but it appears to be perfectly safe
Most attack sites do. Appear to be perfectly safe, that is.
For anyone who wants to read the article, this works. (I put in URL_HERE as a placeholder because HN truncates the URL):
lynx --dump URL_HERE > output.txt
Ironic that to read the article, I used a technique that solves the problem the article is talking about solving. In a much easier way, I might add!
For anyone who wants to read the article, this works. (I put in URL_HERE as a placeholder because HN truncates the URL):
lynx --dump URL_HERE > output.txt
Ironic that to read the article, I used a technique that solves the problem the article is talking about solving. In a much easier way, I might add!
The patterns could be formed in a more formalized way by applying Shannon's principles of information entropy across the document, lines of text, word patterns, N-Grams, etc. Then Bayesian inference can be applied for probabilistic pattern matching (vs the neural network).
Not that this approach is any easier, its just perhaps more robust for applying to other problem sets.
Nothing new of course, according to their website, Autonomy (Europe's 2nd largest software company) uses these techniques as the basis for their core technology to analyze text, audio, video, etc.
Not that this approach is any easier, its just perhaps more robust for applying to other problem sets.
Nothing new of course, according to their website, Autonomy (Europe's 2nd largest software company) uses these techniques as the basis for their core technology to analyze text, audio, video, etc.
It's an interesting idea and probably useful in some situations. It would be much more useful, though, if the parser kept structural information about where in the html tree a particular text fragment was found. Lines could still serve as the unit to which statistical analysis is applied (although that seems error prone), but knowing more about the structure would enable further processing down the line.
[deleted]
You can get pretty good results just by checking if the line’s density is above a fixed threshold (or the average), but the system makes fewer mistakes if you use machine learning — not to mention that it’s easier to implement!
for whose who too scared to follow link...