Ask HN: Web Scraping with Node.js
6 comments
I found https://github.com/cgiffard/node-simplecrawler to be good, I used it for https://github.com/tmpfs/linkdown - a cli for validating web pages etc.
I haven't used this for web scraping, but have you tried out http://phantomjs.org/. It's a great tool that lets your process webpages; I don't think it would be that difficult to set it up to scrape sites.
Here is a place to start:
https://github.com/lorien/awesome-web-scraping/blob/master/j...
You might want to check out the Nightmare project (https://github.com/segmentio/nightmare) which makes automating an Electron instance pretty simple.
You might want to check out the Nightmare project (https://github.com/segmentio/nightmare) which makes automating an Electron instance pretty simple.
Should it be necessarily with Node.js? I've tried to find a good toolset in JS, but it seems other languages (e.g. Python and Java) are a lot more mature in scraping.
Try this out https://github.com/IonicaBizau/scrape-it
I have the intuition that my Google searches are a bit inadequate. Thank you!