Ask HN: Has anyone used web-scraping-as-a-service that didn't suck?
8 comments
Give Feedity - https://feedity.com a try. Drop us a message https://feedity.com/contact.aspx if you have any questions or concerns.
I've used http://www.mozenda.com in the past. Worked on some fairly complex Web directories with multiple levels.
work with a group that use Mozenda - I haven't worked much with it directly. They made it through a couple years of scraping with Mozenda. They recently decided to spend more and installed Kapow. More mature platform - lots of features but also a lot more expensive if I understand correctly.
it depends, tools like Kimono can gather data on a topical layer, but cant really interact with the page(via js). I wanted to scrape some data on a site I was interested in, but I need to interact with the site to get the data, so I ended up writing my own crawler
I got great experiences with http://scrapinghub.com/, you can hire them as consultants also and short time contract is enough to get in the right direction. really smart people
I heard good things about https://import.io/
My colleagues at https://www.archive-it.org do a pretty darn good job
Seconding this question...
I've wanted a way to get HN top posts each hour, aggregated (and deduped) by day/week, so that I dont have to feel the need to check as often.
Thanks for this links, hopefully one of these will get me there!
You don't have to scrape HN, it has an API. It's linked to at the bottom of each page.
There seems to be a large number of web-scraping-as-a-service products out there, and very little to recommend between them. Has anyone used one which didn't suck, or am I likely to need to write my own scraping tools?