So here's the deal. I'm creating a web scraping tool which can be used to scrape whatever domain you point it.
It's goal is to be able to do an unsupervized web scraping of any website. Regardless of the website layout which differ domain to domain, it is able to survive page layout changes and does not need a proxy list as it constantly refreshes IP addresses.
It's also a web based URL api so for example one could potentially do this in the terminal or anywhere you can do a GET request (probably will be a POST with authentication token in the future).
wget http://scrape.it/scrape/seedydvdstore.com/"grab title, price, ratings, comments" alldvds.csv would result in alldvds.csv file on your file system containing those said column fields with thousands of rows obtained from seedydvdstore.com with navigating pagination links.
What do you guys think? Is this something you would sign up for?
I'm just trying to get the word out there early in the game so I can stay motivated and focus in the coming weeks to release a beta.
Thank you, and to email subscribers I will post some screenshots in the coming weeks, and videos demonstrating how it works.
It's goal is to be able to do an unsupervized web scraping of any website. Regardless of the website layout which differ domain to domain, it is able to survive page layout changes and does not need a proxy list as it constantly refreshes IP addresses.
It's also a web based URL api so for example one could potentially do this in the terminal or anywhere you can do a GET request (probably will be a POST with authentication token in the future).
wget http://scrape.it/scrape/seedydvdstore.com/"grab title, price, ratings, comments" alldvds.csv would result in alldvds.csv file on your file system containing those said column fields with thousands of rows obtained from seedydvdstore.com with navigating pagination links.
What do you guys think? Is this something you would sign up for?
I'm just trying to get the word out there early in the game so I can stay motivated and focus in the coming weeks to release a beta.
Thank you, and to email subscribers I will post some screenshots in the coming weeks, and videos demonstrating how it works.
Cheers,
umphetico.