hi all, I've got something like 18000 websites I need to visit and extract some info from but every single one has a different organisation, there is no way I can make up rules unless I make 18000 of them or I spent weeks coming up with some sort of AI stuff.
At the same time I'm not an expert in the fields hence my question: would you got for manual scraping or you are aware of very sophisticated tools that I could try out? I'm aware of the well-known OSS ones (Weka, Scrapy, Beautiful Soup) but unless I'm missing something they fall into the "write as many rules as different websites you've got" category.
At the same time I'm not an expert in the fields hence my question: would you got for manual scraping or you are aware of very sophisticated tools that I could try out? I'm aware of the well-known OSS ones (Weka, Scrapy, Beautiful Soup) but unless I'm missing something they fall into the "write as many rules as different websites you've got" category.
TA!