Yeah. Me too. The css selector is quite convenient. The only problem is that I need to pick the selector set for each website I need to scrape, and there are tens of them, which makes the work itself time-consuming...
Thanks. I used beautiful soup for the parser, and actually have written a crawler framework for my scenario. But I was wondering whether there is any tool that could automate the selection of css selector or xpath.
I was using mechanizer + beautiful soup in python before, but it seems that this one also needs human to read the html source code to pick a css selector instead of automating it...