For a quick, one-time job, it works very well, and you don't have to bother configuring SSH/FTP for it :). Of course, one could build a lot on top of DRb, I just wanted to show how easy it would be :)
Most of the coders are used to work solely with an IDE, without ever knowing how things work in the background. I know this is something basic for people that know how an IDE works, but most of them don't.
It is recommended that you use the API that google provides for searching, but I fail to see why telnetting would be illegal. After all, both firefox and telnet use sockets to do their job.
Yes, I have worked with mechanize before. I was using mechanize even when there only was the perl version. I added a comment to the article explaining my choice.
The example wasn't written in Java for performance gain. It so happened that I had NetBeans open , and it was easier for me to write it in Java at the moment :).
More and more websites are adding Turing tests after relatively short intervals of time. I consider that to be a good thing, and there are some good ways to stop/identify spiders. You can check out this site: http://stackoverflow.com/questions/450835/how-do-you-stop-sc... . It shows some pretty good techniques that one can use to defend from spiders.