Querying Google/a search engine through my webapp?

7 pointsby schtog18년 전8 comments

Googling "query searchengine" and the like doesnt give a good answer so i try here.

Searchengines generally do a good job of sorting out relevant data/links but they rarely ever take you all the way to the goal.

So I have some different webapps I'm working on and I would like to query Google or some other searchengine for a subject and then go through those links and present the relevant information.

I could obv write something to do this, but is it allowed? Or I have to pay for this service at Google? If so, is there some automated way to do this?

7 comments

simonw18년 전

Yahoo! have a great search API: http://developer.yahoo.com/search/

Google quietly re-introduced their search API a few months ago under the guise of the "Ajax Search API that doesn't require JavaScript" - it requires you to provide a referrer which is a bit weird, but it should work OK: http://code.google.com/apis/ajaxsearch/documentation/#fonje

schtog18년 전

Went with Yahoo's, you are right, it is very easy to use.

Just get a Yahoo-ID and download the API. The following code searches for Zlatan Ibrahimovic and prints the top 40 results:

  from yahoo.search.web import WebSearch
 
  srch = WebSearch(app_id="YahooDemo", query = "Zlatan  Ibrahimovic", results = 9)

  for res in srch.parse_results():
     print res.Url

schtog18년 전

sorry top 9 results obv.

bluelu18년 전

Google doesn't allow automated queries. An old soap api key might still work. (At least it works for the former company I worked for, but I think they also had a special deal with google)

FiReaNG3L18년 전

Once upon a time there was a Google API to do that, but its not available anymore. And im pretty sure scraping their result pages is against the TOS and will get you banned pretty fast.

[deleted]18년 전

aneesh18년 전

There is a perl module that you can use to search Google: http://search.cpan.org/~bstilwell/Net-Google-1.0.1/lib/Net/G...

I'd imagine other languages have similar libraries.

kp21218년 전

I wrote a page scraper for google some time back. Its actually easier to parse their pages than yahoo's imo. You will get blocked for hours if you intend to post hundreds of times a minute though.

alexstaubo18년 전

Seconding simonw's Yahoo recommendation. The API is great and the results are often as good as Google's. If you scrape Google searches, they will eventually block you.