Why you should use curl and firebug to scrape(nachivpn.blogspot.com)

nachivpn.blogspot.com

Why you should use curl and firebug to scrape

http://nachivpn.blogspot.com/2015/05/why-you-should-use-curl-firebug-to.html

1 comments

Newbie here. cURL is a command line tool, firebug is a browser tool. So how can I use the two tools at the same time?

HTML of a page is received in response to a HTTP request. A page load might involve multiple HTTP requests. Often parts of a web page that interest you might be loaded specifically by a request. By using firebug, you could examine which HTTP request fetched you the content you need and what parameters (headers, cookies, etc) where associated with it. Using this information, you could simulate a HTTP request of your own using cURL (in fact, firebug provides a "copy as cURL command" feature which can be very useful). Firebug is used only in the initial stages to understand. Post which, cURL can be used in your shell script (or a suitable language of choice) to implement your bot.