There are a few things that I hope you just left out in your description...
- There is no mention of implementing a crawl-delay. You should always wait for several seconds (better yet, a minute) between requests to the same host.
- Do you follow redirects when requesting the robots.txt? You should! Some sites send you a redirect to a different URL even for robots.txt. In most cases it is just a slightly different hostname, like www.domain.com instead of domain.com. But it can redirect you to somewhere completely different in some cases.
- You probably don't want to crawl anything that ends with .jpg, .gif, and definitely not something like .avi, .wmv or .mkv. There are a LOT more file-extensions that you'll want to ignore.
I agree with cmiles74 that using a database is probably a bad idea. For a sizeable crawl (say a billion pages) this database will get pretty damn big. I doubt that you will be able to get decent performance out of anything with "SQL" in its name for such a use-case, unless you throw a ton of hardware at it. Building your own specialized solution for this would probably be a lot faster and less resource-intensive.
- There is no mention of implementing a crawl-delay. You should always wait for several seconds (better yet, a minute) between requests to the same host.
- Do you follow redirects when requesting the robots.txt? You should! Some sites send you a redirect to a different URL even for robots.txt. In most cases it is just a slightly different hostname, like www.domain.com instead of domain.com. But it can redirect you to somewhere completely different in some cases.
- You probably don't want to crawl anything that ends with .jpg, .gif, and definitely not something like .avi, .wmv or .mkv. There are a LOT more file-extensions that you'll want to ignore.
I agree with cmiles74 that using a database is probably a bad idea. For a sizeable crawl (say a billion pages) this database will get pretty damn big. I doubt that you will be able to get decent performance out of anything with "SQL" in its name for such a use-case, unless you throw a ton of hardware at it. Building your own specialized solution for this would probably be a lot faster and less resource-intensive.