It's a simple redis list containing JSON task. We have a custom Scrapy Spider hooked to next_request and item_scraped [1]. It check (lpop) for update/discovery tasks in the list and build a Request [2]. We only crawl max ~1 request per second, so performance is not an issue.
For every website we crawl we implement a custom discovery/update logic.
Discovery can be, for example, crawl a specific date range, seq number, postal code.... We usually seed discovery based on the actual data we have, like highest_company_number + 1000, so we get the newly registered companies.
Update is to update a single document. Like crawl document for company number 1234. We generate a Request [2] to crawl only that document.
We monitor exceptions with Sentry. We store raw data so we don't have to hurry to fix the ETL, we only have to fix navigation logic and we keep crawling.
I maintain ~30 different crawlers. Most of them are using Scrapy. Some are using PhantomJS/CasperJS but they are called from Scrapy via a simple web service.
All data (zip files, pdf, html, xml, json) we collect are stored as-is (/path/to/<dataset name>/<unique key>/<timestamp>) and processed later using a Spark pipeline. lxml.html is WAY faster than beautifulsoup and less prone to exception.
We have cronjob (cron + jenkins) that trigger dataset update and discovery. For example, we scrape corporate registry, so everyday we update the 20k oldest companies version. We also implement "discovery" logic in all of our crawlers so they can find new data (ex.: newly registered company). We use Redis to send task (update / discovery) to our crawlers.
What's wrong about billing hourly? Sometimes a client ask me to check something that's not working as expected, it could take less than 10mins, I won't do it for free nor bill a week, I'll bill an hour. I'm happy and he is. If it takes longer I go with half-day, full-day increment.
I do document parsing/data grooming so it's a lot of tweaking/fix as the client do the Q&A on the data.
Actually I overbook myself and offload some work to reliable part-time employees (the client is happy to known that not only me but other people are working on the project).