CloudFlare has an approach that may scale to the levels you are looking for; rather than storing the logs, they analyze and rollup the expected responses in realtime, and store additional detail for items that appear anomalous. John Graham-Cumming performed a talk on this topic earlier this month at dotScale: