Tried posting this comment to your page, but I think the cron job listening to the logfile may have died
I do something like that for an internal brand safety product I created for the company I work for - it is surprisingly small and scales ridiculously well. Using zmq (ahem, now known as crossroadsio due to stupid trademark issues), the loglines are then processed asynchronously. I use a python module called apachelog to process the log lines.
Using the-formerly-known-as-zmq async processing, we can actually scale it up massively (to the point where between-machine latency is the cause of slowdown)
In your case I think you can put that + websockets to good use - so users get instant feedback while you rewrite the html in the background
I do something like that for an internal brand safety product I created for the company I work for - it is surprisingly small and scales ridiculously well. Using zmq (ahem, now known as crossroadsio due to stupid trademark issues), the loglines are then processed asynchronously. I use a python module called apachelog to process the log lines.
Using the-formerly-known-as-zmq async processing, we can actually scale it up massively (to the point where between-machine latency is the cause of slowdown)
In your case I think you can put that + websockets to good use - so users get instant feedback while you rewrite the html in the background