I find it difficult to test a server in "real" conditions, these being at least 300k pps rx and 6 gbps tx, on 2x10G (with LACP) (context = http video streaming)
Using a tool like iperf is useful for getting xx gbps, but I've never managed to reproduce high pps, while it seems that high rx pps is often the limiting factor (causing high softirq %cpu thus packet drops if reaching 100%).
I see a possible market for a distributed job scheduler in Go, with features from Chronos and Rundeck (and possibly based on an equivalent of the Quartz scheduling library?)
Not every bash is HTTP-accessible. I'd even say it's a minority. SSH, on the other hand, seems to be the main access to bash, as only a crafted login name is enough to exploit.
A bit better, in case your API is down, maybe you could have a script that grab a strace (and maybe a gdb backtrace, see http://poormansprofiler.org/) of the usual suspect and join that to the alert.