I've only maintained a small number of servers, but I've found a good solution in what I'll call the GEL (Graylog2 - Elasticsearch - Logstash) stack. It's been some time since I last used Graylog2, and I can recall that it was somewhat lacking in the pretty charts and graphs department, though that may have improved recently, and the search functioned beautifully.