doesn't htop require you to go onto the box? We haven't released our custom pcp modules yet which allow more complex visualizations such as flamegraph generation from perf event sampling.
I don't work on the recommendations team but I feel I can safely say your recommendations are calc'd each day. Unfortunately I can't share more about our recommendation model other than to say its a strong business advantage.
The microservice architecture results in a larger footprint than you might find with a more monolithic app stack; which provides us with improved engineering velocity and reliability. Although the content playback is served by our CDN, all other device interactions are handled by the services on AWS. We also have some blogposts which provide a glimpse into some of the work we do behind the scenes (http://techblog.netflix.com/2015/01/netflixs-viewing-data-ho...). Oh yeah, we also encode and run our big data services on AWS...
Our architecture and the sheer number of microservices contributes to much of the scale. In order to achieve the engineering velocity and reliability goals we felt the explosion of instances with this architecture was worth it. If you consider the number of microservice instances serving user traffic (and include persistency tiers such as memcache and cassandra) you would still be in the tens of thousands.
From my experience Little's law is typically used to quantify the number of users in the system, a measure of concurrency. For the purpose of our tools we leverage the calculation to provide insight on "offered load" or the time spent w/in the service for a given interval. We do have a challenge in that many of our downstream dependencies are called concurrently. At the current time this prevents us from easily decomposing the demand in a service cleanly among it's dependencies. Some of this has to do with our transaction tracing framework and the granularity at which we require call behavior to be easily time-ordered. We believe we can solve this overtime with an improved framework.
In the case of Mogul we leverage the demand calculation to understand who is the largest contributor, pointing us in the direction of possible optimization. If we are using the utility to triage an issue we typically find that an increase in the demand or offered load within the problematic dependency tends to easily correlate with the demand of the calling service. I think we are just at the beginning of leveraging this data in a more effective manner, and getting away from having eyeballs look at a dashboard is definitely a goal.
we don't share our actual requests per-second numbers on the front door. We have mentioned that we run tens of thousands of instances across three AWS regions. Per the Atlas techblog, these instances can generate in aggregate upwards of 1.2B time series which are exposed at the minute level.
As with the other hires on my team I'm thrilled to have Brendan onboard. I would also hope that if he happens to leave Netflix due to issues with the culture he will share that experience as well, publicly.