The max thread pool size definitely affects the maximum I/O we can do and we've tuned that based on perf testing the memcache/cassandra clients in isolation.
However the bottom line is that we are still limited by the max spout pending or the max number of queries we can run concurrently. There will be short spikes that might use up all the I/O threads but they are short lived because there are other CPU intensive operations that the topology is doing at the same time.
In terms of blocking for each bolt, we don't block, the bolt tries to execute the operation in the current thread if the I/O thread pool is saturated. This way if there are pending tuples they queue up in storm's receive queues and it creates a back pressure when we have too much work outstanding.
Hi, I'm the author of that post. If a thread is not available from the pool the calling thread tries to execute the operation itself. That allows the bolt to continue making progress but adds a simple back pressure mechanism.
This also avoids the overhead of any sort of locking to wait for a thread to be free. Since the number of bolt executors will always be much less than the number of threads in the IO pool it doesn't add too much the concurrency overall. Hope that makes sense.
The max thread pool size definitely affects the maximum I/O we can do and we've tuned that based on perf testing the memcache/cassandra clients in isolation.
However the bottom line is that we are still limited by the max spout pending or the max number of queries we can run concurrently. There will be short spikes that might use up all the I/O threads but they are short lived because there are other CPU intensive operations that the topology is doing at the same time.
In terms of blocking for each bolt, we don't block, the bolt tries to execute the operation in the current thread if the I/O thread pool is saturated. This way if there are pending tuples they queue up in storm's receive queues and it creates a back pressure when we have too much work outstanding.
Hope this helps! Manu