Scaling matters, but why pay for abusive clients or bots? Adding a cache is easy; the hard part is invalidation, sync, and thundering herd. Use it if the product needs it, not as a band-aid.
We had a critical service that often got overwhelmed, not by one client app but by different apps over time. One week it was app A, the next week app B, each with its own buggy code suddenly spamming the service.
The quick fix suggested was caching, since a lot of requests were for the same query. But after debating, we went with rate limiting instead. Our reasoning: caching would just hide the bad behavior and keep the broken clients alive, only for them to cause failures in other downstream systems later. By rate limiting, we stopped abusive patterns across all apps and forced bugs to surface. In fact, we discovered multiple issues in different apps this way.
Takeaway: caching is good, but it is not a replacement for fixing buggy code or misuse. Sometimes the better fix is to protect the service and let the bugs show up where they belong.
Excellent read. It highlights key aspects like health checks, server restarts, warm up, and load shedding, all of which make load balancing an already hard problem even harder.
Willy from HAProxy has a good write-up on this. In their benchmarks, least-connections usually beat P2C, but P2C was never the worst and is arguably a saner default when least-connections isn’t available.
The article link: https://www.haproxy.com/blog/power-of-two-load-balancing
Author here. Two quick thoughts:
1. As I covered in an earlier part of this series, service discovery is not always easy at scale. High churn, partial failures, and the cost of health checks can make it tricky to get right.
2. Using server-side metrics for load balancing is a great idea. In many setups, feedback is embedded in response headers or health check responses so the LB can make more informed routing decisions. Hodor at LinkedIn is a good example of this in practice:
https://www.linkedin.com/blog/engineering/data-management/ho...
Author here. Absolutely, HAProxy’s sticktables is a powerful way to implement advanced routing logic, and they’ve been around for years. This series focuses on explaining the broader concepts and tradeoffs rather than diving deep into any single implementation, and since it also covers other aspects of reverse proxies, the focus on load balancing here is mostly to present the challenges and high-level ideas.
Glad you found it a good effort, and I agree there’s room to go deeper in future posts.
Author here. Thanks for sharing these thoughts. You’re right that DSR, ASN-based routing, SRV records, and other lower-layer approaches are important in certain setups.
This post is focused primarily on Layer 7 load balancing, connection and request routing based on application-level information, so it doesn’t go into Layer 3/4 techniques like DSR or network-level optimizations. Those are certainly worth covering in a broader series that spans the full stack.
Thanks for sharing this! I’m the author of the blog post.
Happy to answer any questions about the scenarios in the article or dive deeper into specifics like slow start tuning, consistent hashing trade-offs, or how different proxy architectures handle dynamic backends.
Always curious to hear how others have tackled these issues in production.
One of the key aspects of feature flag systems is enabling A/B analysis, measuring the impact of features on metrics like click-through rate, session duration, and revenue. This doesn’t seem to be mentioned in the highlights. Is the product targeting these kinds of insights?
A useful way to find ideas is by observing asymmetries. When something appears easy for some people but difficult for others, or vice versa, it often reveals underlying complexity, missing tools, or flawed mental models. Frustration can also be a signal. If something feels unnecessarily painful or awkward while others seem fine with it, there may be an opportunity to simplify, rethink, or build something better. These patterns often lead to meaningful insights or practical solutions.
In my experience, unit tests and logging code generated by LLMs tend to be overly verbose, miss meaningful assertions, and often produce boilerplate that looks correct but doesn’t test or log anything useful. It’s easy to get misled by the surface structure.
At my company, we went from using detailed RFCs to one-pagers, and eventually to writing no formal docs at all. Over time, RFCs turned into artifacts optimized for promotions rather than alignment or clarity. In many cases, people ended up spending more time crafting the perfect document than actually implementing the solution.