Tollbooth: HTTP rate limiter middleware in Go(didipkerabat.com)
didipkerabat.com
Tollbooth: HTTP rate limiter middleware in Go
http://didipkerabat.com/posts/1432264032306462173-tollbooth-http-rate-limiter-middleware-in-go.html
20 comments
If interested, I wrote a different type of rate limiter in my spare time for my use case.
http://badactor.org
http://badactor.org
I really like your simple splash demo. Good work, I might even check out your code.
The code is solid. I gauge this by the simple fact that when I review the project, I don't absolutely hate it ;-)
I need to circle back around as there are some features I'd like to add. PRs are welcome.
I need to circle back around as there are some features I'd like to add. PRs are welcome.
I was writing an netfilter/iptables rate limiter and a nginx rate limiter (both able to handle quite a lot of requests) but all the system admins I talked to shamed me for trying to rate limit. They raised all kinds of points about blocking access to large companies or small countries and it not being effective for DDoS anyway. Any thoughts on this?
A bandwidth rate limiter is indeed of dubious utility. There's a lot of ways that can go wrong, and there's a lot of ways it can fail to do what the user expected anyhow, and there's a lot of ways it can do both.
But there's a lot of things that can be sensibly rate limited, such as logins attempts to a given account. Now that has its own considerations, too, if pushed to the limit... you'd prefer that an attacker can't lock down your service just by spuriously trying to log in to all your accounts 5 times every 5 minutes or something. But loud downtime (which you can then react to) may be preferable to getting your users silently hacked.
I definitely agree that they are less useful than they look at first glance, a great deal more complicated than you'd like, and more subtle than you'd think. But they can still be a useful tool.
But there's a lot of things that can be sensibly rate limited, such as logins attempts to a given account. Now that has its own considerations, too, if pushed to the limit... you'd prefer that an attacker can't lock down your service just by spuriously trying to log in to all your accounts 5 times every 5 minutes or something. But loud downtime (which you can then react to) may be preferable to getting your users silently hacked.
I definitely agree that they are less useful than they look at first glance, a great deal more complicated than you'd like, and more subtle than you'd think. But they can still be a useful tool.
The only effective counter against a proper DDoS is to bring more bandwidth than your attacker has.
A small site is unlikely to attract a proper DDoS, but will get hit by the usual misbehaving web scrapers, broken clients in a silly loop (I'm looking at you, iTunes) and keyboards with super sensitive F5 key. In those cases, serving an occasional 503 might work better than the alternatives.
A small site is unlikely to attract a proper DDoS, but will get hit by the usual misbehaving web scrapers, broken clients in a silly loop (I'm looking at you, iTunes) and keyboards with super sensitive F5 key. In those cases, serving an occasional 503 might work better than the alternatives.
It's built around github.com/juju/ratelimit
It looks to me like the map of rate buckets will grow unbounded and never be cleaned up?
Looks to be the case based on the config. I'm suprised storage wasn't abstracted to accommodate redis or the like which is essential in a load balanced scenario.
The API doesn't seem quite idiomatic either, I'd expect to create a struct containing options and a function that closes over the http.Handler interface e.g. func(l *Limiter) Limit(next http.Handler) http.Handler or a function that takes options and a next http.Handler that creates a struct implementing http.Handler.
The API doesn't seem quite idiomatic either, I'd expect to create a struct containing options and a function that closes over the http.Handler interface e.g. func(l *Limiter) Limit(next http.Handler) http.Handler or a function that takes options and a next http.Handler that creates a struct implementing http.Handler.
https://github.com/etcinit/speedbump is a Redis backed rate limiter that we're currently evaluating for our app.
This is neat. What's the best way to do rate limiting in from of Apache for wordpress/drupal sites these days? everytime I go looking for something new I'm dissapointed.
You could use the OP's project and point everything that makes it through the limiter at Apache with a reverse proxy http://golang.org/pkg/net/http/httputil/#ReverseProxy
Use mod_security as most Apache installs already have it enabled and it achieves the same thing.
Serious question: why would you not use HAProxy, nginx or similar software which arguably are more battle tested in this domain?
For general rate limiting, yes HAProxy or Nginx make more sense.
Having different rate limits for different endpoints becomes unmanageable in HAProxy/Nginx; and having different rate limits say across different plans or users is all but impossible.
Having different rate limits for different endpoints becomes unmanageable in HAProxy/Nginx; and having different rate limits say across different plans or users is all but impossible.
You can do that when the HTTP call is simple, when you need some logic in the rate limitation ( auth, login ect... ) it's impossible.
The best reason I can imagine is so that it can all be wrapped in your portable Go binary. Because of the portability aspect, there are a lot of use cases for Go similar to Java where there is an incentive to do everything "in Go".
Four startups ago, I did a simpler one in Nginx using embedded Lua and Redis. It can definitely be done, but writing it in Go is easier, and funner :D
(I thought rate limiters were very simple timers)