Mixpanel went down, so we're open sourcing a better pure Python memcache client(code.mixpanel.com)
code.mixpanel.com
Mixpanel went down, so we're open sourcing a better pure Python memcache client
http://code.mixpanel.com/2012/07/16/we-went-down-so-we-wrote-a-better-pure-python-memcache-client/
18 comments
pylibmc may be fine, as well.
Yet again, instead of trying to fix the existing, someone writes yet another library.
This was my thought, too: there seems to be a limit beyond which memcached simply breaks. I learned this 5 years ago, and it had nothing to do with Python.
Well, if there are too many users of an existing library, it makes it hard to do large changes. So in their case, it seems like the original library would silently fail on exceptions, they want to make things bubble up to the top which could be bad for a large number of users who have been depending on the silent-fail operation.
This is the type of response that makes me a happy Mixpanel user. Hit a snag; fix it, talk about it, open source it.
Perhaps it could be named better; all the python memcache clients have almost identical names.
You might want to set TCP_NODELAY on the socket to disable Nagle's algorithm (http://en.wikipedia.org/wiki/Nagles_algorithm). We ran into some issues with memcached where if Nagle's algorithm was not turned of, GETs would often take several milliseconds longer than was necessary.
"It turns out that socket operations that fail in python-memcached don’t return an error or raise any exceptions."
...Whats surprising is that this went untested for so long.
...Whats surprising is that this went untested for so long.
We've been very well aware of it and safely worked around it at the time. It was also something we just disliked and so we changed it across our entire codebase when we decided to write our own library.
When we said "it turns out", we didn't mean that we had suddenly learned that after failure. It's just something we meant to note that we decided to fix.
When we said "it turns out", we didn't mean that we had suddenly learned that after failure. It's just something we meant to note that we decided to fix.
I would argue that python-memcached has been replaced as "the de-facto standard" by pylibmc. I'm not sure if pylibmc works with eventlet or not.
Companies open sourcing code is always good. But, I would be surprised if there wasn't already a python memcached client that would suit your needs.
Companies open sourcing code is always good. But, I would be surprised if there wasn't already a python memcached client that would suit your needs.
I've been unable to compile pylibmc Centos 5.
[deleted]
> We enforce the calling convention with code in our library that looks like this:
if not isinstance(val, str): raise ValidationException('value must be str', val)
I love Python, but the more I used it the more I come to see dynamic typing as a mistake. On large projects with shared library code and deeply nested calls, one of the most pervasive problems seems to be wrongly typed data. And building validation exceptions into all of your code seems like a poor solution.
if not isinstance(val, str): raise ValidationException('value must be str', val)
I love Python, but the more I used it the more I come to see dynamic typing as a mistake. On large projects with shared library code and deeply nested calls, one of the most pervasive problems seems to be wrongly typed data. And building validation exceptions into all of your code seems like a poor solution.
They could have implemented it using a simple decorator for non var-args methods which personally I think is more attractive.
Obviously you still pay the cost at run-time but at least the actual source looks more sensible that way.
Obviously you still pay the cost at run-time but at least the actual source looks more sensible that way.
I'm confused, why would a cache being down cause data loss?
Two months is also not battle tested, perhaps the terminology used here needs to be adjusted...
Two months is also not battle tested, perhaps the terminology used here needs to be adjusted...
> Calling .get() on a dead server returns None on failure, the same thing that is returned when there’s no value.
I suppose they were doing a .get() to check if the server was still up, and assuming None meant "there are no values to get, but the server is up". If .set() works in the same way (silently failing) they were essentially using /dev/null as their task queue. :)
I suppose they were doing a .get() to check if the server was still up, and assuming None meant "there are no values to get, but the server is up". If .set() works in the same way (silently failing) they were essentially using /dev/null as their task queue. :)
Yeah. It's a cache, it says so in the name. Why would you make not-a-cache on a cache protocol and library? In-sane.
Not true. ultramemcache (https://github.com/esnme/ultramemcache) is an example of a c++ memcached client that is friendly to gevent's monkeypatching. It does this by importing the socket library via the python runtime (https://github.com/esnme/ultramemcache/blob/master/python/um...), rather than using a native construct.
My experience is with gevent, but I suspect that eventlet's monkeypatching is similar to gevent.