Redis Virtual Memory: the story and the code(antirez.com)
antirez.com
Redis Virtual Memory: the story and the code
http://antirez.com/post/redis-virtual-memory-story.html
6 comments
It's amazing what one super bright developer can do: DHH with Rails, Rich Hickey with Clojure and Salvatore antirez Sanfilippo with Redis.
That's very true. I think two factors have helped in these kinds of cases:
1. There wasn't a "boss" or someone dictating what they should do. Creativity reigns and ideas simmer for a while to mature into good ideas.
2. They have a mentality of "if I were to solve this problem from scratch, what would I really need?" This question expands into "What is my wish list of things that are hard to do now but me/everyone has asked for?"
1. There wasn't a "boss" or someone dictating what they should do. Creativity reigns and ideas simmer for a while to mature into good ideas.
2. They have a mentality of "if I were to solve this problem from scratch, what would I really need?" This question expands into "What is my wish list of things that are hard to do now but me/everyone has asked for?"
Giles Bowkett has a quote I like that touches on that: (paraphrased)
We [programmers] create things that didn't exist before; when someone who doesn't have our knowledge is calling the shots we are working with a subset of possibility.
We [programmers] create things that didn't exist before; when someone who doesn't have our knowledge is calling the shots we are working with a subset of possibility.
> solve this problem from scratch
I'm not detracting at all, but object databases with way more advanced feature sets have been around for well over ten years (http://web.progress.com/en/objectstore/). Making a stripped down version of an object database is not exactly forging new ground. I know you weren't necessarily implying otherwise.
I'm not detracting at all, but object databases with way more advanced feature sets have been around for well over ten years (http://web.progress.com/en/objectstore/). Making a stripped down version of an object database is not exactly forging new ground. I know you weren't necessarily implying otherwise.
Redis does not invent nothing new. It's 50 years ago technology mostly, at work. VM, Data structures, and so forth. Even the language it is implemented in, C, is old. But it's not an object database, and this is the point.
Databases are like languages. Does Ruby invented something new? No, but it's my preferred high level language. It's a matter of "interpretation" of already implemented concepts, not a matter of features set.
AFAIK Redis is one of the first DBs exposing basic computer science data structures as abstract types. You have sets, linked lists, a sorted type that is the equivalent of a balanced tree, and an dictionary (an hash table) on top of this.
What matters is if this interpretation of already existing technologies fit well with the needs of today. I'll leave the task of inventing new breaking things to CS departments as I'm a programmer and not a scientist, with all the limits and strengths that this implies.
As long as we, programmers, developers, hackers, computer entusiasts, don't realize that our work is an interpretation work, like architects, we will continue to be nerdy engineers that the society will continue to look at without to recognize the real value.
Do you evaluate a building based on his number of rooms, the ability to resist to forces, and so forth? I don't think so. Why you should use this meters for software?
Databases are like languages. Does Ruby invented something new? No, but it's my preferred high level language. It's a matter of "interpretation" of already implemented concepts, not a matter of features set.
AFAIK Redis is one of the first DBs exposing basic computer science data structures as abstract types. You have sets, linked lists, a sorted type that is the equivalent of a balanced tree, and an dictionary (an hash table) on top of this.
What matters is if this interpretation of already existing technologies fit well with the needs of today. I'll leave the task of inventing new breaking things to CS departments as I'm a programmer and not a scientist, with all the limits and strengths that this implies.
As long as we, programmers, developers, hackers, computer entusiasts, don't realize that our work is an interpretation work, like architects, we will continue to be nerdy engineers that the society will continue to look at without to recognize the real value.
Do you evaluate a building based on his number of rooms, the ability to resist to forces, and so forth? I don't think so. Why you should use this meters for software?
> As long as we, programmers, developers, hackers, computer entusiasts, don't realize that our work is an interpretation work, like architects, we will continue to be nerdy engineers that the society will continue to look at without to recognize the real value.
You have hit the nail on the head with perfect precision.
You have hit the nail on the head with perfect precision.
Chill out man. I'm just pointing out this stuff has been around in commercial products for a very long time.
> AFAIK Redis is one of the first DBs exposing basic computer science data structures as abstract types
With something like ObjectStore you've had persistent storage of arbitrary templatized data structures for ages. You want a hash of hashes ultimately pointing to a void* value? Easy.
> AFAIK Redis is one of the first DBs exposing basic computer science data structures as abstract types
With something like ObjectStore you've had persistent storage of arbitrary templatized data structures for ages. You want a hash of hashes ultimately pointing to a void* value? Easy.
Even with an SQL table you can build every data structure. Pointers to pointers to pointers. But this is not the point(er) and I can't explain it any better than in my previous message.
You don't have to "build" the data structure. You are using your native structures.
As a recent redis convert let me put it differently:
Redis is a tool in the unix-sense of the word. OOStore a solution in the enterprise sense.
Hackers tend to prefer tools, because they can combine them to build superior solutions.
Redis is a tool in the unix-sense of the word. OOStore a solution in the enterprise sense.
Hackers tend to prefer tools, because they can combine them to build superior solutions.
I could be wrong. But let me say it anyway. Is it arrogance? The arrogance could come from the fact that you have clarity of thought. It's the same feeling I get when I see DHH, PG, Linus and a few more very bright people I idolize. Their arrogance doesn't unhelpful. They are some of the most resourceful people I have seen. Otherwise we wouldn't have known them and their work. But then again I haven't seen too much of Rich and Antirez. So I can't comment on them.
Antirez does not seem at all arrogant to me. He's just looked around at the technologies he needs and found that there was something missing, then set about filling that gap.
We all do that in our own way (even if we can't do it in our day jobs). In my case I'm solving far more trivial problems than Antirez.
Redis is a very significant technology for me. The great thing about Antirez is that he has provided this extremely useful tool whilst maintaining a very modest online presence.
Maybe he's a prima donna at home :-)
We all do that in our own way (even if we can't do it in our day jobs). In my case I'm solving far more trivial problems than Antirez.
Redis is a very significant technology for me. The great thing about Antirez is that he has provided this extremely useful tool whilst maintaining a very modest online presence.
Maybe he's a prima donna at home :-)
The app we use for analytics currently works by logging events to the file system and loading them into a mysql database every 5 minutes. I'm trying to convince our sysadmin to let me switch the backend out to use redis, since the only thing I'm using sql for is date filtering.
The main roadblock for him is redis' lack of support for automatic failover (at least for the python client libraries I've seen), which I admit has some validity to it. I don't think it's that big of an issue for our use case, but try telling that to an opinionated sysadmin.
The main roadblock for him is redis' lack of support for automatic failover (at least for the python client libraries I've seen), which I admit has some validity to it. I don't think it's that big of an issue for our use case, but try telling that to an opinionated sysadmin.
If you have trouble with the Python client, please report a bug at: http://github.com/andymccurdy/redis-py/issues
You can patch the open source libraries to add automatic failover. Clearly it's a requirement for many people.
This is great, I was really hoping for Redis to take this direction, since keeping everything in memory wasn't great for my current project.
In the same vein, I have an idea about forking memcached and turn it into "memd". Instead of just caching for reads, it would allow writes to be done in memory too, which are asynchronously synced to disk when an item expires, or based on certain conditions (I'm thinking based on both time since last updated, and when enough updates are made). Now obviously, this only works if the data you store isn't all that important, which is probably the case for most web applications.
In the same vein, I have an idea about forking memcached and turn it into "memd". Instead of just caching for reads, it would allow writes to be done in memory too, which are asynchronously synced to disk when an item expires, or based on certain conditions (I'm thinking based on both time since last updated, and when enough updates are made). Now obviously, this only works if the data you store isn't all that important, which is probably the case for most web applications.
and that's called Redis
Well, the idea preceded Redis and applies more widely. The advantage of having it the way I mentioned is that any database could be plugged in at the back, just as memcached as no requirements for what method of persistence is used. That makes it attractive for other reasons.
I can't help but wish for a similar sql-solution. In-memory, with writes journaled and close enough to mysql that porting wouldn't be a nightmare. <sigh> one can only dream...
There are absolutely super performant in memory sql databases. You just have to pay for them.
There's also csql. http://en.wikipedia.org/wiki/CSQL
There's also csql. http://en.wikipedia.org/wiki/CSQL
Thanks!
Well worth the read to learn the design decisions and the capabilities that are opened up with Redis by keeping just the keys and least used data in memory.
Antirez is my hero of the week :-)
Antirez is my hero of the week :-)