Microsoft`s Open Source Subsidiary Releases First Deliverable (Redis)(eweek.com)
eweek.com
Microsoft`s Open Source Subsidiary Releases First Deliverable (Redis)
http://www.eweek.com/c/a/Linux-and-Open-Source/Microsofts-Open-Source-Subsidiary-Releases-First-Deliverable-535785/
12 comments
How do you do that on Unix / Linux?
Don't you have to set the size when you map the file?
Don't you have to set the size when you map the file?
You can change the size of the allocation on the fly, and just keep writing bytes as you go. Pages aren't alloc'd on disk until they're touched. When you hit the end of the file, you call a final resize to trim off the virtual bytes at the end.
You can see an example of the technique here: http://www.gnu.org/software/libc/manual/html_node/File-Size.... (scroll to the bottom).
You can see an example of the technique here: http://www.gnu.org/software/libc/manual/html_node/File-Size.... (scroll to the bottom).
> You can change the size of the allocation on the fly, and just keep writing bytes as you go.
How do you do this technically? mremap(), I assume?
You can do something very similar on windows, by VirtualAlloc()ing at the end of the previous allocation, and if that fails, be committing, VirtualFreeing and remapping at a different address.
Yes, I agree mremap() is much cleaner. I am mostly a Linux guy myself. What I'm missing in Linux is "mremap2()", which would map an existing block of memory onto a file (whereas mmap maps an existing file onto a block of memory ). redis achieves similar functionality (and more) by fork()ing and write()ing instead, but it would be nicer if it could be done with one call and no fork()ing. a "copy this file from here to here" could then be done by mmap()ing the source into memory, mremap2()ing the memory into the destination file, and forgetting about it.
added: This would also allow btrfs/zfs to have O(1) file copy, even between independent media if possible (i.e. source file already exists on target media, even if in a different directory) - without any consideration from the cp implementation.
How do you do this technically? mremap(), I assume?
You can do something very similar on windows, by VirtualAlloc()ing at the end of the previous allocation, and if that fails, be committing, VirtualFreeing and remapping at a different address.
Yes, I agree mremap() is much cleaner. I am mostly a Linux guy myself. What I'm missing in Linux is "mremap2()", which would map an existing block of memory onto a file (whereas mmap maps an existing file onto a block of memory ). redis achieves similar functionality (and more) by fork()ing and write()ing instead, but it would be nicer if it could be done with one call and no fork()ing. a "copy this file from here to here" could then be done by mmap()ing the source into memory, mremap2()ing the memory into the destination file, and forgetting about it.
added: This would also allow btrfs/zfs to have O(1) file copy, even between independent media if possible (i.e. source file already exists on target media, even if in a different directory) - without any consideration from the cp implementation.
> mremap(), I assume
Yes, with ftruncate(), see the link above. The problem is Windows doesn't have an ftruncate() that works while the file is open and being written to. (Note: my memory is a little fuzzy here, it might be that there was no implementation that took a file descriptor instead of a filename, either way I couldn't pull it off). As for your 'zero-copy dump memory into a file after-the-fact' idea, I'm sure it's done somewhere. My brain is too frizzled right now to scrounge up how though :).
Yes, with ftruncate(), see the link above. The problem is Windows doesn't have an ftruncate() that works while the file is open and being written to. (Note: my memory is a little fuzzy here, it might be that there was no implementation that took a file descriptor instead of a filename, either way I couldn't pull it off). As for your 'zero-copy dump memory into a file after-the-fact' idea, I'm sure it's done somewhere. My brain is too frizzled right now to scrounge up how though :).
When I was still following the kernel mailing lists, there were discussions about how to achieve this for sockets, and none about files, so that would surprise me.
However, if there is a way to achieve that without forking, I would be very happy to hear.
However, if there is a way to achieve that without forking, I would be very happy to hear.
[deleted]
This is good news, I've been waiting for the MS supported version of Redis so I can play with it on a Windows box.
I wonder why they hosted on Github vs Codeplex. Codeplex supports Mercurial, Team Foundation Server or Subversion and Git.
Microsoft recently posted the source for _parts_ of ASP.NET (minus web forms) on codeplex.
(just curious, not trying to troll.)
I wonder why they hosted on Github vs Codeplex. Codeplex supports Mercurial, Team Foundation Server or Subversion and Git.
Microsoft recently posted the source for _parts_ of ASP.NET (minus web forms) on codeplex.
(just curious, not trying to troll.)
I'm pretty sure it's a case of Codeplex is going through a massive overhaul[1]. The addition of git support seems to be the first step.
[1] here's a preview of the new look they are moving to: http://blogs.msdn.com/b/codeplex/archive/2012/03/30/new-code...
[1] here's a preview of the new look they are moving to: http://blogs.msdn.com/b/codeplex/archive/2012/03/30/new-code...
They are trying to garner good will from the community, it's only logical that they choose whatever is most popular right now.
Redis is hosted on Github. Most users of Redis are on Github, too. So if MS wants to exchange code back and forth between their fork and the original, it makes sense to host their code on Github, too. A pull request is much more friendly than "Hey, check out my Codeplex", even if antirez decides not to merge the changes.
[deleted]
The memory-mapped file APIs on Windows are seriously lacking. Last time I dealt with them, there was no documented way to write to a constantly-growing mmaped file (unless you knew its size already). All the workarounds I tried ended in segfaults :/. Curious if anyone here has had success with that.