Linux ate my RAM(linuxatemyram.com)
linuxatemyram.com
Linux ate my RAM
http://www.linuxatemyram.com/
127 comments
> For some reason (probably because of the way it's implemented internally)
(this is tautologically true)
(this is tautologically true)
he means the rationale behind developing it that way was due to the way other stuff is laid out internally, not "the number shows up because the existing code codes for it and causes it to show up".
Linux also counts files on tmpfs as part of the cache number--it's really annoying that no number in top can serve to show you a decent approximation of how many page-size malloc()s you can do before OOM (with overhead and fragmentation and so on, of course it's infeasible to get an exact number).
It's worse. No line of /proc/meminfo has that value, and since not all of the cache can be instantly discarded (I didn't know that), I don't see any combination of them that would help.
That sentence:
> There are no downsides, except for confusing newbies.
Just isn't true. There is the downside that we can't know how much memory is available.
That sentence:
> There are no downsides, except for confusing newbies.
Just isn't true. There is the downside that we can't know how much memory is available.
Yeah, and obviously some (though not all) forms of shared memory are actually implemented using tmpfs behind the scenes which makes life even more interesting. I think the problem I encountered involved SysV shared memory which isn't backed by tmpfs and does show up in /proc/meminfo, whereas other types may not.
Can't you use something like this:
https://raw.github.com/pixelb/ps_mem/master/ps_mem.py
I use that to help determine "least incorrect" ram usage on linux. It is ok, maybe not perfect but it at least lets you get a better view on memory.
I use that to help determine "least incorrect" ram usage on linux. It is ok, maybe not perfect but it at least lets you get a better view on memory.
So... what exactly do I have to type on my command line to see how much RAM can actually be used by applications?
I used to have a similar problem on OS X. When I was working on a 32-bit app that could eat tons of memory, I ended up using a bunch of weird heuristics and allocation tracking just to get a probable assurance that the program could recover properly from running out of address space.
Previous Discussions:
https://news.ycombinator.com/item?id=3698652 (1 year ago)
https://news.ycombinator.com/item?id=6368562 (2 months ago)
https://news.ycombinator.com/item?id=1831322 (3 years ago, no comments)
https://news.ycombinator.com/item?id=3698652 (1 year ago)
https://news.ycombinator.com/item?id=6368562 (2 months ago)
https://news.ycombinator.com/item?id=1831322 (3 years ago, no comments)
How bizarre. In the 1993-94 time frame, one of the big talking points about Windows NT was that it was the only OS with an integrated file system cache. That wasn't true even at that time, as Solaris also had such a cache, but such is the nature of propaganda.
I'm tickled that what once was a big "technical" advantage becomes a major source of confusion. Presumably, OSes of Windows NT heritage have retained file system caching, which offers a huge performance benefit. Why don't Windows users have confusion over this issue? Doesn't Windows offer an easy way to check or is the whole thing presented clearly, or is Windows so complicated not many people get this far?
I'll also draw a parallel to DUI - without a breath-o-meter, DUI is a lot like driving while distracted. It's impossible to quantify, and therefore not that much of a crime. With a breath-o-meter, or with "top", it's possible to put a number on drunkeness or lack of free memory, and the whole thing becomes a problem.
I'm tickled that what once was a big "technical" advantage becomes a major source of confusion. Presumably, OSes of Windows NT heritage have retained file system caching, which offers a huge performance benefit. Why don't Windows users have confusion over this issue? Doesn't Windows offer an easy way to check or is the whole thing presented clearly, or is Windows so complicated not many people get this far?
I'll also draw a parallel to DUI - without a breath-o-meter, DUI is a lot like driving while distracted. It's impossible to quantify, and therefore not that much of a crime. With a breath-o-meter, or with "top", it's possible to put a number on drunkeness or lack of free memory, and the whole thing becomes a problem.
From what I can tell, Windows does not show the disk cache allocation in Task Manager. So even though all the RAM is likely in use, that is hidden from the user. You can probably see it in PerfMon, as that is less "typical user" oriented.
In windows 7, take "available" and subtract "free", that's your cache usage.
Windows Vista started using RAM very aggressively for caching, including pre-filling cache with frequently used pages (apps/data) from disk... ReadyBoost was basically an extension of this...
Windows Vista started using RAM very aggressively for caching, including pre-filling cache with frequently used pages (apps/data) from disk... ReadyBoost was basically an extension of this...
It's not in the graph but it's there in the "Cached" field.
Why don't Windows users have confusion over this issue?
Windows' Task Manager has generally counted memory occupied by the disk cache as "available."
Windows' Task Manager has generally counted memory occupied by the disk cache as "available."
> Windows' Task Manager has generally counted memory occupied by the disk cache as "available."
Windows Vista got lots of complaints about memory usage, because Task Manager listed memory as Total, Cached, and Free. People saw that "Free" was close to 0, and complained that Windows Vista ate all their RAM.
Windows 7 changed this to: Total, Cached, Available, and Free. People saw that they had a lot of "Available" memory, so they didn't panic. Well, most of them didn't panic.
Windows 8 removed the "Free" line entirely, and now lists memory as: Total, Available, and Cached. If you care about the actual amount of free memory, you can run Resource Monitor or Process Explorer.
Windows Vista got lots of complaints about memory usage, because Task Manager listed memory as Total, Cached, and Free. People saw that "Free" was close to 0, and complained that Windows Vista ate all their RAM.
Windows 7 changed this to: Total, Cached, Available, and Free. People saw that they had a lot of "Available" memory, so they didn't panic. Well, most of them didn't panic.
Windows 8 removed the "Free" line entirely, and now lists memory as: Total, Available, and Cached. If you care about the actual amount of free memory, you can run Resource Monitor or Process Explorer.
Because there are a lot of things in linux/unix-land that is completely braindamaged. But they have been with us for so long that they become impossible to change. Like dotfiles, the ps command and Linux memory handling.
Windows 8 has a nice graph which shows memory usage over time in which the filesystem cache is not visible. If it was included, memory usage would constantly hover at about 90% which wouldn't be very useful and very confusing.
When Windows XP was introduced famous tech journalist John Dvorak complained that "System Idle Process" was using 99% of his cpu and slowing down his computer. In later Windowses, System Idle Process is no more.
Also even when you subtract the fs cache and get number of megabytes free memory you have available. That number has almost no bearing on how many number of megabytes an application of yours can allocate. Linux memory management is much more complicated than that.
Windows 8 has a nice graph which shows memory usage over time in which the filesystem cache is not visible. If it was included, memory usage would constantly hover at about 90% which wouldn't be very useful and very confusing.
When Windows XP was introduced famous tech journalist John Dvorak complained that "System Idle Process" was using 99% of his cpu and slowing down his computer. In later Windowses, System Idle Process is no more.
Also even when you subtract the fs cache and get number of megabytes free memory you have available. That number has almost no bearing on how many number of megabytes an application of yours can allocate. Linux memory management is much more complicated than that.
Once upon a time John Dvorak's word was taken seriously.
Whatever happened to him?
Whatever happened to him?
what's wrong with dotfiles?
> Linux memory management is much more complicated than that.
I don't know if I'm being trolled, but memory management in any modern OS that makes proper use of shared memory is fundamentally very complicated. I suspect what is happening is Windows is lying to you to give you a simple (though essentially wrong) answer.
I don't know if I'm being trolled, but memory management in any modern OS that makes proper use of shared memory is fundamentally very complicated. I suspect what is happening is Windows is lying to you to give you a simple (though essentially wrong) answer.
NT was able to use a simpler (maybe not simple, but simpler) design because it was written from scratch once we knew that caches and buffers were desirable. So it's got a unification of swap and cache that would be difficult to retrofit onto *nix.
The Linux kernel and the NT kernel are approximately the same age.
Please read LWN more.
It was at least changed to lie to avoid complaints from users.
I remember that running taskmgr.exe from an XP machine on Vista gave a more accurate info, ie. no lying to keep ignorant users happy.
Wait, so it's good to not show people everything?
Iknow my taste in user interfaces is very different than most, but this really explains a lot.
Iknow my taste in user interfaces is very different than most, but this really explains a lot.
UI/UX design includes knowing what users don't want to see and appropriately hiding it from them.
Must be the same principle the NSA is operating on then. ;)
Even htop doesn't show you everything it can by default. See the (default off) option "Detailed CPU time (System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)"
Windows 7 still has "System Idle Process" showing CPU usage.
And Dvorak's complaint about "System Idle Process" had a little bit more context. He was complaining that XP would become non-responsive while still being mostly idle (something I remember well from that era too). He could definitely have done a much better job of expressing it though:
IDLE-TIME PROCESS. Once in a while the system will go into an idle mode, requiring from five minutes to half an hour to unwind. It's weird, and I almost always have to reboot. When I hit Ctrl-Alt-Delete, I see that the System Idle Process is hogging all the resources and chewing up 95 percent of the processor's cycles. Doing what? Doing nothing? Once in a while, after you've clicked all over the screen trying to get the system to do something other than idle, all your clicks suddenly ignite and the screen goes crazy with activity. This is not right.
http://www.pcmag.com/article2/0,2817,1304348,00.asp
IDLE-TIME PROCESS. Once in a while the system will go into an idle mode, requiring from five minutes to half an hour to unwind. It's weird, and I almost always have to reboot. When I hit Ctrl-Alt-Delete, I see that the System Idle Process is hogging all the resources and chewing up 95 percent of the processor's cycles. Doing what? Doing nothing? Once in a while, after you've clicked all over the screen trying to get the system to do something other than idle, all your clicks suddenly ignite and the screen goes crazy with activity. This is not right.
http://www.pcmag.com/article2/0,2817,1304348,00.asp
Well writing "hogging all the resources and chewing up cycles" shows he doesn't really get it at all. Not to mention that there's many reasons things can become non-responsive without using the CPU. Like a shitty driver blocking on some timeout.
Sigh, "tech journalism".
Sigh, "tech journalism".
Why don't Windows users have confusion over this issue?
Because windows users don't care about how much memory a particular application is using. New Windows developers get confused by it all the time. There are plenty of questions on stack overflow where someone is using task manager to try and measure their application's working set.
Because windows users don't care about how much memory a particular application is using. New Windows developers get confused by it all the time. There are plenty of questions on stack overflow where someone is using task manager to try and measure their application's working set.
An overgeneralized question deserves an overgeneralized answer. Well done.
Not sure what you mean. There is a general lack of sophistication among windows users, so they believe what the system is telling them if they go so far as to even ask the question. Among developers it is well known that, that number is at best wrong. If you actually want to know that number use a memory profiler. If I wanted to know about memory consumption on Linux I wouldn't be looking at top I would be looking at valgrind.
Top and task man are like stepping on the scale to keep track of your weight the exact number doesn't mean much, and it regularly varies in an unmeaningful way vs a memory profiler which would be like a high precision body fat measurement. Right tool for the job and all that crap.
Top and task man are like stepping on the scale to keep track of your weight the exact number doesn't mean much, and it regularly varies in an unmeaningful way vs a memory profiler which would be like a high precision body fat measurement. Right tool for the job and all that crap.
A quick practical answer is to use htop and look at the memory bar.
* Green = resident memory used (probably what you want most of the time)
* Blue = buffers
* Yellow = cache
* Green = resident memory used (probably what you want most of the time)
* Blue = buffers
* Yellow = cache
htop author here. It was nice seeing the table in the article with the memory split in three colors and thinking "well, but that's how I already split it in htop, isn't it?".
The "Linux calls it" column should probably be called "top calls it"...
The "Linux calls it" column should probably be called "top calls it"...
Thank you for writing htop, it is very useful tool.
I just about fell off my chair when I realized I could click with my mouse on the columns to sort them!
I just about fell off my chair when I realized I could click with my mouse on the columns to sort them!
I always forget the key bindings for this, you just changed my life!
It was accidental too. I was just showing my coworker something and I have a bad habit of highlighting stuff I am reading and randomly clicking on text, and I noticed that column was selected to be sorted. It blew my mind, this was in an SSH session logged into a remote server.
+1 for htop, it not only shows excellent memory stats in an easy to absorb way but it also shows CPU cores usage. Definitely one of my "wtf is going on?" tools.
I'll give a pitch for atop as well.
Here's a screenshot of it showing systemwide and per-process memory information: http://www.atoptool.nl/images/screenshots/memory.png
Here's a case study of using it to observe a memory leak: http://www.atoptool.nl/download/case_leakage.pdf
Here's a screenshot of it showing systemwide and per-process memory information: http://www.atoptool.nl/images/screenshots/memory.png
Here's a case study of using it to observe a memory leak: http://www.atoptool.nl/download/case_leakage.pdf
That's how I check true free memory available:
root@joe-pc:~# free -m
total used free shared buffers cached
Mem: 32057 30347 1710 0 1312 21568
-/+ buffers/cache: 7466 24591
Swap: 0 0 0
root@joe-pc:~# sync; echo 3 > /proc/sys/vm/drop_caches
root@joe-pc:~# free -m
total used free shared buffers cached
Mem: 32057 5628 26429 0 4 891
-/+ buffers/cache: 4732 27325
Swap: 0 0 0
root@joe-pc:~#Yikes! Hopefully not on production systems. I've had fileservers which needed 60+GB of dentry cache to serve up files in a reasonable time, something like that could bring the site to a crawl for 10 minutes or more.
Good point. I am doing it on my laptop and server without load. So this is not a big problem in my cases.
Then you should http://downloadmoreram.com
Haha, that's brilliant. From their Twitter account:
> Have an iPhone 3? Download RAM twice to make it a iPhone 5.
https://twitter.com/DownloadMoreRam/status/38502626000858726...
> Have an iPhone 3? Download RAM twice to make it a iPhone 5.
https://twitter.com/DownloadMoreRam/status/38502626000858726...
Crap, I wanted to post that!
Brilliant. I love it.
When my circle of friends found that site it became an in-joke. I'd never heard of it, and eventually they came up to me and asked "Can you download more RAM?".
I don't think they were expecting my answer:
"Sure you can, just use plan 9."
I don't think they were expecting my answer:
"Sure you can, just use plan 9."
The page seems to be down.
Here's how it looked: http://web.archive.org/web/20131011053519/http://www.linuxat...
Here's how it looked: http://web.archive.org/web/20131011053519/http://www.linuxat...
Forget the kernel, someone really needs to do chromeatemyram.com
This is the worst piece of Linux advice ever. I use Linux for many years and every time I had a memory leak or a process using too much memory, every google search about the problem would lead to some "guru" giving this advice.
This explanation which has been replicated thousands of times on every linux forum on the planet, prevents people from solving real problems. It started as a useful piece of information but nowadays it is spam.
This explanation which has been replicated thousands of times on every linux forum on the planet, prevents people from solving real problems. It started as a useful piece of information but nowadays it is spam.
How does this understanding prevent people from solving real problems, simply by virtue of it being commonly cited? I don't understand your complaint.
I understand this is a complaint about Google. It won't help you find pages that cite a problem when a diffrent, but similar problem is too popular.
I've seen that happening several times, but I'm not the GP, so I may still be missunderstanding him.
I've seen that happening several times, but I'm not the GP, so I may still be missunderstanding him.
Yes, this is exactly what I mean. Maybe I should've describe it as noise rather than spam.
But it isn't only a Google search issue. Every time you will ask a question about a memory problem in Linux, everybody will give you this answer and classify you as newbie. When you explain them that this isn't the case, they will simply leave the thread/question and nobody would ever bother reading your problem again.
People just go in denial mode because of this common knowledge, as Linux can't have memory leaks or any memory related problem.
But it isn't only a Google search issue. Every time you will ask a question about a memory problem in Linux, everybody will give you this answer and classify you as newbie. When you explain them that this isn't the case, they will simply leave the thread/question and nobody would ever bother reading your problem again.
People just go in denial mode because of this common knowledge, as Linux can't have memory leaks or any memory related problem.
I think what you're running into is the shallow knowledge of people on forums/mailing lists. With a few exceptions, the people who actually know the most about the Linux kernel, glibc internals, etc. do not frequently post on StackOverflow. They're too busy, you know, working on the kernel and shit.
So you get Joe Chucklefuck, he installed Ubuntu three months ago and posted on the Ubuntu forums about how he has no free RAM and Linux sucks. Someone showed him this page, and now when he sees your question about why malloc isn't working right, he thinks, "Aha! A memory problem! I shall post that linux ate my ram site!" Meanwhile, Ulrich Drepper isn't reading the list, even though he could tell you that the version of glibc you're using has a known bug in malloc. (He'll also tell you that you're stupid, but that's just because he's an asshole)
So you get Joe Chucklefuck, he installed Ubuntu three months ago and posted on the Ubuntu forums about how he has no free RAM and Linux sucks. Someone showed him this page, and now when he sees your question about why malloc isn't working right, he thinks, "Aha! A memory problem! I shall post that linux ate my ram site!" Meanwhile, Ulrich Drepper isn't reading the list, even though he could tell you that the version of glibc you're using has a known bug in malloc. (He'll also tell you that you're stupid, but that's just because he's an asshole)
Exactly. But I believe that developers have their hands full solving real problems, no need to ask them play level 1 support too.
So when I find an issue that I can't be sure if it isn't my own fault, or I don't know the root cause of it, I try to gather some information before heading to the bugzilla. Memory issues are the only thing it is absolutely impossible to gather any useful information. Your only option is to learn C and debugging tools.
So when I find an issue that I can't be sure if it isn't my own fault, or I don't know the root cause of it, I try to gather some information before heading to the bugzilla. Memory issues are the only thing it is absolutely impossible to gather any useful information. Your only option is to learn C and debugging tools.
A suspected memory-leak is definitely the sort of thing that you should report to the authors/maintainers. Even if you think it is triggered by something that you are doing wrong, their program should not respond to that incorrect input by leaking memory.
Places like stackoverflow and IRC support channels are really only useful if the answer you are looking for can probably be found in some form of documentation somewhere. For things like suspected bugs or issues with specific implementation, it is best to go to the source (either the actual source, or the responsible devs (which almost always means going to a mailing list).
Some particular projects are infamously dismissive specifically of memory leaks, really no matter how much detail you put into your report (Mozilla, I am looking squarely at you), but those should be the minority. Typically if you see run-away memory usage, and are able to provide enough details about what you are seeing, the developers should be receptive. That's been my experience anyway.
Places like stackoverflow and IRC support channels are really only useful if the answer you are looking for can probably be found in some form of documentation somewhere. For things like suspected bugs or issues with specific implementation, it is best to go to the source (either the actual source, or the responsible devs (which almost always means going to a mailing list).
Some particular projects are infamously dismissive specifically of memory leaks, really no matter how much detail you put into your report (Mozilla, I am looking squarely at you), but those should be the minority. Typically if you see run-away memory usage, and are able to provide enough details about what you are seeing, the developers should be receptive. That's been my experience anyway.
The gap between 'Joe Chucklefuck' and the lead developer for glibc is wide enough to fit a whole lot of people who can triage a bug without necessarily being skilled (or interested) enough to submit patches.
I think describing the phenomenon as 'noise' is reasonable. You find a moderate number of answers with a reasonable degree of certainty as to their accuracy and a large number of answers that are just whatever came to mind first (these aren't necessarily provided by completely distinct groups of people). The result is a lot of almost-but-not-quite-related nonsense that drowns out the answers with any relevant content.
This probably isn't purely a problem of numbers, either. Even attempting to be helpful takes a certain amount of effort that naturally limits the impact of the former group at any particular instant. It's also easy (as evidenced by this very article) to take helpful information and render it non-helpful by repeating it without context.
I think describing the phenomenon as 'noise' is reasonable. You find a moderate number of answers with a reasonable degree of certainty as to their accuracy and a large number of answers that are just whatever came to mind first (these aren't necessarily provided by completely distinct groups of people). The result is a lot of almost-but-not-quite-related nonsense that drowns out the answers with any relevant content.
This probably isn't purely a problem of numbers, either. Even attempting to be helpful takes a certain amount of effort that naturally limits the impact of the former group at any particular instant. It's also easy (as evidenced by this very article) to take helpful information and render it non-helpful by repeating it without context.
Just give more information right off the bat. Don't say "XYZ is chewing up lots of RAM"; the best case scenario if you say that is they have to respond back asking for more information. Give that information to them in the first place "XYZ starts at XXXXMB shared and XXMB resident. Over five minutes it steadily climbs to XXXXMB resident. Shared remains constant at XXXMB."
Doing this, I can't say I've ever been brushed off using this page.
Doing this, I can't say I've ever been brushed off using this page.
Exactly that!
I had similar experience with MacOS memory issues, and all the answers I was getting: "MacOS is superior to Windows, its just Cache, the memory is actually free". Yet my applications could not allocate memory and were hogging whole OS as a consequence.
Back on topic, never had similar 'cache' problems on Linux though.
I had similar experience with MacOS memory issues, and all the answers I was getting: "MacOS is superior to Windows, its just Cache, the memory is actually free". Yet my applications could not allocate memory and were hogging whole OS as a consequence.
Back on topic, never had similar 'cache' problems on Linux though.
Wow, I can't believe how many times I've had to explain this to people as they log on to shared VNC servers. "I thought this had 16G of RAM free, I asked for a 16G machine!". "It does, stop trying to read top!"
Now I'll just point them to this website.
Now I'll just point them to this website.
Also HN ate your server apparently.
I hate the invention of "burstable" RAM on VPSes for this reason. Eventually your IO cache grows until it has most of your guaranteed RAM. Now every new process is playing a game of Russian roulette with your hypervisor to see if it can actually allocate into your "burstable" RAM what the kernel is reporting is freely available.
Actually this is a general issue when running multiple Linux VMs under a hypervisor. In that case you don't want them filling up RAM with prefetch data from disk because that's likely to cause contention and reduce page sharing between VMs.
https://www.kernel.org/doc/Documentation/sysctl/vm.txt
You can drop cache by sending a byte into drop_cache.
You can drop cache by sending a byte into drop_cache.
But why would you do that? (The only time I've ever used that is when benchmarking and you need a clean cache between runs). Otherwise the kernel will just empty it as needed when you run a program.
First, it's not why. It's that this is possible. Please don't say "this is impossible because this doesn't make sense".
Second, I agree with you. Normally people don't need to do this unless they hate their hard drives. :-) But there's some special cases, for example when the disk is "volatile", or when you want a huge chunk of continuous physical address and you want to tell the OS to stay away and don't touch any pages inside.
PS: An interesting thing is that Windows people always seek for a way to enlarge disk cache. It's fine on server versions, but on desktop versions such as Windows 7, if I don't remember wrong, disk cache is capped at ~4GB(not sure if globally or per file). Oh well.
PS: An interesting thing is that Windows people always seek for a way to enlarge disk cache. It's fine on server versions, but on desktop versions such as Windows 7, if I don't remember wrong, disk cache is capped at ~4GB(not sure if globally or per file). Oh well.
I can't answer for everyone, but I can tell you why I have a cron job dropping my cache every several minutes:
The application I develop consumes most of the ram on my machine when running, and takes several minutes to rebuild and start up to begin with. When most of my memory is being used for the cache, this process takes several minutes longer, because I'm making millions upon millions of calls for more memory -- and each one has to get some of that cached memory back for itself. If I simply drop the cache all at once, every minute, it takes a split second. If I shrink it over a million increments, it takes around a minute.
Even typing that I feel I must be doing something wrong; and yet, it worked.
The application I develop consumes most of the ram on my machine when running, and takes several minutes to rebuild and start up to begin with. When most of my memory is being used for the cache, this process takes several minutes longer, because I'm making millions upon millions of calls for more memory -- and each one has to get some of that cached memory back for itself. If I simply drop the cache all at once, every minute, it takes a split second. If I shrink it over a million increments, it takes around a minute.
Even typing that I feel I must be doing something wrong; and yet, it worked.
Your app is making more calls to malloc when there is memory being used by disk cache, or you mean something else by "millions upon millions of calls for more memory"?
You control hope many times you call malloc. Just call it once with what you need.
You control hope many times you call malloc. Just call it once with what you need.
that's funny, like avery complex or "smart" algorithm, if it works it's cool, but when something doesn't work, it makes it very hard to pinpoint a culprit.
I'm more and more inclined to either use stupider algorithm that at least I can explain to the user, incremental stuff where they can check intermediate results or taking into account the fact that this algorithm has to be conveyed to users in one way or another since inception.
I guess in this case a measure of the swap activity would be a good indicator that the current limiting factor is the RAM, but it also has to be credible. Somehow having 16G written on the box of the ram and 7G written in the "available ram" of a software makes everything more credible that saying "I have a pressure of 1.5madeupunit on the ram front". Human factors, again.
I'm more and more inclined to either use stupider algorithm that at least I can explain to the user, incremental stuff where they can check intermediate results or taking into account the fact that this algorithm has to be conveyed to users in one way or another since inception.
I guess in this case a measure of the swap activity would be a good indicator that the current limiting factor is the RAM, but it also has to be credible. Somehow having 16G written on the box of the ram and 7G written in the "available ram" of a software makes everything more credible that saying "I have a pressure of 1.5madeupunit on the ram front". Human factors, again.
I've been doing performance analysis and debugging as part of being a unix/linux sysadmin now for something like 20 years. And even though I have many well-cherished anecdotes about explaining the VM free list to software architects who were the go-to guys for the vision of our entire global platform, complete with PhDs, I still don't want to go back to the bad old days without a shared buffer cache. There's a reason why that's there, and you don't want to go back to 1992-era memory management.
It also makes perfect sense from the perspective of writing the software. The free list really is the list of VM pages that have been free'd and that has a specific meaning, and changing that name internally would be terrible. You just need to understand how the machine actually works.
It also makes perfect sense from the perspective of writing the software. The free list really is the list of VM pages that have been free'd and that has a specific meaning, and changing that name internally would be terrible. You just need to understand how the machine actually works.
Another thing, you know how sometimes a USB stick needs some time to un-mount? This non-sync'ed data (say, a copy of a big file) eventually has to be written to the physical disk, and through the little pipe, takes time. Find out how much of this "dirty" buffer you have yet to physically write to the device, while giving the kernel 'hints' to sync the dirty blocks quickly, with:
while true; do sync && grep Dirty /proc/meminfo; done
This seems a bit magic, what does it do? (stuck in Windows-land at the moment)
I do this a lot, on machines which are annoyingly vendor-tweaked to allow a lot of dirty pages. Copying files to USB sticks happens instantly, but the unmount then takes an age - like, 20 minutes. This is a way to get a measure of the progress of the unmount. I generally have a sleep in there as well though ;)
Best way to check how much ram is actually free is to use a higher level tool like a task manager (from various DEs) or a command line tool like htop, they report the actual memory usage of processes
Not strictly accurate, in the "Why would you want to do this" regard.
Ages ago I was hacking linux onto an ARM soc that claimed to have memory it didn't (it wasn't aware of the display's DMA segment). It was near the top of the address space, so generally speaking it didn't cause too many problems, but until I could debug it I needed to keep memory usage low, wchih meant disabling the disk cache. What a shitshow that was :/
Ages ago I was hacking linux onto an ARM soc that claimed to have memory it didn't (it wasn't aware of the display's DMA segment). It was near the top of the address space, so generally speaking it didn't cause too many problems, but until I could debug it I needed to keep memory usage low, wchih meant disabling the disk cache. What a shitshow that was :/
I always run Ganglia on my servers. You can see memory used, memory shared, memory cached, memory buffered, and memory swapped all over time as well as total in-core memory. This has saved me (and my team) from ever having to misinterpret command line output. Takeaway, Linux is smart about using memory, probably smarter than you. http://ganglia.sourceforge.net/
I think it's still disinformative. Disk cache also uses RAM which is allocted and reserved by applications, of course data is swapped out in this situation. o you'll need more swap, but it isn't a bad thing at all. Even OS/2 did this a long time ago, nothing new at all.
So basically this is the inverse of swap. Using memory to make disk reads/writes faster, instead of using disk to make more memory.
If Linux is consistently doing this, why would you still need a swap partition?
If Linux is consistently doing this, why would you still need a swap partition?
> why would you still need a swap partition?
You don't, but if something decides to grab memory, the OOM-killer will kill processes abruptly. Swap may allow the system to limp along until there's some solution.
It's not like 10 or 20GB HD these days matter much either.
You don't, but if something decides to grab memory, the OOM-killer will kill processes abruptly. Swap may allow the system to limp along until there's some solution.
It's not like 10 or 20GB HD these days matter much either.
Though of course some people prefer the OOM-killer to kick in fast instead of the whole system being unresponsive for about an hour or so.
That's why my desktop and laptop systems don't have swap enabled.
That's why my desktop and laptop systems don't have swap enabled.
Would you rather consume physical pages of RAM by storing less recently used programs and data, or by caching more recently used disk pages?
Myself, I'd generally prefer to evict the older stuff to swap in order to cache the newer stuff. But maybe that's just me.
Myself, I'd generally prefer to evict the older stuff to swap in order to cache the newer stuff. But maybe that's just me.
Quite a while ago, HP-UX (v9, if memory serves) got this balance wrong - the system was keeping files in RAM, and kicking active processes to swap.
Much gnashing of teeth ensued until HP fixed that ...
Much gnashing of teeth ensued until HP fixed that ...
Because you can (and usually do) have areas of memory and disk that are more or less active, meaning it can make sense to evict some ram pages to disk while still caching other disk pages.
> why would you still need a swap partition?
Because sometimes you (read applications) need to use all/most of your memory. Ability to swap out all the not being used memory to disk maximizes amount of memory available to applications.
Because sometimes you (read applications) need to use all/most of your memory. Ability to swap out all the not being used memory to disk maximizes amount of memory available to applications.
You don't. Get rid of it. I've been running without swap for five years now. (Note that suspend-to-disk requires a swap partition though)
[deleted]
FWIW, it's relevant to OS X too (even Mavericks)...'sudo purge' will reclaim some of the resources.
OS X does not behave exactly in this manner, but is indeed confusing.
Linux never needs a 'purge' or equivalent. Doing a 'drop_cache' only slows the system slightly because nothing is cached. 'sudo purge' can prevent OS X from using swap – i.e. running out of RAM even though there is RAM it could reclaim. This behavior has been tweaked but not fixed in Mavericks.
Linux never needs a 'purge' or equivalent. Doing a 'drop_cache' only slows the system slightly because nothing is cached. 'sudo purge' can prevent OS X from using swap – i.e. running out of RAM even though there is RAM it could reclaim. This behavior has been tweaked but not fixed in Mavericks.
Actually, purge is completely unrelated to swap. Purge forces disk cache to be flushed to disk, and can be used to approximate a cold disk buffer cache scenario for performance analysis.
Some small amount of swap usage (compared to total RAM) is perfectly normal, by the way. More cache, for instance, is a better use of RAM than avoiding swap usage by keeping never-accessed pages present.
Some small amount of swap usage (compared to total RAM) is perfectly normal, by the way. More cache, for instance, is a better use of RAM than avoiding swap usage by keeping never-accessed pages present.
Or you can just http://downloadmoreram.com/
(joke)
(joke)
Ehm. But sometimes stupid linux kernel prefers moving something usefull to swap and caching some shit instead of keeping stuff in ram and not doing caching.
So linux sometimes eats your RAM (but in different way).
So linux sometimes eats your RAM (but in different way).
For situations where you have a time-critical app, you can pin the app's memory into RAM with the mlock() family of functions.
There may be a command-line utility that lets you pin an arbitrary running a process too, I haven't looked for it on linux.
There may be a command-line utility that lets you pin an arbitrary running a process too, I haven't looked for it on linux.
I just love the website design :-) It makes me happy.
ITT tech startups learn about Lunix.
right. so there is a bug in linux. gotcha.
No. Well, not technically... There is a UX bug in Linux. Not the only one at that, but they don't count much in Linux-land.
Just a difference in semantics -- I mean, c'mon UX in a utility like top? which is probably specified in some Unix convention anyway. Pure BSD Unix do this as well in their own way.
Free is used, black is white, truth is a lie, lie is a truth.
Leaving your memory with no use at all would be the bug. It is a free resource that could be used to improve system performance at basically no cost (the disk is already read, the memory is still available immediately for any app that request it, etc), so it is a good default to let linux do it.
But could be cases where it could be side effects of it (i.e. running in a VM could make less memory available for the host, and it could be caching disk already) so you could consider to disable it sometimes.
But could be cases where it could be side effects of it (i.e. running in a VM could make less memory available for the host, and it could be caching disk already) so you could consider to disable it sometimes.
it confuses users. its a bug. thank you for the -4 points... :)
Lovely! This domain could iframe that SO question :)
What question?
Why, that one, of course.
I've thought the answer to that one was 42.
ITT: reddit-style comments that don’t get downvoted. I’m all for it.
http://i.imgur.com/lrlFq0y.jpg
http://i.imgur.com/lrlFq0y.jpg
Windows loves to cache and "prefetch" apps, I still need to figure out how to disable prefetching since I have a very fast SSD.
Why? All it does is make it go faster.
And drain your battery for no reason.
Do you have any proof of that? "Free" memory uses as much energy as "used" memory, there is no distinction in SD|DDR-RAM. And if the OS is smart, it only has to keep a simple index of memory pages still relevant. I'd argue that this has a measurable battery-hit.
You're only accounting for the RAM power. Accessing the SSD costs power, too.
Then I think I'd rather Windows prefetched my programs while I'm plugged in so it doesn't have to do so when I'm on battery.
> You're only accounting for the RAM power. Accessing the SSD costs power, too.
A typical modern SSD reads at 500 MiB/s, consumes 3 watts when reading, and 0.3 watts when idle.
A stick of DDR3 RAM consumes about 3 watts, just sitting idle.
The power consumption from prefetching is minor compared to that of the RAM.
A typical modern SSD reads at 500 MiB/s, consumes 3 watts when reading, and 0.3 watts when idle.
A stick of DDR3 RAM consumes about 3 watts, just sitting idle.
The power consumption from prefetching is minor compared to that of the RAM.
Reading from the SSD to laod it into RAM does drain battery. No idea if it is enough to matter though.
Instructions for Win7. http://www.tomshardware.com/reviews/ssd-performance-tweak,29... Why not leave it on, though? RAM is still faster than SSD.
Memory is still an order of magnitude faster than your “very fast SSD”.
And memory is still an order of magnitude more expensive than SSD, per GB.
I just bought 4 GB of PC-6400 for a laptop at £40. A fast 128GB SSD is £100.
Summary: use RAM judiciously
I just bought 4 GB of PC-6400 for a laptop at £40. A fast 128GB SSD is £100.
Summary: use RAM judiciously
All the more reason for your OS to use memory aggressively. Surely you don't want your expensive investment sitting around just doing nothing?
Unused memory is wasted memory.
Unused memory is wasted memory.
The prefetched data is managed by the OS. If a program needs the RAM, the prefetched data is immediately discarded. Prefetching never creates RAM pressure.
This. It blows my mind how many people who write software don't understand how OS caching and virtual memory work. They are great filter interview questions ;)
just one?
At least one. Potentially more. The actual multiple depends strongly on the specific SSD and memory configurations in question, and on the benchmark of interest.
Windows7 are SSD aware. You dont have to do anything about it.
I've actually had buggy desktop applications use several gigabytes of RAM as shared memory, so this can make a big difference.