Anatomy of a Program in Memory(duartes.org)
duartes.org
Anatomy of a Program in Memory
http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory
23 comments
Submit more articles like this one and less Web 2.0 gossip please.
According to the comments on that article, the people still don't understand the basics of virtual memory.
"Great article! / Quite shocked to know that windows takes double the kernel memory compared to Linux."
"I’m wondering, though, why does the kernel space consume 1gb? That seems like a lot.."
"Great article! / Quite shocked to know that windows takes double the kernel memory compared to Linux."
"I’m wondering, though, why does the kernel space consume 1gb? That seems like a lot.."
As narag says, it's only recently that this has become an issue. When physical memory size was orders of magnitude smaller than than the address space, it didn't matter that the kernel carved out 1GB of address space for itself. User applications still had the other 3GB, which was far more than any system had physical memory.
It's only during this limbo period when 4GB of RAM is relatively cheap, yet there are still 32-bit systems kicking around, that it becomes an issue. On a 64-bit system with a 16 exabyte address space, 1GB is a drop in the bucket. Once again, it won't matter.
(I suspect Linux allocates more than this for itself in 64-bit systems, but I don't know the details. At any rate, out of 16EB, even 100GB would hardly be noticed.)
It's only during this limbo period when 4GB of RAM is relatively cheap, yet there are still 32-bit systems kicking around, that it becomes an issue. On a 64-bit system with a 16 exabyte address space, 1GB is a drop in the bucket. Once again, it won't matter.
(I suspect Linux allocates more than this for itself in 64-bit systems, but I don't know the details. At any rate, out of 16EB, even 100GB would hardly be noticed.)
From the way I read those comments, I gathered that the comments implied that Windows "consumed" two gigs of ram, where as Linux took 1 gig. I thought there was a dissonance between how people understood "virtual memory," which is essentially a mapping function, to "memory," which people assume to be the physical ram.
It's easy to make that mistake at first, but it is a very important distinction. Most of the time, for most processes, most of that space is unallocated. However, since it's basically impossible to change the size of the kernel area without relinking the process binaries, and since that area is shared between all processes, the size of that area has to remain fixed.
It is interesting that Linux and NT differ. Perhaps this is due to the fact that NT needs more kernel memory to house, e.g. the windowing system, whereas Linux systems put X in userspace? I'm speculating here.
Something I never considered: that design difference probably makes Wine's job easier, because between the 2GB and 3GB marks is a region of memory the Windows process assumes is inaccessible, yet in Linux is in userspace. They could (and probably do) load the Wine libraries here without interfering with the Windows process. It also means that it would be very difficult to implement Wine's complement, a way of running Linux binaries on Windows.
It is interesting that Linux and NT differ. Perhaps this is due to the fact that NT needs more kernel memory to house, e.g. the windowing system, whereas Linux systems put X in userspace? I'm speculating here.
Something I never considered: that design difference probably makes Wine's job easier, because between the 2GB and 3GB marks is a region of memory the Windows process assumes is inaccessible, yet in Linux is in userspace. They could (and probably do) load the Wine libraries here without interfering with the Windows process. It also means that it would be very difficult to implement Wine's complement, a way of running Linux binaries on Windows.
Linux used to be 2GiB/2GiB too, just like NT. It changed because people with big processes were unhappy; for a while there was an unofficial kernel patch for the 3GiB/1GiB split. Maybe NT didn't change because not as many people were running supercomputers on NT, or because changing NT is harder.
You don't have to relink your userland binaries to do this because they don't contain virtual addresses of kernel data structures, only their own data structures. Expanding the space available for mmap won't force those data structures to move.
You can pretty much load Wine libraries wherever you want; it's pretty rare for a program to break if memory it assumed was unmapped gets used for a library, and programs that break that way will have a lot of trouble when Windows versions change too. Generally programs are only linked with knowledge of where they themselves will be loaded in virtual memory. (Linux ELF shared libraries don't even know that.) All the other addresses are given to the program at startup. It is unusual for Linux binaries to believe that they will be loaded above 2GiB; the usual start address is 0x08048000.
Here's the memory map of a process running under Wine: http://gist.github.com/53853 --- note that there's a lot of Wine and X11 stuff loaded below 2GiB (0x80000000).
You don't have to relink your userland binaries to do this because they don't contain virtual addresses of kernel data structures, only their own data structures. Expanding the space available for mmap won't force those data structures to move.
You can pretty much load Wine libraries wherever you want; it's pretty rare for a program to break if memory it assumed was unmapped gets used for a library, and programs that break that way will have a lot of trouble when Windows versions change too. Generally programs are only linked with knowledge of where they themselves will be loaded in virtual memory. (Linux ELF shared libraries don't even know that.) All the other addresses are given to the program at startup. It is unusual for Linux binaries to believe that they will be loaded above 2GiB; the usual start address is 0x08048000.
Here's the memory map of a process running under Wine: http://gist.github.com/53853 --- note that there's a lot of Wine and X11 stuff loaded below 2GiB (0x80000000).
It's pretty arbitrary really. It's rare to need more than 2 gigs of memory in a single user process. On the other hand, the more address space you have in the kernel the easier it is to use spare memory for IO buffer caching.
From the way I read those comments, I gathered that the comments implied that Windows "consumed" two gigs of ram,
I'm afraid I didn't make myself clear before. I also think that commenters were wrong. I just tried to explain why. The article says that's the default arrangement, but if non-kernel memory grows bigger than 2 GB, I assume that it'll be allocated in the virtual space that's unused by kernel.
I'm afraid I didn't make myself clear before. I also think that commenters were wrong. I just tried to explain why. The article says that's the default arrangement, but if non-kernel memory grows bigger than 2 GB, I assume that it'll be allocated in the virtual space that's unused by kernel.
Address space is not much bigger than physical memory now for 32 bits. I've read that my laptop could in fact have 4GB instead of the nominal 3GB. So even if the memory is not "consumed", the default must be changed at run time, unless the kernel space literally consumes 2/3 of available memory.
Given you don't reference what operating system is in use, you're probably using Microsoft Windows here. The implementation of virtual memory has common traits, but details vary by OS and even OS version.
Physical address space and virtual address space are two different aspects of system design, and it is completely feasible to have these values be the same, or to have either of the two be larger than the other. Yes, you can have a 32-bit virtual memory system and 34 or more bits of physical memory, and there can be good reasons to do that, too.
It's also feasible to have address "holes" in virtual address space, and "holes" in physical address space. Address ranges or gaps that are simply not implemented by the particular processor, or that are not instantiated by the memory controller. But I digress.
Save for low-end boxes, all current gear is 64-bit virtual addressing (variously with "holes"), and most current systems can have 48 bits (x86-64) or 50 bits (IA-64) or other implementation-specific ranges for physical addressing. Details vary by processor and by implementation. Available memory varies by budget - 50 bits of memory is expensive - and target market for the box.
And consuming address space is different than consuming memory. On various systems, address space is an inexpensive resource. And on a virtual memory system, memory is limited by physical memory and by backing storage and by your willingness to wait for the transitions between these two resources; for memory paging to occur.
And what your application thinks is in memory might be on disk. When referenced, the data is valid or is paged in and made valid.
Parts of an OS might be memory-resident, and parts can be paged. Applications tend to be paged.
If you're programming in an area that has performance constraints, knowing details of the particular virtual memory and physical memory implementation can be a key to success. Issues around data alignment, multiprocessor memory caching and data structure layout can all be relevant; having a large and improperly-designed data structure can lead to memory paging for each data structure reference, for instance.
Physical address space and virtual address space are two different aspects of system design, and it is completely feasible to have these values be the same, or to have either of the two be larger than the other. Yes, you can have a 32-bit virtual memory system and 34 or more bits of physical memory, and there can be good reasons to do that, too.
It's also feasible to have address "holes" in virtual address space, and "holes" in physical address space. Address ranges or gaps that are simply not implemented by the particular processor, or that are not instantiated by the memory controller. But I digress.
Save for low-end boxes, all current gear is 64-bit virtual addressing (variously with "holes"), and most current systems can have 48 bits (x86-64) or 50 bits (IA-64) or other implementation-specific ranges for physical addressing. Details vary by processor and by implementation. Available memory varies by budget - 50 bits of memory is expensive - and target market for the box.
And consuming address space is different than consuming memory. On various systems, address space is an inexpensive resource. And on a virtual memory system, memory is limited by physical memory and by backing storage and by your willingness to wait for the transitions between these two resources; for memory paging to occur.
And what your application thinks is in memory might be on disk. When referenced, the data is valid or is paged in and made valid.
Parts of an OS might be memory-resident, and parts can be paged. Applications tend to be paged.
If you're programming in an area that has performance constraints, knowing details of the particular virtual memory and physical memory implementation can be a key to success. Issues around data alignment, multiprocessor memory caching and data structure layout can all be relevant; having a large and improperly-designed data structure can lead to memory paging for each data structure reference, for instance.
I really don't understand. At home I've got an Asus F6V. At work it's Asus G50V. Both with Vista/Kubuntu dual boot. I really don't know what's happening with 32/64 bits here. Both use 32 bits OSs but G50V (64 bit machine) has 4GB, so it seems it isn't OS the piece that decides memory addressing capabilities :-)
Memory addressing capabilities are determined by the processor. A 32-bit processor can only use up to 32-bits to represent an address. That means it can address 2^32 (4,294,967,296; 4GB) distinct bytes. But the memory is managed by the OS. So it's possible to have an OS that won't use more than 4GB even though the processor can handle more than that.
Also, most modern x86 processors have a hack - Physical Address Extension (PAE) - that let the 32-bit processors address up to 64 GB.
Also, most modern x86 processors have a hack - Physical Address Extension (PAE) - that let the 32-bit processors address up to 64 GB.
Totally worth the price of admission. Every linux dev, especially c/c++ programmers, should be required to read this post. Once a year.
That blog is a gold mine. Subscribed.
I'd like to understand what's being described in this article, but unfortunately I lack the necessary educational background. Does anyone have any links to other resources that will allow me to better understand what is being discussed in this post?
Virtual memory is part of the fields of computer architecture and operating systems; popular textbooks are Computer Architecture: A Quantitative Approach and Modern Operating Systems.
http://books.google.com/books?id=pqYl3SWkA64C http://books.google.com/books?id=tqtZGQAACAAJ
http://books.google.com/books?id=pqYl3SWkA64C http://books.google.com/books?id=tqtZGQAACAAJ
CPUs prior to 386 had direct access to physical memory. From 386 on, you could enable protected mode (default for Linux and Windows 95+) in which a pointer doesn't point to a physical location but to a virtual address that system translates to physical memory or disk-cached memory. That's the most basic fact. With much luck, that's the only thing you need to know to understand it :-)
This article is already a very high level description of Linux memory management (specifically on 32 bit Intel processors). If you want to move down to lower levels of abstraction and learn more details, then keep reading that blog! On the other hand, if you want a less technical description of these concepts then I'm not sure where to look.
> This helps prevent pointer bugs, though not as effectively as avoiding C in the first place.
I hate to sound like the average Reddit poster and write something like "quoted for truth", but... quoted for truth.
I hate to sound like the average Reddit poster and write something like "quoted for truth", but... quoted for truth.
I found this the strangest and most out-of-place statement in the whole article.
if you're interested in the security aspects of this stuff, there are a number of presentations by openbsd developers on the openbsd site (http://openbsd.rt.fm/papers/):
http://openbsd.rt.fm/papers/nycbsdcon08-pie/
http://openbsd.rt.fm/papers/ven05-deraadt/index.html
http://openbsd.rt.fm/papers/nycbsdcon08-pie/
http://openbsd.rt.fm/papers/ven05-deraadt/index.html