I've got 4GB of RAM!!1! No you don't...

I thought this might interest some of you who considered getting 4GB of RAM - it's always good to know just how much you're gonna get out of the stuff you buy... This isn't necessarily news, just a (hopefully useful/interesting) FYI (I would hazard a guess that not all our patrons are aware of this...)

The article has some links, in case you are interested in a further read, but the links didn't make it into the pasted version below, so you'll need to go to the original article to get those.

http://blogs.msdn.com/dcook/archive/...my-memory.aspx

Spoiler Alert, click show to read: 

Who ate my memory?

The consumer versions of 32-bit Windows XP and Vista have a stated limit of 4 GB RAM, but a practical limit of about 3.1 GB. A lot of partial explanations have been floating around, so I thought I would try my hand at clearing up the issue. (Wish me luck!)

The design of the Intel 386 architecture supported access to up to 4 GB of physical memory (32-bit physical addresses) and unlimited virtual memory (4 GB at a time via 32-bit virtual addresses). 4 GB of physical memory seemed quite unthinkable at the time the chip was released, so the actual CPU did not have enough address pins to actually do this. Back then a 32-bit address space seemed extravagant for anything less than a supercomputer or mainframe. Nowadays, you can get 4 GB for under $400, and what was unthinkable in 1986 is within reach of anybody thinking about a new computer.

So at least I can access 4 GB, right? Nope.

The original IBM PC’s processor could access 1024 KB of physical address space, but you could only use 640 KB for RAM. The remaining 384 KB of address space was reserved for memory-mapped hardware and ROM. A similar situation exists with current systems: hardware reserves large chunks of the upper 1 GB of physical address space. Because of these reserved areas, a system with a 32-bit physical address space will be limited to somewhere around 3.1-3.5 GB of RAM.

To overcome the 32-bit limitation, recent x86 CPUs (Pentium Pro and later) have 36 address pins and can address 64 GB of RAM. The original design of the x86 32-bit protected mode only provided access to 32-bit addresses, so PAE (Physical Address Extensions) mode was created to allow access to 36-bit addresses.

PAE mode changes the layout of the page tables. Page tables map virtual addresses to physical addresses. Without PAE, the 32-bit virtual addresses map through 2 levels of page tables (1 level for huge pages) and are translated to 32-bit physical addresses. With PAE, the 32-bit virtual addresses map through 3 levels of page tables (2 levels for huge pages) and are translated to 64-bit physical addresses.

PAE doesn’t do anything to the virtual memory limit. Pointers are still 32 bits, so a process can only access 4 GB of address space at a time. However, using PAE, two or more processes could each access a different 4 GB of physical memory. With proper operating system support (i.e. AWE on Windows operating systems) PAE also allows a process to allocate additional memory outside its normal address space, then swap portions of that additional memory into its address space as needed.

So PAE is the answer, right? Well, maybe…

One thing that can prevent access to more than 4 GB of RAM is motherboard design. PAE can only access 64 GB of memory if all 36 address pins are properly wired up on the motherboard. This is not always the case, since those extra 4 wires make the motherboard just a little bit more expensive to design and manufacture (and use just a little bit more power). Many motherboards (especially on laptops) only have 32 address pins connected. If that is the case, no OS will be able to access more than 4 GB of address space.

Another hardware limitation is the ability of the chipset to remap RAM. If you have 4 GB of RAM, and 600 MB of address space is used up by PCI/AGP reserved areas, the only way to access the top 600 MB of RAM is to remap it into the addresses above the 4 GB boundary. Not all chipsets are able to do this, so some systems will just waste any RAM that happens to be shadowed by a PCI/AGP reserved region.

My BIOS reports 4 (or more) GB of RAM, I’ve enabled PAE, and I still only see 3.1 GB. What gives?

Unless you’re running one of the advanced server varieties of Windows, you won’t see more than 4 GB of physical memory. This is a limitation of Windows designed (I assume) to encourage people building expensive servers to pay more for Windows than those who are using it for normal day-to-day activities.

As for that last 0.9 GB, it all comes down to drivers and system stability. Not all drivers behave well in the presence of 64 bit physical addresses. Many driver authors assume that only the bottom 32 bits of the physical address are valid. Others don't properly handle the creation of bounce buffers when necessary (they’re needed when transferring data from a hardware device to/from a buffer that is above the 4 GB mark in physical memory).

Windows XP originally supported a full 4 GB of RAM. You would be limited to 3.1-3.5 GB without PAE, but if you enabled PAE on a 4 GB system with proper chipset and motherboard support, you would have access to the full 4 GB. As more people began to take advantage of this feature using commodity (read: cheapest product with the features I want) hardware, Microsoft noticed a new source of crashes and blue screens. These were traced to drivers failing to correctly handle 64-bit physical addresses. A decision was made to improve system stability at a cost of possibly wasting memory. XP SP2 introduced a change such that only the bottom 32 bits of physical memory will ever be used, even if that means some memory will not be used. (This is also the case with 32-bit editions of Vista.) While this is annoying to those who want that little bit of extra oomph, and while I would have liked a way to re-enable the memory “at my own risk”, this is probably the right decision for 99.9% of the general population of Windows users (and probably saves Dell millions in support costs). See the relevant KB article and a TechNet article for details.

Some of the server Operating Systems still allow the use of larger amounts of memory. I’m guessing that this is done with the assumption that higher quality parts will be used and drivers will be more likely to have been tested in PAE mode with large addresses.

Side-note: PAE is also related to page execution protection, called "hardware DEP" (Microsoft term), "NX" (AMD term), and "XD" (Intel term). In 32-bit x86 processors, this can only be used in PAE mode. This is why you might see PAE mode used even on systems with less than 4 GB of memory.

Performance note: 3-level page table lookups are inherently slower than 2-level page table lookups. However, the processor has substantial dedicated circuitry that usually eliminates most of the performance impact.

I will note that I pointed out in one of my posts here, a while ago (which I'm too lazy to look for now), that Linux has similar limitations (although in a slightly different way). The main difference is that with Linux you can increase the amount of RAM accessed by a process (by decreasing the amount reserved for the kernel), but it's a pretty ugly hack, and you need to know what you're doing.

The gist of this post is that you should be aware that if you get 4GB of RAM, neither OS will be able to actually benefit from that properly, and your applications will effectively be using a little over 3GB.

Except if you get a 64bit OS.

Yes, i was about to burst your bubble on reading the title, but you've done it for me.

'i herd theres rumors on those internets....that 4gb of RAM is a waste of moneys...'

Quote:

---

Originally Posted by Husar

Except if you get a 64bit OS.

---

Yes, that is correct. Wonder why not everybody is running them...

Quote:

---

Originally Posted by Mikeus Caesar

Yes, i was about to burst your bubble on reading the title, but you've done it for me.

'i herd theres rumors on those internets....that 4gb of RAM is a waste of moneys...'

---

What's the market share for 64bit OSes ? Have 32bit OSes disappeared ? No ? Well, then, I guess the issue is still relevant.

Quote:

---

Originally Posted by article

Unless you’re running one of the advanced server varieties of Windows, you won’t see more than 4 GB of physical memory. This is a limitation of Windows designed (I assume) to encourage people building expensive servers to pay more for Windows than those who are using it for normal day-to-day activities.

---

Rubbish; 64bit vista (client) is stable enough to run for the consumer.

Quote:

---

Originally Posted by sapi

Rubbish; 64bit vista (client) is stable enough to run for the consumer.

---

Oi! What did I say about how new this is ? ~D

Quote:

---

Originally Posted by The Article

Published Sunday, March 25, 2007 3:03 AM by dcook

---

Furthermore, I find it a bit sad/strange that so far all the comments in this thread have focused on something completely irrelevant (or minor at best) to the main point of the article...
Not to mention that I posted this as an item of interest, not as one of argument... Who is arguing that 64bit OSes do not have this problem ? The main point is that 32bit ones DO have this problem, and since the overwhelming majority of people are still running 32bit OSes on this forum, it is relevant to them, and applicable to their OSes.

Finally, before you dismiss what he's saying as "rubbish", you might wanna check his credentials. He's not some shill paid by the Linux/Mac community, or some anti-MS fanatic/zealot:
http://blogs.msdn.com/dcook/about.aspx

Quote:

---

Originally Posted by The Link

I'm Doug Cook. I've been at Microsoft for about five and a half years now. (Six years if you count the internship, but I've learned that internships don't count for much, so it's five and a half.) I've been on the Platform Builder ("PB") IDE team since I started, so I'm kind of the old fogey of PB. I've successfully avoided management positions so far (with great power comes great responsibility).

---

Quote:

---

Oi! What did I say about how new this is ?

---

Whoops, missed that :grin2:

Well, I know for a fact that 64 bit vista wasn't stable then, so ~:)

Erm, hey, sorry, just wanted to add that, didn't want to say you're incorrect or that your point was invalid, just think this way more people may get a 64bit OS once they realise that 32bit is gone, over, old, outdated, not useful anymore and also not to be encouraged anymore. ~D

Quote:

---

Originally Posted by Husar

Erm, hey, sorry, just wanted to add that, didn't want to say you're incorrect or that your point was invalid, just think this way more people may get a 64bit OS once they realise that 32bit is gone, over, old, outdated, not useful anymore and also not to be encouraged anymore. ~D

---

No, yours was a very good and valid point, I should have said that myself (I showed the problem, but not the solution, and this IS the best solution one can choose - to switch to a 64bit OS; the others are more-or-less effective hacks). So, nothing to be sorry for at all.

Not many people upgrade their own PC and those that do are generally aware of the pitfalls of mixed hardware, OS and application limitations.

Those who go and buy a new middle of the line PC with 64-bit architecture are going to be fine.

The ones who are going to have issues are those running 32-bit XP and buy some extra RAM. Mind you the first issue they will run into is not that the 4GB only works as 3.1 (mind you the extra 0.9 is speeding up things by being dedicated)... it is that they can't find their original XP disk and license number from half a decade ago.

It's a complete mystery to me why 64-bit OSs aren't the standard already. I'm in the process of buying a new PC, and ordered (for free) 64-bit Ubuntu.
32-bit is at a dead end, and we've known it was coming for quite a while now. Most hardware has been 64-bit for quite some time, 64-bit OSs have existed for quite some time, and on 64-bit hardware, a 64-bit OS is faster than a 32-bit OS.

So why the hell is 32 bit still the standard? If they still want to sell it, sell it as an explicit "legacy" version for older hardware (on which Vista doesn't work anyway).

Few people actually use 4GBs of ram, and to most people, trading the ability to use extra ram that they don't actually have is not worth the tradeoff of application fussiness you get with any kind of switchover.

In short, to most people, there's no actual gain, and compatibility risks. That's why it's not standard.