64 Bit: Why we Need it
darkhunt333
Join Date: 2012-11-01 Member: 165414Members, Reinforced - Shadow
Join Date: 2012-11-01 Member: 165414Members, Reinforced - Shadow
(~ I am a bit tired and this is some research I did, let me know if anything is off, and Feel free to correct me. Most is pulled from Microsoft and Wiki)
Why 64 bit will correct a lot of issues with Alec Baldwin.
I will start with the biggest meanest elephant in the room, memory limits of 32 bit OS and applications. Whenever graphics get degraded and your FPS is low, this is why:
- The 4GB Limit
32bit OS and 32 Bit programs like Natural Selection 2 all have a maximum of 4 Gigs. The limit is Max due to running out of addresses to assign(think of it like phone numbers). You would think the 64 bit OS is smart enough to keep infastructure required to run 32bit apps out of the program address space. Nope. Because the 32bit program wouldn't be able see the address in order to communicate with the OS. 32bit apps use 2 GB by default. If the application is large address space aware (linked with /LARGEADDRESSAWARE), it gets 3GB~.
- The Problem
So what are we left with? Nothing. Vram and OS both use ram addressess regaurdless of LAA on. Addresses beyond 4GB dont exist to a 32 bit apps/OS(Think of it like being out of phone numbers, 64bit is like adding an area code). That powerfull vcard with 3 gb of ram? Useless, causing a massive loss in performance. Good CPU? It'll be busy re-mapping addresses to avoid crashing. What the process does when its memory allocations fail. Typically, the program will try to squeeze its use of memory by throwing away anything it doesn't need(Like graphics), unmapping files it wasn't using, and so on. But ultimately, it will simply have to refuse to do things.
The system will map your graphics card into the cpu's address space. 32bit systems can only utilize 4GB at a time, but if it's PAE enabled, the system actually has a 37-bit limit on physical addresses, so depending on how windows virtualizes the ram for a different processes you can effectively use all of it, just not at once.
- Windows Version Dependencies - Why PAE is Never On
http://en.wikipedia.org/wiki/3_GB_barrier#Windows_version_dependencies
"These operating systems do not permit addressing of physical memory above the 4 GB address boundary. This is not an architectural limit; it is a limit imposed by Microsoft. Either the BIOS simply disables the conflicting RAM; or, the BIOS remaps the conflicting RAM to physical addresses above the 4 GB point, but x86 Windows client editions refuse to use physical addresses higher than that, even though they are running with PAE enabled. The conflicting RAM is therefore unavailable to the operating system whether it is remapped or not."
Here is a quote from MSDN explaining:
"Devices have to map their memory below 4 GB for compatibility with non-PAE-aware Windows releases. Therefore, if the system has 4GB of RAM, some of it is either disabled or is remapped above 4GB by the BIOS. If the memory is remapped, X64 Windows can use this memory. X86 client versions of Windows don’t support physical memory above the 4GB mark, so they can’t access these remapped regions. Any X64 Windows or X86 Server release can. X86 client versions with PAE enabled do have a usable 37-bit (128 GB) physical address space.
The limit that these versions impose is the highest permitted physical RAM address, not the size of the IO space. That means PAE-aware drivers can actually use physical space above 4 GB if they want. For example, drivers could map the "lost" memory regions located above 4 GB and expose this memory as a RAM disk."
REAL (not the troll version lol) TLDR:
On 64 bit, 32 bit applications can assign 2^32 = 4294967295(4GB) memory addresses (w/LAA) just like a 32bit OS, this max NEVER changes. Vram also uses addresses from the same pool. So you got windows adding its stuff(takes about 0.5~GB), the game, and VRAM. All eating the same 4gb address limit.
Why can't windows and VRAM use the free space on the 64 bit side? The 32 bit application wouldn't know they existed if something got assigned outside the programs 4294967295 possible memory addresses. So it has to stay within the limit of 4gb.
Example:
The old phone numbers were 7 digits XXX-XXXX(32bit). If an area issues all the phone numbers, it obviously can't issue anymore without using the same ones twice. So an area code is implemented, 3 more digits are added (123)456-789X. This is what 64bit does by increasing addresses to 16,000,000,000,000,000,000 or 16 exbibytes vs 4GB of RAM. If "Mr. 32bit CPU" has an old phone that only allows 7 digits. He can't never call someone with an area code until he gets a newer one from "Ms. 64 bit". lol
Why 64 bit will correct a lot of issues with Alec Baldwin.
I will start with the biggest meanest elephant in the room, memory limits of 32 bit OS and applications. Whenever graphics get degraded and your FPS is low, this is why:
- The 4GB Limit
32bit OS and 32 Bit programs like Natural Selection 2 all have a maximum of 4 Gigs. The limit is Max due to running out of addresses to assign(think of it like phone numbers). You would think the 64 bit OS is smart enough to keep infastructure required to run 32bit apps out of the program address space. Nope. Because the 32bit program wouldn't be able see the address in order to communicate with the OS. 32bit apps use 2 GB by default. If the application is large address space aware (linked with /LARGEADDRESSAWARE), it gets 3GB~.
- The Problem
So what are we left with? Nothing. Vram and OS both use ram addressess regaurdless of LAA on. Addresses beyond 4GB dont exist to a 32 bit apps/OS(Think of it like being out of phone numbers, 64bit is like adding an area code). That powerfull vcard with 3 gb of ram? Useless, causing a massive loss in performance. Good CPU? It'll be busy re-mapping addresses to avoid crashing. What the process does when its memory allocations fail. Typically, the program will try to squeeze its use of memory by throwing away anything it doesn't need(Like graphics), unmapping files it wasn't using, and so on. But ultimately, it will simply have to refuse to do things.
The system will map your graphics card into the cpu's address space. 32bit systems can only utilize 4GB at a time, but if it's PAE enabled, the system actually has a 37-bit limit on physical addresses, so depending on how windows virtualizes the ram for a different processes you can effectively use all of it, just not at once.
- Windows Version Dependencies - Why PAE is Never On
http://en.wikipedia.org/wiki/3_GB_barrier#Windows_version_dependencies
"These operating systems do not permit addressing of physical memory above the 4 GB address boundary. This is not an architectural limit; it is a limit imposed by Microsoft. Either the BIOS simply disables the conflicting RAM; or, the BIOS remaps the conflicting RAM to physical addresses above the 4 GB point, but x86 Windows client editions refuse to use physical addresses higher than that, even though they are running with PAE enabled. The conflicting RAM is therefore unavailable to the operating system whether it is remapped or not."
Here is a quote from MSDN explaining:
"Devices have to map their memory below 4 GB for compatibility with non-PAE-aware Windows releases. Therefore, if the system has 4GB of RAM, some of it is either disabled or is remapped above 4GB by the BIOS. If the memory is remapped, X64 Windows can use this memory. X86 client versions of Windows don’t support physical memory above the 4GB mark, so they can’t access these remapped regions. Any X64 Windows or X86 Server release can. X86 client versions with PAE enabled do have a usable 37-bit (128 GB) physical address space.
The limit that these versions impose is the highest permitted physical RAM address, not the size of the IO space. That means PAE-aware drivers can actually use physical space above 4 GB if they want. For example, drivers could map the "lost" memory regions located above 4 GB and expose this memory as a RAM disk."
REAL (not the troll version lol) TLDR:
On 64 bit, 32 bit applications can assign 2^32 = 4294967295(4GB) memory addresses (w/LAA) just like a 32bit OS, this max NEVER changes. Vram also uses addresses from the same pool. So you got windows adding its stuff(takes about 0.5~GB), the game, and VRAM. All eating the same 4gb address limit.
Why can't windows and VRAM use the free space on the 64 bit side? The 32 bit application wouldn't know they existed if something got assigned outside the programs 4294967295 possible memory addresses. So it has to stay within the limit of 4gb.
Example:
The old phone numbers were 7 digits XXX-XXXX(32bit). If an area issues all the phone numbers, it obviously can't issue anymore without using the same ones twice. So an area code is implemented, 3 more digits are added (123)456-789X. This is what 64bit does by increasing addresses to 16,000,000,000,000,000,000 or 16 exbibytes vs 4GB of RAM. If "Mr. 32bit CPU" has an old phone that only allows 7 digits. He can't never call someone with an area code until he gets a newer one from "Ms. 64 bit". lol
Comments
Another thing is, going 64bit is probably not easy and there would probably be problems with drivers/OS/display compatibility. We have enough bugs as it is.
Also, they can't just leave 32bit, because that's the minimal system requirement they revealed on release. Otherwise they would have to return the money or something.
Anyways, they will fix it on their own, I want to believe.
As for memory usage: One round of ns2_veil causes a peak of 1.7/2.1/1.6 GB (private bytes, virtual size, working set).
So no, the system requirements are not right and no system with 1/2 GB even of free RAM will be able to run this game.
64 bit does have other advantages though. If you compile your executables for 64bit, the compiler will autoamtically be able to use certain features of the CPU like SSE 4 etc, because he knows those features are present. By default with 32bit those advanced features are not used. So yes, move to 64 bit, but not just for ns2
SSE4 is a processor instruction set and can be used without 64 bit.
SSE4 is only available on newer CPUs and if enabled, CPUS without SSE4 won't be able to run NS2. If they want it to be backwards compatible, they need to ship SSE4 and non SSE4 libraries.
SSE4 may only provide a speed boost IF ns2 actually can make use of any of the instructions provided in the instruction set. If it doesn't, there won't be a difference. Also chances are difference might be neligble.
Well, no. The problem is virtual address space. On x86 Windows this is limited to 2 GB (up to 3 GB with the LARGE ADDRESS AWARE switch) per process.
Typically you cannot re-organize allocated memory blocks because then all pointers would become invalid, so after some time the fragmentation can be high enough to exhaust the virtual address space.
Look at what I posted above. A round on ns2_veil with ~16 players caused a peak of 2.1 GB virtual memory on my Windows 7 Pro x64 (amd64)! Luckily the limit is 4 GB of virtual address space for 32 bit processes with the switch thanks to amd64, else it would have crashed as well (2 GB limit like with x86)!
Btw, with 64 bit the limit is 8 TB = 8192 GB. With that you will never run out of virtual address space. So yes, 64-bit would fix the problem.
Did you understand my reply? Virtual address space is severely limited for 32 bit processes regardless if you're on a x86 or amd64 (x64) OS. Sure, with Windows x64 you get 1 GB more virtual address space but even that extra GB may be exhausted due to fragmentation.
64 bit processes are virtually unlimited.
Even if you had 8 GB free RAM, a 32 bit process could still fail allocating a chunk that is only a couple of megabytes big.
With the latest patches the crashes are a lot rarer on my system. The optimizations are quite noticeable and kinda annoying though: extremely blurred textures until the higher resolution ones are loaded, missing sprites .. until they finally get loaded, etc.
I'd really prefer a build where all of that could be loaded when the map is loaded, so that no on-demand loading needs to be done during a play. Quite annoying when you get that nasty FPS drop on the first marine/alien encounter because something needs to be loaded on demand..
Read the OP guys. I have gone over this.
64 Bit does NOT resolve the issue, it addresses a symptom of the issue.
Hey boss please once over the OP. You might not be entirely correct. Looks like you think 32 bit apps read more memory on 64 bit operating systems. Or I'm reading this wrong. ;/
TLDR Version
On 64 bit, 32 bit applications can assign 2^32 = 4294967295(4GB) memory addresses (w/LAA) just like a 32bit OS, this max NEVER changes. Vram also uses addresses from the same pool. So you got windows adding its stuff(takes about 0.5~GB), the game, and VRAM. All eating the same 4gb address limit.
Why can't windows and VRAM use the free space on the 64 bit side? The 32 bit application wouldn't know they existed if something got assigned outside the programs 4294967295 possible memory addresses. So it has to stay within the limit of 4gb.
Example:
The old phone numbers were 7 digits XXX-XXXX(32bit). If an area issues all the phone numbers, it obviously can't issue anymore without using the same ones twice. So an area code is implemented, 3 more digits are added (123)456-789X. This is what 64bit does by increasing addresses to 16,000,000,000,000,000,000 or 16 exbibytes vs 4GB of RAM. If "Mr. 32bit CPU" has an old phone that only allows 7 digits. He can't never call someone with an area code until he gets a newer one from "Ms. 64 bit". lol
I could be wrong though. Went over 16 hours of material last night tring to understand all of this. I would love a dev to explain in finer detail. I'm interested now. :P
@Omega_K2: The real issue is that NS2 needs more virtual memory than 32 bit OS' can provide which is most probably due to the architecture of the engine.
The on-demand loading we see in this patch (maybe already in the previous patch) is imho a workaround. It's an attempt to only allocate stuff when you absolutely need it, but it will still fail when you need a lot of stuff at roughly the same time... Actually, allocating and freeing stuff all the time could cause more fragmentation in the long run...
64 bit *would* resolve this issue and also make development easier, because they could just load the stuff on map change, like in older versions, and don't mess around with the on-demand crap.
As darkhunt333, performance could actually improve too. On my non-SSD system it definitely would, because I get crazy FPS drops everytime the on-demand loading starts..
I rewrote a tldr version for you. I was editing.
I'm running XP 32 bit, 4Gb, Intel Core 2 Duo E6600 Conroe 2.4GHz.
NS2 runs fine 75 percent of the time now on build 257 but it will eventually crash. *Which is expected given the "memory management fragmentation" issues.
PS - (in case you read any of my other posts) I do plan to upgrade to Win 7 64.
I just wish they would add an option to disable female marine model, that seems to be the biggest memory hog what with it having its own complete set of distinct animations. Also upgrading NS2 to 64bit might be bloody hard.
I have experienced weird... lag and latency spikes when my ns2 was pushing 2GB~ so perhaps those problems are mem related also. (yes, latency.. perhaps its slow down after the packets are received or something)
Also I just realised. Addons add more stuff and may very well push someone over the limit faster
On a 32 bit windows yes, but that's the problem of the end user if he has enough ram but not running the appropinate system.
On 64bit windows you have more virtual space available up to the known 4gigs.
Problem with NS2 is that keeps using more memory without freeing memory which is why it eventually crashes. That's not a 32 bit or 64 bit or whatever bit problem, it's an issue with the code. What's happening here is that memory does not get deallocated:
Think of it like this:
(X = used, [ ] = free, U unused)
Memory blocks -> crash, no more free memory, "lots of memory usage"
64 bit would resolve an issue if NS2 actually would need more then 4 GB of virtual memory. Thing is it doesn't necessarily need that much.
Now you're reffering to more or additional caching, in which's case 64 bit might make sense, so they can cache more. But you'd still have to do on-demand loading, unless you have shitloads of memory available, it's good to have on demand loading, or why would you need descent textures in memory if you haven't played descent for 10 rounds? Unless they'd change the requiremens of NS2 to 64 bit os, 16GB of ram and a gfx card with 3GB VRAM, a bold move to finally kill NS2 userbase (except the hardcore gamers that have high end systems already).
They still need to do on-demand loading, especially for lower end systems with less memory.
I wrote you a nice reply. It needs to be reviewed or something. :S
Originally we thought fragmentation was the issue since our code only allocates about 1 GB of memory, but after going through and eliminating places where memory fragmentation was potentially occurring, we found it didn't really make much difference. We've been building some tracking and visualization tools to help understand where the address space is going. We're also working on cutting down the amount of memory we use for things like animation and sound (which is about 75% of data). We're hoping to get a patch out very soon with our fixes because we know this is a very significant issue.
What you see in task manager only shows RAM usage. VRAM requires a 3rd party program to see its usage. VRAM adds to the 32bit address limit. Windows also adds 0.5 GB to the address limit.
Someone thought because Microsoft fixed the problem that caused all addresses on VRAM to be written to RAM as well. Now it only writes to the VRAM giving the impression that you can use more ram. It increased performance in some areas. In reality if you shift more load to the VRAM more RAM will become available, but it will not usable. It would have to increase the 4294967295 32bit memory address limit shared between the OS, VRAM, and RAM. Only 64 and special server CPUs with PAE enabled can achieve this
With 32 bit, If you want some M&M chocolate cake, you will have to give up some of your vanilla ice cream cake.
With 64 bit, you can have your cake, and eat it too.
EDIT:
Whoa a dev.
and more importantly, an answer to your question/statement/demands..
Thanks for the update, nice to know your actually getting somewhere with this, been feeling like you just gave up.