9582 Event ID Errors. Hey Exchange admins, does that ring a bell?! 9582 events have been a huge call generator for PSS. I know- my group troubleshoots them all the time with Exchange 2000 support cases, not to mention I see the topic come up a good bit on the public newsgroups I regularly check. Even though the 9582's are not something relatively new, I find there is still a huge misunderstanding around the topic. I want to use this log entry to talk a little about the infamous 9582 events that appear on many of the Exchange 2000 servers out there (note: 2003 has a lot of built-in self tuning that IÂ’ll talk about in one of my next weblogs). First of all, the 9582 events started showing up with Exchange 2000, but only the event IDs were added to the code in E2K SP1. Virtual Memory Fragmentation (IÂ’ll refer to it as "VMF") has been around long before Exchange.
Ok- With today’s servers easily containing more than 1GB RAM, let’s talk about the confusion around the /3GB switch on an Exchange 2000 Server… one point of confusion is taken right from a KB article, “Exchange 2000 requires /3GB switch with more than 1-GB of physical RAM”. The reason why we state this is because when the Information Store process starts up it goes through some calculations. Some of the variables within this calculation take into account how much physical memory, how many CPUs, etc., are installed on the system. For the purpose of my point, the important one here is obviously the amount of RAM. That is, “How much physical memory is in the box?” When the server has 1GB of RAM or more, the Exchange Store.exe assumes that this must be a pretty beefy box… "Because I (Store.exe) have a lot of physical memory, I want to take advantage of more virtual memory too."
This leads me to a quick review of Virtual Address (VA) space in the NT OS’s. Windows 2000/2003 implement a virtual memory system based on a flat, linear 32-bit address space; 32 bits of address space translates into 4 GB of virtual memory. On most systems, Windows 2000/2003 allocates half of this address space, the lower half, to individual processes for their unique private storage, and it uses the other half, the upper half of the address space, for its own protected operating system memory utilization. The mappings of the lower half change to reflect the virtual address space of the currently executing process. But the mapping of the upper half always consists of the operating system's virtual memory. This VA can be modified by using a few special switches in the Boot.ini. The /3GB “switch” can be placed in the Boot.ini file (Note: Windows 2000 Standard does NOT support the /3GB). This gives a 3-GB private address space to processes running specially marked executables with the “large address aware” flag set in the header of the executable, leaving 1 GB for the OS. This option allows applications, such as database servers (e.g., Exchange and SQL Server) to keep larger portions of the database in the process address space. Being able to keep more of the database in memory keeps some pressure off of the disk subsystem.
Anyways, the above is why the 9582Â’s are generated frequently on Exchange 2000 Servers running on Windows 2000 Standard Servers with >=1GB of RAM. Without the option to use the /3GB switch, the Store is restricted to just 2GB of VA. (Remember, the /3GB switch can not be used on Windows 2000 Standard edition, but ALL versions of Windows 2003 support it! Whoot!!) It's almost like a double-edged sword... Upon startup, the Store sees 1GB+ of RAM installed and it wants to take advantage of more virtual addresses by caching more of the database information, but with W2K Standard edition, we can't use the /3GB switch. Therefore, when the Store caches more data in the VA, that leaves less free VA to map in and out of during runtime. With less free VA, it takes less time for the VA to become fragmented. So you're options are somewhat limited. It's not an operating system limitation. There are a handful of tweaks, but the major one in this case is unavailable for a server running on Windows 2000 Standard. Unfortunately, there are a lot of customers running in that configuration. Keep in mind too that the 9582 warning is generated when the Store.exe determines the largest contiguious block of memory in the VA is less than 32MB. If the largest block is less than 16MB, the 9582 error is generated. When the error is generated, immediate action is required like restarting the Store service.
Allow me to review what VMF is. It has long been the focus of research by mathematicians and scientists. It occurs over time because of the varying size of memory allocations in conjunction with the varying lifetimes of those allocations. When I talk to customers about virtual memory fragmentation, I like to use the example of a hard drive in a PC. For example, I have a PC at home for which I just bought a 60-GB hard drive. I just installed it into my PC. The first thing I'm going to do, obviously, is load Windows XP on it. After that I'm going copy family photos, MP3s, and install other files and applications. All of those files, more than likely, are going to be different sizes. And many of those files I might keep forever and some of those files I might delete after X-amount of time. It's the same concept as VMF- that here are different-sized files that I'm keeping for different amounts of time. So what would happen after, say, six months or a year, when I run the Windows Disk Defragmenter utility? When I analyze my hard disk, it's going to be very colorful. It's going to have a lot of red and green, right? In other words, the hard disk is going to be fragmented. The same sort of thing happens in virtual memory as well. With your hard disk, you combat the problem by running a disk defrag tool. In memory, you combat the problem by restarting the processes/system to clear the VA and start over.
Since most 32-bit applications are built on the theory of utilizing different size memory chunks, fragmentation will continue to be a pain point. However, since Exchange relies a lot on memory, VMF seems to show up more often. Well, there are some steps that you can take to prolong the affects of VMF if your E2K box is generating 9582s. Going back to what I mentioned above, one thing to keep in mind is planning for it ahead of timeÂ… for instance, not setting up an Exchange 2000 server on Windows 2000 Standard with 1GB or more of RAM. With all versions of Windows 2003 supporting the /3GB, we shouldnÂ’t have that limitation too much more in future deployments with W2K3 and E2K3. But, for those still running Exchange 2000 on W2k Std w/1GB+, there are things that can be done, plenty actually. But, donÂ’t take this literally and go rip out a stick or two of RAM from your server! ThatÂ’s never a good solution. If you are getting the 9582 errors on your Exchange 2000 Server then check out *the* KB article on troubleshooting VMF, Q325044. This KB is setup as a guide. It has a handful of VMF-related tweaks that can be made on the server, plus other things like what to monitor, links to other KBs, etc.
The combination of Exchange 2003 and Windows 2003 have a lot to offer in terms of prolonging the negatives of VMF. So much so that I haven't heard of one case in PSS yet where an E2K3 Server was generating 9582's! I think this is a good pausing place for now...
