Windows Application Troubleshooting 101: Process States Deconstructed Day 3

Let's begin by looking at three pseudo-processes shown to us by Process Explorer. Next, we're going to look at brand new processes brought to us with Windows 10 and Server 16. Then we're going to identify critical processes that are in Windows 10, Server 2016 and above that are in every edition of Windows. In a preemptive multitasking operating system like Windows, it's the hardware that's the major driver for preempting the scheduler. On the south bridge of most motherboards and servers contains the IOPIC or the Advanced Programmable Interrupt Controller. All hardware has to go through the APIC to prioritize interrupting the CPU. I'm going to come down to my search bar and type in system information, and I'm going ahead and launch system information so we can take a look. And we're going to go to hardware resources, and we're going to go to IRQs. This is the interrupts that are prioritized by number from zero to whatever. These are all hardware devices on your motherboard that have the right to interrupt your CPU. The lower the number IRQ, the higher priority it gets. So this brings us to our first pseudo-process in Process Explorer. We're going to go up under system. Below that is a child process called interrupts. Remember, interrupts allows us to see this action going on, interrupting the CPU. So when you see CPU utilization going really high on the interrupt process, you know that it's either hardware failure or a software driver, one of those two items. Our second pseudo-process is called the system idle process. When there is no process or a process that needs threads executed by the scheduler, the system idle process put in. Remember, process is the technical definition of a process is that it runs in user mode. This is a kernel mode only. If I go to system idle process, go right mouse click, go to properties, go to threads, you can see it's strictly NTOS kernel, which is strictly kernel mode. Our third pseudo-process is the system process. This is our one view really into the kernel. You can see I've added the threads column and the handles column, and you can see what a complex process this is. This also includes a driver. If I go up to the show lower pane, you'll see that when I select the system process, you can see all the drivers and their description. So this is a real handy way to take a look at the drivers loaded on your kernel. When you have high CPU utilization on your system, you probably have a driver issue. So you might want to look at the threads. You can right mouse click, go into your properties, go into the threads tab, and take a look at any of the drivers, SOS files. Look at CPU utilization and track that down that way. Another interesting note, under properties, if you click the performance tab, you'll see that all of the CPU time is in the kernel mode. So there's no user mode time used by the CPU. It's strictly a kernel mode process. Windows 10 and Server 2016 and above introduce some new processes that we want to take a look at. Virtual secure mode is a new approach to security for credentials and code integrity. In the past, malicious code was finding its way into the kernel and going after what was a process known as LSA or the local security authority. This is where pass the hash stealing of credentials was coming. So Microsoft has come up with a virtual secure mode, which puts a hypervisor on the platform, which pulls the OS up and puts a hypervisor underneath, builds a child partition, and creates what's known as an isolated LSA. So now it takes the credentials and puts them into this child partition, making them more difficult for malicious code to access. Virtual secure mode introduces two new processes. One is called secure system, and it usually is at the top of process explorer. You'll notice that it's always suspended. The second one is, is this one here, the isolated LSA process. And it's this one right here. This one is now in a secure child partition sitting on top of the hypervisor. If we right mouse click, we go to properties. When we open up the image, you'll notice you can't access anything because it's hid away by the hypervisor in that child partition. So when you see those two new processes in your process explorer, you know you have VSM enabled. VSM is this underlying technology that allows device guard and credential guard to function. In order to use VSM, remember you have to have UEFI running in native mode, Windows x64, you have to have the newest processors that support SLAT, second layer address translation, and they recommend TPM. Our next new process with Windows 10 is memory compression. The memory compression process is a child of system. When the memory manager feels memory pressure, it will compress unused pages in RAM instead of writing them to the disk or the page file. This used to be in the system process, and it had such a huge working set. In other words, the amount of memory in RAM was so large, people were freaking out. So they took it out of system, created a child process, and labeled it memory compression. When we go to process explorer, we can see memory compression. And the one thing that catches our eye is the column under working set. That means how much RAM that process is taking. And it's gigantic. So yes, and it should because it's in the memory compressing unused pages so we don't have to write them to disk. So it's working set is high. That's normal. Don't panic. That's the way it's supposed to be. When I pull up the properties for a memory compression process, you can see it's nothing but kernel mode. If I go to performance, you can look at the time, CPU time, that's in kernel mode versus CPU time in user mode. So this is a kernel mode process. When Microsoft launched the compression process in memory, they had a little bit of problem with registry. And so they ended up creating a new process called the registry process where they copy large portions of critical portions of the registry into that process. And this helped improve the speed for accessing the registry. This was kind of in compensation of the memory compression technology. So in process explorer, here is my registry process. You'll notice it's right under secure system. If you have VSM enabled, right mouse click, go to properties. You can see it's all kernel. If I go to performance, all the CPU time is in kernel mode. Again, this is another curve. Now, notice I'm seeing their processes. And I've already clearly stated that anything that is running strictly in kernel mode is not really a process, but they are exposed in the process explorer and in task manager so that you can see them. Let's look and identify critical processes that are in Windows 10, Server 2016, and above. These are fundamental to the Windows operating system. The first one we begin with is the Windows Session Manager, smss.exe. It is a child process of system. It is the first user mode process created in the system. It starts the kernel and user mode of the Win32 subsystem. And it begins by creating virtual memory paging files. So our Windows Session Manager, it's a child process under system. Also, if you go to options and you go to your colors and we check the protected processes, we'll see that it is also one of the protected processes. We literally could spend an hour on this one process and all that it does in preparing the system. The next is our Windows subsystem. It is also called Client Server Runtime. The executable is csrss.exe. The Client Server Runtime process is responsible for creating and deleting processes and threads. It's also foundational for setting up the GDI or the graphic device interface. It also controls Windows displays and much, much more. All of these processes that we're talking about are part of what a category of user mode processes called system processes. And you can see them. We've already walked through some of these. These are critical for setting up services, applications, and everything else that the user needs to run user mode. Notice that all of these user processes, user mode processes, are needing the ntdll.dll to go into kernel to do their kernel work for them. Here in Process Explorer, we can see that our Client Server Runtime process is a protected process. Next is the Windows Initialization process. And it's the wininit.exe. This initiates user mode scheduling. It creates the Windows Station winstat0 and two desktops. It starts the service.exe, which is going to be very important. It also starts the local security authority, the lsass.exe. Remember, we just talked about that in VSM, the virtual secure mode. It also is responsible for shutdown. So it's the one that sits and waits forever for your system to shut down. The Windows Initialization process is also a protected process. The wininit process, as you can see, is the parent process for services.exe. And then everything below it are all your services that run below it in getting all your services up and running. Next is our Service Control Manager, services.exe. It is responsible for starting, stopping, and interacting with our service processes. It uses a generic executable called svchost.exe. And by using switches and arguments, it can launch all the services on your operating system. So the default color for services is pink. And you can see under services.exe, you can see all the svchost.exe. It seems like how can one image launch all these different services? The trick is arguments and switches. I'm going to come up to my columns and I'm going to select columns. And I'm going to come over to image. And I am going to choose the option of command line. And I'm going to add this to Process Explorer. And I'm going to just slide this over so you can see it. And then I'm going to stretch it out so you can see all the. And you can see that svchost here is launched by many arguments and switches. In this case, you can see it's launching a DCOM and plug and play. And you can look at each of these svchosts and they use a variety of arguments and switches so they can launch different services even though they're using the same exe. Mark has also made a way that you can take your mouse and do a mouseover over any svchost.exe. And you can see the different services that it launches. When you are looking at Windows Server and comparing this to Windows 10, this is where they really differ is services. The kernel of Windows 10 and Windows Server 2016 and above are very similar. But when you get into services, this is where it differs. Because the client, Windows 10, has a ton of services to support. Bluetooth, wireless, printing, all kinds of additional functionality that you need in the Windows 10 box you don't need in your server. So it's services that really differentiates Windows 10 and Server 2016. So I have my services app and you can just see this incredible list of software that is running or installed on my Windows operating system. The difference between services and applications is that services are, in a sense, applications that are controlled by the operating system. Whereas when you log on and you launch Word or PowerPoint, these are applications under control of the user. It's also important to remember that about 90% of services are written by Microsoft. There are third-party services. As you add applications, they can add their own server. But in most cases, I would say 90% is Windows. The Local Security Authority is run by the lsass.exe that launches the process and it's responsible for the local system security policy, who's allowed to log on, password policies, privileges granted to users and groups, system security auditing sessions, and user authentication. This is the one that's often being attacked by malicious code. If you have VSM, virtual secure mode, turned on, then you'll get the isolated process as well as this one. Looking at the properties of the Local Security Authority, we can come up here to Services. You can see your EFS is running under that, the EFS service. A number of critical security services are running under the Local Security Authority. Here in this diagram, you can see the Local Security Authority process is really in between Active Directory, our local SAM registry key, and the WinLogon. So you can see a lot of important things. Kerberos DLL is there. It is responsible for determining which database, either the registry key, the local SAM accounts, if you're logging on locally, or Active Directory, if you're logging on to the domain. Our Windows Logon application is responsible. WinLogon.exe is responsible for that interactive user logon and logoff. Back to our diagram, you can see the WinLogon communicates to the Local Security Authority. Under WinLogon, you can see we have two child processes. One is for fonts, and the other one is called the Desktop Windows Manager. Once successfully logged on, these are responsible for launching your shell, which is explorer.exe, which is your GUI desktop. Let's clarify some concepts concerning processes and threads. Remember, processes are containers that hold the resources for threads to store and manage all the necessary data and executables to function. This diagram kind of gives you a picture or a block diagram of a process. We see our process. Notice it has an access token. This would be the security access. Then we have virtual address descriptors, which we're going to get into memory in just a minute. We have handles, graphical objects, and then we have threads that it executes. This is kind of a big picture of processes. A thread is the software code within a process that is scheduled for execution in the CPU. Here is chrome.exe. This is a process. If I right mouse click and go to Properties, and I go to the Threads tab, you can see a list of threads. These are typically DLLs. They don't have to be, but they're typically DLLs. We can see that it is using NTDLL.DLL to go into the kernel. We also see that it does have a thread called chrome.exe. In most cases, they're DLLs, dynamic link libraries. When Chrome is brought to the CPU for execution, it's actually the threads that are executed. If I have a dual-core CPU, it is possible that two threads of Chrome can be executed simultaneously. If I have a quad-core CPU, I could potentially execute four threads simultaneously. When multi-core, dual-core, quad-core CPUs came on the scene, it really pushed developers to develop their processes with multi-threads so they could take advantage of multi-cores. In our WANinit.exe process, you can see I've only got one thread in that process. When I take that process to a four-core CPU, it doesn't improve that process's performance at all because it only has one thread to execute. The more threads and the more CPUs you have, the greater chance that you're going to see a performance boost in that application and operating system. Inter-process communication or IPC. Processes need to communicate with each other. They need to share data. They need to message each other. And this is all based on strict rules to allow them to do this. This is called IPC or inter-process communication. Threads also need to be able to communicate and share data between threads when they're in a process. They are also given a thread ID or a TID. Often when you're dealing with processes that are causing you problems, it's really not the process. It's the threads inside. Very often, threads misbehave, don't communicate properly or share data correctly and they crash and burn. Often the problems that we face with processes are not processes at all. They're threads. Let's now move into the concept of memory. When we begin the concepts of memory, it can be quickly confusing, even overwhelming, as Microsoft applies so many different terms to memory. We're going to look at virtual memory. We're going to look at current commit. We're going to look at commit limit. We're going to look at physical memory. We're going to look at page pool. We're going to look at non-page pool. We're going to look at page faults. All of these terms relate to memory. If we're not careful, we can get quickly lost in the subject of memory. The operating system and all user mode processes use two types of PC memory. Volatile RAM, which is your RAM chips. Non-volatile RAM, which is a file on your hard drive known as a page file. The operating system can read data from a typical RAM DIMM reaching as high as 10,400 megabits per second. On the other hand, when an operating system has to use a page file on a hard drive, even when we're talking about a premium hard drive, an NVMe SSD hard drive, that can reach 3,000 megabits per second, there's still a four-fold loss of performance on the operating system. The Windows Kernel Memory Manager is tasked with the responsibility for efficiently managing RAM so as much of the operating system and the processes are maintained in RAM and not having to be paged out to the page file on the hard drive. New with Windows 10 is our memory compression process. It resides in RAM and compresses any unused pages in our RAM so that they don't have to be paged out to the hard drive. In a Windows 32-bit operating system, Windows presents to every process in user mode a two-gigabyte virtual memory space to run in. It doesn't mean that the process uses two gigs, but it gives it a two-gig virtual memory space. In a 64-bit operating system, the operating system gives every process in user mode a eight-terabyte virtual memory space. I know you're freaking out right now. You said, I don't have but 16 gigs of memory. How can I just hang on? Stay with me. But keep in mind, there are SQL databases that could easily use eight terabytes of virtual memory space. This is very important. Operating systems present processes in user mode only virtual memory space. Processes do not know anything about physical memory. So they're happy the operating system gives them virtual memory space and they're happy as a lark. They do not know anything about physical memory. So what's the deal of virtual memory? Virtual memory cannot actually store data or code, but processes can create large blocks of virtual memory and then, this is the key, commit it as needed to ensure that the committed memory is contiguous in address space. Remember, from a programmer's point of view and a processor's point of view, they like to have memory that's contiguous, not broken up and fragmented. By having virtual memory, they have that kind of space. Both developers and processes need simplicity and virtual memory gives them a large contiguous range of virtual addresses. They will allow the memory manager to figure out how to map what they need to physical memory. They can simply use virtual addresses. It's simple and straightforward. Second reason why we use virtual memory is a process can use a range of virtual addresses to access a memory buffer that is larger than the available physical memory. The major advantage of virtual memory is that it allows more processes to execute concurrently than might otherwise fit physical memory. The third reason we use virtual memory is the virtual addresses used by different processes are isolated from each other. This isolation brings stability to Windows. Processes know nothing about physical memory. An operating system decides when and if to store the program's code and data in physical memory and when to store it in the page file on the disk. Now let's look at one of the many terms that Microsoft uses for memory. We can see that Chrome is a process here and we know that if it launched in a 64-bit operating system it now has an 8 terabyte virtual memory space. But that is not important. Once that process launches, it must communicate to the operating system a committed memory value. In this case, if you look, its committed memory value is 797 megabytes. That way the operating system will guarantee that it can maintain all the data, image, whatever that process needs, either in physical memory or on the page file on the hard drive. In your system internal suite of tools, as you go down to the Vs, you'll find a virtual memory map64.exe. You can double-click it. This application will pop and give you a list of processes. If you want to see them all, you click show all processes. But I'm going to go ahead and look at Camtasia Studio, which is the video editor I'm using right now. We've pulled up Camtasia Studio.exe. You can see its process ID. And at the top we can see the commit memory value. That's almost 8 gigs. That is what the operating system has to guarantee either in RAM or in the page file. Now private bytes is what technically is in the page file. It is also important to note that private bytes is not shareable by any other process. It is strictly restricted to that process. We can see that I've got 8 gigs in private bytes. Working set is my RAM. I've got almost 3 gigs in RAM. Mr. Vanderpool, can processes change that commit value? Yes, they can. The committed memory value can change. Remember, processes are on demand when requiring memory. But they have to keep the operating system informed. They can also give up memory so they can demand more and they can relinquish it. Let's keep going. Now I've got the entire, all the services, all the user apps, the kernel, everything involved. Every one of these objects have to give the memory manager a commit memory value so that it can look at RAM, it can decide how big the page file needs to be so that it can guarantee the commit memory space that everybody requires. So here's my process explorer's system information. Focus in on the commit charge section. You see my current commit. That's the sum of everybody's committed memory value. So right now that's at 16 gigs. Notice below it is limit, current commit limit. That means that the page file has been created and if we ask the memory manager more than 21 gigabytes, it's going to have to make the page file larger. You can see the peak right below it. That's almost 24 gigs. So sometime during the power on cycle, I've actually demanded up to 24 gigs of commit memory space. So it had to make the page file larger and then after a while you can see it's went back down to 21 gigs. So it can shrink and grow the page file as need be. The purpose of that memory manager is to use RAM as efficiently as possible and only where possible. Move things to the page file. Remember, there's a fourfold performance hit on the operating system whenever we have to page memory out or page memory back from the hard drive. So here I've got Foxit Reader and you can see committed is the process's committed use of virtual memory. Private bytes is a portion of the process's committed virtual memory backed by the page file on the hard drive. And finally, our working set, a portion of the process's committed virtual memory that's in physical memory. The operating system maps the processes from virtual memory to physical memory. This is controlled by the memory manager. This is an example of virtual pages mapped to pages in physical memory. Kernel mode memory that is allocated in the page section can be moved out to a page file as needed. The kernel mode memory that is allocated in the non-paged section can never be moved to a page file. It has to stay in memory. Remember, adding RAM keeps your page file use low and improves the performance of the operating system. When you use your page file, you're going to decrease the performance of your operating system. The virtual memory map is a very cool tool from System Eternals and one of it, one of the nice features, is a tool called Empty Working Set and basically allows you to take any process and push out of RAM as much of that process as you can. So if you'll see right now, I've got three gigs in RAM and if I hit the Empty Working Set, it will force as much as it possibly can out of RAM. Boom, just like that. Another tool that Mark Russinovich has created is called RAMmap.exe. You can launch it and it has a nice tool in that it doesn't just get rid of, it doesn't flush RAM with just one process. It will flush RAM of all processes. So let's take a look. So I launched my System Information and Process Explorer and you can see I'm using, right now I've got about nine gigs available of RAM. If I come over to RAMmap and I come over here and I say Empty All My Processes Working Sets, that means clear out RAM, and I hit that, you can watch. You can see my available RAM goes up. Just cleaning those guys out of RAM. Now they have to go on the page file if it's data that's critical to the process. So my page file is going to grow, but my RAM gets emptied. Let's go back and look at this term that can be very confusing, virtual memory. A lot of you on the tech support side say, wait a minute, I've always been told that virtual memory is the page file. Hence the confusion. If we go on the MSDN side of Microsoft and we look at the developer's material, every time they use the term virtual memory, they're talking about a virtual space per process that is allocated to every process in user mode. This has nothing to do with the page file. But if you go to the technical support pages, often they refer to the page file as virtual memory. Very confusing. In this lecture series, I'm going to try to stay very careful, keep those two terms apart. Whenever I say virtual memory, I'm talking about the virtual address space given to each process. When I talk about the page file, I am not going to use the word virtual memory.