Windows Event and Logging Demystified: IT Admin Edition

Hello, my name is Lowell Vanderpool and this channel is dedicated to IT students, IT professionals and anyone who enjoys learning technical subjects. Now, I don't know about you, but I never had any formal training on the Windows Event Viewer, yet alone understanding the Windows Logging System or even understanding what Trace Logs were and Event Logs. Most of us in the late 90s just simply had to jump in and try to figure it out. That's not a good way of learning an operating system. But unfortunately, there was very little training at that point in time. So my goal is to introduce you to the Windows Tracing Architecture, which creates Trace Logs and Event Logs. Really understand what's going on with the Windows Logging System and how to use it more effectively, how to find better lookup websites for looking up events and sources to help you solve your problems faster. I'm going to demo a lot in this presentation, and I'm also going to show you some very effective third-party tools that are portable apps that really can help you with Event Logs. Now, the Windows Logging Architecture is built on top of what is known as ETW, Event Tracing for Windows. If you look at the picture on the screen, you see Event Providers. Those are typically modules of software, primarily a DLL in a process. One of its functions is to look at how code is executed and determine whether a success or failure happens or something that should be alerted. It then captures bits of information and then puts that into a binary file on your hard drive. That's a log file. Now, ETW, which is the foundation for all logging, produces two types of log files. One is Trace Logs, and the other is Event Logs. Trace Logs are used primarily for developers. So as an IT pro, I don't look at Trace Logs very much. But the other log is called the Event Logs, and those I do want to look at. Now, you may not be aware of it, but Windows logs a lot of stuff. And you have a ton of log files you probably didn't know you had. Now, one group of those log files end in the extension ETL, and those are trace files. You can look at those, but they're primarily for developers. And in many cases, this creation of log files, tracing, is not turned on until you put debugging on. Now, the other one is Event Logging, and those produce files that have an extension of .EVTX. So anytime you have a file with EVTX, those are Event Logs. Now, presently, I'm using my video editor, and I'm using a tool called Everything. It's a file search program, and it's 10 times better than Windows Search. But I'm going to use it to find ETL files. So I'm just simply going to say, look on my hard drive, star dot ETL. And you can see, just like instant, it has looked on my hard drive and has found every location there is an ETL file. That's a trace file. If you look down here at the bottom, it says I have over 1,411 ETL files. Guys, that's a lot of trace files you didn't know you had. Now I'm going to ask Everything to look for EVTX files, Event Log files. And again, you can see, just instant. If I did this with Windows Search, we'd be here a long time, and I'd have to do a lot of video magic. But you can see, just instantly, all those EVTX files. Notice something interesting. Most of them are in one directory. That was not true when we looked for trace logs. They were everywhere. But in Event Logs, they're primarily in one location, Windows System 32, WinEVT, and Logs. Now I'm at Nirsoft's website. And I'm going to go to the Pre-Release Tools, where he has a nice, portable app that allows us to look at providers in the Windows environment that generate trace files. So let's take a look. I'm going to go to his main page, come to Pre-Release Tools, and slide down to where I see Event Log Provider View. And you can download, extract it. And for me, I just simply put it in a network share or a flash drive, and you can run it anywhere on the network. So here I have Event Log Providers View running right now on my video server. And you can see under Provider Name, you can see the modules of software that are generating log files. And then over Message File 1 column, you can actually see the DLL that's responsible for generating those logs. Interestingly enough, the DLL is called iologmsg.dll. Go figure. Now, just because, as I scroll through here, you can see there's a lot of providers and a lot of potential log files that can be generated. Now, just because they're in place, they're in the registry, they're on your system, doesn't mean they're active and in use. I'm going to quickly wrap up our talk about trace logs because we really want to move to event logs. But there's two categories of trace logs, analytic, and those are typically used for performance evaluation. So if you want to look at what is impacting your boot up or your shutdown or a particular application's performance, you're going to use analytic trace logs. There's another category of trace logs called debug. Those are primarily for developers and the debug channel is disabled by default. So, Mr. Vanderpool, if I want to look at a trace file, how would I view that? Well, you can use Windows Performance Analyzer, which is a simple download and install. You can use the SVS trace view, the WPP software tracing, which is used for kernel mode drivers, event tracing for Windows ETW, Windows Event Viewer, which you use for event logs, you can actually open up a trace log with that tool. And, of course, PowerShell. I'm going to take Windows Event Viewer and open up a trace file. I'm looking at my everything search engine and I can see I've got a trace file called bootperfdiagnosticlogger.etl. And it's in the Windows WDI log files. It's pretty good size, so let's go take a look at that one. So, I'm going to go to Event Viewer and I'm going to right mouse click and open a saved file because it is a saved file. And I'm going to browse for it. Go to my C drive, Windows System 32, WDI, and open up log files. And there's the log file. You can see it's quite large. So, I'm going to go ahead and open it. And it's going to ask me to convert it. So, we're going to go ahead and convert that file. It's quite large. And here it is converting that trace file into an event log or something that Event Log Viewer can look at. And then it's going to want me to save it. And I'm going to save it under Event Viewer, Save Logs, which is right over here. You see a section in the navigation. Let's just save logs. And I'm just going to leave it with its original name. And there it is. I've opened up that trace file and I can actually open up. Not that that means a lot to me, but that's how you would open up a trace file with Event Viewer. Now, back to event logs. Event logs are binary files. In Linux and Unix, they're typically text files. But in Windows, they are binary files. They're primarily located in the Windows System 32 Win EVT backslash logs directory. And event logs are organized into channels. So we're going to turn again to Nerfsoft. He's got a portable app that allows us to look at event log channels. Now, back to Nerfsoft's website. Slide down until you get to System Tools in the menu on the left-hand side. Once you get to that, you'll just slide the page down to where you get Event Log Channel View, which is a simple portable app utility. And above it is another event log view. It's called Full Event Log View. Take this one, Full Event Log View, extract it, and put it in the same folder as the one below. And when you do, you get to use two tools together. So it's a very handy way of having both of these tools, put them in the same directory, and they'll launch and work together. Now, here's what I've done. I've got a network share, and I've put the Event Log Channel View utility in the same directory that I put the Full Event Log View utility. Put them in the same directory, and they'll work together. Now, I'm on a domain controller, and I'm looking at my event log channels. And if you scroll down, you can see there's a lot of channels. And if you're kind of puzzled as to where these channels are in relationship to the Windows Event Viewer, let's take a look. I've launched my Event Viewer on my domain controller. But as you start opening up, you'll see Application Security Setup. All of these are what we call general logs. And if you open up the Applications and Services log, you begin to see a lot of additional logs. These are channels. If I open up Microsoft and I just come down to Windows, it just goes on and on and on. So you have many event log channels. Now, Nerfsoft's Event Channel Log Viewer allows you to see all the active, and you can see it by the green dot beside the channel. That means it's active. You can actually disable it. Come up here, hit the red button, and you actually now have disabled that particular channel from logging. So here you can see it's no longer green. It's now red. That's amazing. You can now actually turn on or turn off any type of logging channel that you so desire. That's not necessarily helpful in every situation, but it's pretty cool. I can also come back. I'll go ahead and turn that back on. If I go up to Options, I have an option of Hide Channels with Zero Events. So I'm going to uncheck that, and you'll see we have a lot of inactive event log channels. It's refreshing. And now you can see all the channels that are on this domain controller. The ones that have the red dot are off. They're disabled. And if you look at them, it makes perfect sense. Some of these have to do with media foundation device proxy. That may have to do with a role or a service that I could add to this server. It's not on, so those event log channels are disabled. Now, each of these channels have their own event log. If you go and look at File Name Column, you can see the name of each of these event logs. Now, I'm going to scroll over, and we'll take a look at what additional information we can see. First of all, we can see the maximum size of that log. Has the file reached its maximum size? No or yes. We can see is it a debug, operational, admin? Is it analytic? Now, under the column Channel Isolation, it lists application and system and custom. Those have to do with the security settings. So, why they call it Channel Isolation, I have no idea. That is all about security. Now, this is a good time to pop back in and talk briefly about properties of logs. There are many ways that we can save log files. We can do what's known as circular. In other words, once a log file fills up, we can begin overriding the oldest log entries. That's normally what we call overwrite. Then we can do archive, where the log file fills up, reaches its maximum size, and then it's saved, and then we create a new one, and we continue on capturing logs. Then we also have do not overwrite. And that allows us to, once the log file fills to its fullest size, it then stops. It no longer captures events. Now, in the event log channel view, you can see it talks about, has a column called retention mode. If I right-mouse click on any channel, I can see that I can look at that retention mode. Is it set to overwrite? Is it set to archive? Is it set do not overwrite? And I actually can change it. In the right-click menu options, I can choose a new channel, and I can also set the maximum file size. So here I can set how large I want the file that's capturing events to be. In this column, I can see what channels actually have events captured, which channels have nothing. Another nice feature is if I want to capture all the events in a channel, I can also save them. Right-mouse click, save selected items. And then I can save them as a text file. I can save them as a common delimited, an HTML, or even a JSON file. This is very, very nice, especially if you need to generate a report or include it in a compliance report. You can just easily export this data out and attach it to a report that's going to somebody. Now, I also have the ability to go to any channel. I'm going to go to the system channel. Right-mouse click, and notice I can launch the full event log view utility, the other tool that we put in that same directory. So I can select the channel and say, now launch this second utility. So we'll go ahead and do that. Now I'm looking at strictly the system channel, and I'm only looking at seven days. So by default, the full event log view pulls up seven days' worth of events. If I want to change that, I simply go to options and hit F9 or click the advanced options, and here's the beauty of this tool. I can choose the type of events I want to look at, or seven days, or in this case, two days, or two seconds, two minutes, two hours, or from this date and this time, from this date to this time. Or I can say, show me all events, or I've got a common delimited text file that says only capture these events or these event IDs. Very powerful. Show me all providers or show me only the providers that I'm going to show you in this common delimited text file that I created. So it's extremely powerful search tool that allows you to go through a lot of logs and just filter out what you're looking for. It has a lower pane down here, so if I click on this error, you can see information shows up in the lower pane. This is really easy because it's text, so I can take it, copy and paste it, put it into Google, and maybe research that event. Now under the full event log view, under options, under the lower pane, if I want to look at the event itself, I can display that event in XML, or as you see down here, I've got it displayed in XML, or if I want that error event displayed in data and description, so I'll get both data and description. So a lot of flexibility in how you can use this tool. Again, if I'm trying to save this particular event, I can save the selected item and I can save it in many formats. Text file, comma delimited text file, HTML, JSON, and even raw XML. So very flexible if you want to save this in another data format. Now I will spend some time looking at Event Viewer and what you can do with that, and I will definitely show you some of the limitations and some of the real pluses by using Event Viewer rather than these third-party tools. Again, a nice diagram showing us the big picture. Application sends its event. The event logging service plays a role in writing that event to a file on the hard drive. If you'll notice the services, we have the Windows event log. That's the service that's part of this process. If you'll notice, you have another event log service called Windows Event Collector. That one is right now set to manual, but if you decide to make this workstation a centralized collector of all events from all your critical servers, you can put those events to one server or one workstation. When that happens, that workstation becomes a collector, and then that event collector service will be on. So who generates events? Applications, Windows operating system, services, Windows local security authority, that's who does the security events, and also key management service. There are channels that are installed by Windows when it installs. One is the application security setup system and forward events. These cannot be deleted, and they're independent of any existing publisher. When we look at the general logs under Windows log under event viewer, we'll see the application log. This is for all user-level applications. This channel is not secure, and it's open to any application events that need to go into this log. This is very different from the next one, which is the system. This channel is used by applications running under the system service accounts, installed system services, drivers, or a component or application that has events related to the health of the computer system. Now, the security log is controlled entirely by the Windows local security authority. User events may appear as audits if supported by the underlying application. Now, this is one aspect of event viewer that I'm not going to get into because this part of auditing and security is a whole other topic. I'll get into that later. Now, the setup log contains messages generated by installing or upgrading the Windows operating system. Now, if you have a member server and you promote it to, say, a DC, a lot of those events go into that setup. Now, forwarding events are when your workstation or your server becomes a collector and you're pushing all the events from around your network to one device. They will show up under forwarded events. Now, events fall into different types. We have critical. They're typically things that you need immediate attention by the administrator. There's also event types that are error. They indicate a problem, but the category does not require immediate attention. There's also events that are warnings, events that provide forewarning of potential problems. And then information, events describe successful operation of an application. And I'll show you the value of an information event as we continue. Then we have success audit, failure audit. These are specific to auditing events that happen in the security log. If you look at my event viewer for my video editor, you can look down here and see many channels that are involved on my video editor. It really depends on what you have installed. Let's take a look at my domain controller. You can see that some of my channels look very different on my domain controller than it does on my video editor, and that makes perfect sense. Now, when it comes to event log viewers, monitoring tools, and analyzers, there is just this huge available suite of tools out there from third-party vendors. Obviously, you can use event viewer, you can use PowerShell. Microsoft Azure has some of this monitoring, analyzing capability. There are a lot of third-party tools that are simply portable, like I've showed you with Nerfsoft. There's also analyzers, monitors, and viewers of event logs that have an agent-based, and then they push everything into a cloud dashboard. And then there's, of course, fully installable viewers, monitors, and analyzers of event logs. Now, let's turn our attention back to event viewer, the utility itself. It has many drawbacks, very frustrating drawbacks, and yet it has some really handy features. Now, obviously, if you've got a large server room or a data center, you're not going to use this tool. You're going to be using one of those tools we just talked about. But for smaller environments, and especially workstation troubleshooting, this is really handy. Now, at my workstation, I can come right up to event viewer. If I don't want to troubleshoot my local workstation, I can right-mouse-click and connect to another computer. So I'm going to pull that over and go in. And I'm going to go into my domain controller. And in just seconds, I've went right into my domain controller, and I'm using my event viewer to actually analyze its events. Now, you do have to set up a group policy, and you do have to bring down some elements of your firewall. That is in the notes, so just download the video notes, and you can see exactly what you have to do to get that done. Now, I can just go from any server in my environment or any workstation in my environment. Once I have done that. So as we look at the navigation of this utility, we can see the very top, the summary of administrative events. And it gives us up to seven days' worth of events, which is really handy. It's a quick little dashboard that gives you a quick view of, say, critical events in the last seven days. So I have had a critical event in the last 24 hours. And so I can dig in and take a look at that. Down at the very bottom in the center section of event viewer is log summary. And we can look at each of the logs that represent the channels. And we can see that the retention policy, and we can look at their size. So we can take a quick look at our log files if we want to look at them here. So here's an aggravation of event viewer. It calls the center section called recently viewed nodes. Why are we using the term nodes? These are channels. So if I come and I open up key management service and I go back, there's nothing there. It will show that I went into key management service as my recently viewed. Why not call it channels? Why call it nodes? You know, don't you love Microsoft? So again, I can go to the navigation pane. I can click on system and there's all my events. I can double click on any one of these and it pops information about it. Now, this is one of the aggravations. One, you can't copy and paste out of this into Google. You can't copy and paste it out of anything to anything. So this is an aggravation. You have to either sit down and retype it into Google exactly if you're trying to search for that type of general information about that event. Very poor. Why we can't copy and paste. The other ridiculous issue is the event log online help. That sounds exciting, which is not help at all. So having a very poor interface here and absolutely useless online help is some of the real drawbacks to event viewer. Now, if you choose an event, let's say I wanted to choose and save this event. What you do have over here on the right hand side, I can save selected events. And again, I can give it a name and I can save it as an XML file, a text tab eliminated or a comma separated file. Remember, I told you I would show you something very interesting that you could do with something like an information event. Let's say you have an information event that when this happens, you want to run a script. Well, you can click on that information or any event. You can do this, but just to show you that information events can be very helpful. So on this one, it tells me that something happened. And when this happens, I want to run a script so I can come over here and say attach a task to this event. So any time this event happens, I want to go to task scheduler and I want you to automatically run a PowerShell script, a batch file, whatever. That is a powerful feature. Also, on the left hand side, you can clear a log so you can come up and clear the log. You can also create custom views. So let me create a custom view. In the last 24 hours, I want to create a custom view of any critical error having to do with my backup agent. So I'm going to open this up and I run Veeam's backup agent. So anything that is critical from the Veeam backup agent, I want you to show me. And so I'm going to say OK. And I'm going to call this mine. It's going to say under custom views. And here you can see now I have my Veeam backup events and it will indicate any time I have a critical event with Veeam. And I can look at that and say the system has rebooted without cleanly shutting down first. So those are things I want to know about my system. And I can do that very easily and create a custom view in Event Viewer. If I'm in a system event and I want to find an event ID, I can just quickly type in that and it quickly searches my Event Viewer. Now, if you remember in the beginning of this video, we talked about traces where we have analytic and debug traces on our system. They're not normally shown in Event Viewer. But you can come over to View on the left-hand side and say I do want to see those analytic and debug logs. And you click that and more things show up over here that you can view. And here I've opened up some of these Microsoft NS drivers and you can see I have some additional logging. This is really tracing that I can now view in Event Viewer. Now, there is one custom view that is always created under the custom view. You will always find the administrative events. And that primarily is for a quick way of seeing critical warning and error messages quickly populate for a quick look at that workstation. So that is a custom view that's almost always on each event viewer pre-set up. Now, let's talk about troubleshooting. The whole purpose of Event Viewer is to help you with troubleshooting. So let's look at some practical elements of troubleshooting. It was actually this problem. I was working on a DNS server service event error. My gut told me that it was probably a misconfiguration. Having just built this home domain, I probably needed to do some tune-ups and clean-ups and configurations that I had not done. And I saw this and I began to dive in to troubleshoot this problem. One of the things you want to do when you're dealing with Event Viewer and troubleshooting is make sure when you're doing your Google searches that you include the event ID number and the source name. In this case, it would be event ID 4015 and source DNS-server-service. Make sure you put all of that in there in your Google search. It's going to give you a lot more effective results. When I'm dealing with an event that is related to a service, one of the first things that I want to do is go look at that service and look at its dependencies. What other additional services are critical to the one that I'm troubleshooting? In this case, DNS-server. I want to make sure that I look up event IDs related to Active Directory Domain Services, Auxiliary Function Driver for WinSOC, RPC, TCPIP, Protocol Driver. I want to look up some of those so that if those show up in my hunting down this problem, it could be these dependent services that are possibly causing the problem in the service that I'm working on. I'm also going to go see if I can find as many related event IDs that are part of DNS-server-service. I'm going to try to find those and create maybe a text file that I can use to pop in as I'm looking for related event IDs specific for DNS-server services and see if any of those pop up in my search. Maybe three weeks ago, two days ago, 24 hours ago, in addition to the one that I'm looking at. Here I put in my text file various event IDs separated by commas, and I can throw that into NerveSoft's tool and say, while you're looking for the event ID that I'm struggling with, check these also, and I can expand my research from 24 hours to maybe two days to three days to see if any of these pop up related to my DNS problem. A few days before, a few days after, I am attempting to filter out all the unnecessary events and try to focus on events that are related to my specific problem. There's over 24,000 events in my system event log. I could care less about them. I'm looking for any related appropriate event for my DNS-server-service. So here's an example of Google search. I put my event ID, then I put my source name, DNS-server-service. When you're doing your searches, be accurate. Put in the proper name. If it's got dashes, put in dashes. Be deliberate in putting proper information in your Google search. You're more likely to get accurate information in your results. As a matter of fact, this result solved my problem. Now, the best website I found for searching events, if you just want to go to that site and do your own research, this is a site you want to go to. And they include events for Windows, Cisco, Antivirus, Veritas, OpenManage, VMware, and others. So it's not just Windows. When you're troubleshooting event logs, analyze things that happen before and after to determine is there any relationship. Look for that chain of events that are related and showing some kind of defined sequence. Learn to ignore non-related events. Filter out the noise is a big step in finding the right events to analyze. And thoughtfully implement corrections. When you have solved the problem, those events should go away. Avoid the problem of solving one thing and breaking another. How many have done that one? And if you find yourself frustrated and feeling stupid sorting through 24,000 events and not seeing the problem, relax. It's just part of the job. There's not a tech that hasn't been there. So it's experience. It is hard work. It is getting in there and working to try to solve the problem. But don't beat yourself up about it. Trying to find problems with events can be a real pain. When things are not broken, one of the most important reasons to get into your event log is to find misconfigurations. If you're listening to security blogs or security podcasts, you're realizing that one of the number one sources of security breaches is misconfigurations. An event viewer is very good about helping you find misconfigurations. If you're watching this at this point in the video, you are a hardcore technology person. 90% of the people who are on YouTube who watch a video that I create are gone in three minutes. So the fact that you're watching me right now tells me you're pretty hardcore. And you're the very reason we do all the work, all the video editing, all the preparation is because of you. You're the person we're after. You want to learn. You want to understand. And you're willing to watch 25 minutes, 30 minutes of just geek stuff. And we really, really appreciate you. One way that you can help us tremendously is support us by liking a video and subscribing. It's simple. Two clicks and it doesn't cost you anything. And it really, really helps us. If you can join, that's great. It really does help us. It's $2 and something and a month. It's a cup of coffee a month. We really, really appreciate it. But it's more important if you can like and subscribe. And it's the best way of supporting this channel.