Table of Contents

Lab 08 - I/O Monitoring (Windows)

Objectives

You can download the Windows 10 VM via OneDrive.

If you need to use VirtualBox, you can use this .ovf version to import the VM (just on OneDrive) OneDrive.

There is also the option to download as a torrent ep_win10_vm.7z.torrent.txt. DokuWiki is not configured to accept .torrent files so remove the .txt extension. After that, you know what to do…

Alternatively, you can install the following on your own Windows machine:

If Visual Studio prompts you with an “Expired” message, you will have to log in with your (university) account.

Contents

Tasks

Introduction

As you remember from the Linux Monitoring labs there is an endless list of tools for system analysis.

This is unfortunately not true for Windows. The system is closed-source and the development of tools is much slower.

The first (and probably most popular), set of tools for system analysis is Sysinternals. This was later aquired by Microsoft and it is now their recommended tool for analysis. The suite contains a wide variety of tools, but we will only concentrate on the widely used ones.

01. Task Manager

Shows real time information about processes and the system. To start Task Manager you can use any of the following methods:

Tabs description:

Conclusions:

To overcome Task Manager’s limitations, and to perform a thorough analysis, use the Resource Monitor (Resmon) utility, which is built into Windows.

02. Windows Performance Recorder & Analyzer

Windows Performance Recorder (WPR) is used to record the whole activity of the system in a time frame. Compared to Task Manager, this tool only captures information, without displaying it.

To inspect the captured data you will need to use another tool, Windows Performance Analyzer (WPA). This combination of tools is most useful when running tests that take hours and constantly watching Task Manager would be impossible.

03. Process Monitor

Process Monitor is another troubleshooting tool from Windows Sysinternals that displays the files and registry keys that applications access in real-time. The results can be saved to a log file, which you can send to an expert for analyzing a problem and troubleshooting it.

How to Use Process Monitor to Track Registry and File System Changes?

We want to write to the HOSTS file (C:\Windows\System32\drivers\etc\hosts) in order to add new rules. When we try to do this we encounter an error when saving the file.

Following the steps below (or the video) we can record what causes the error. Afterwards, we can send it to an expert or search for a fix ourselves.

The video covers all 3 parts.

Part 1: Running Process Monitor & Configuring Filters
1. Run the Process Monitor application.
2. Include the processes that you want to track the activity on. For this example, you want to include Notepad.exe in the (Include) Filters.
3. Click Add, and click OK.
4. From the Options menu, click Select Columns.
5. Under “Event Details”, enable Sequence Number, and click OK.

You can add multiple entries as well, in case you want to track more processes along with Notepad.exe. To keep this example simpler, let’s only track Notepad.exe.

You’ll now see the Process Monitor main window tracking the list of registry and file accesses by processes real-time, as and when they occur.

Part 2: Capturing Events
6. Open Notepad.
7. Switch to Process Monitor window.
8. Enable the “Capture” mode (if it’s not already ON). You can see the status of the “Capture” mode via the Process Monitor toolbar.
9. The highlighted button above is the “Capture” button, which is currently disabled. You need to click that button to enable capturing of events.
10. Important: Cleanup the existing events list using Ctrl + X key sequence and start afresh.
11. Switch back to Notepad.
12. To reproduce the problem, try writing to the HOSTS file and saving it. Windows offers to save the file with a different name, or in a different location. So, what happens under the hood when you save to HOSTS file? Process Monitor shows that exactly.
13. Switch to Process Monitor window, and turn off Capturing (Ctrl + E) as soon as you encounter the problem. Important Note: You need to do all that as quickly as you can in order to not record unneeded data.

The log file above tells us that Notepad encountered an ACCESS DENIED error when writing to the HOSTS file.

The solution would be to simply run Notepad elevated (right-click and choose “Run as Administrator”) to be able to write to HOSTS file successfully.

Part 3: Saving the Output
14. In the Process Monitor window, select the File menu and click Save.
15. Select Native Process Monitor Format (PML), mention the output file name and Path, save the file.
16. Right-click on the Logfile.PML file, click Send To, and choose Compressed (zipped) folder. You can now send it to an expert.

To recap, Task Manager shows what processes use the disk intensively at the current time.

Windows Performance Recorder / Windows Performance Analyzer show who used the disk during a longer time period, although they were showing the activity as belonging to the System process instead of our process.

Using Process Monitor we could identify our processes' entire activity and determine why one is slower than the other.

04. Process Explorer

Process Explorer is similar to Task Manager in many ways, as both serve the same purpose. Process Explorer is more verbose and shows much more information about different parts of the system.

Even if it doesn't look as pretty as Task Manager, this tool was developed for Windows 2000 initially. The Task Manager of Windows 2000 offered much fewer options than the one for Windows 10. Now, Task Manager and Process Explorer are interchangeable in most cases.

05. Windows API

The previous chapters cover most cases where we encounter an error, we diagnose it, and identify it.

There is always the very rare case where a tool just doesn't cut it. In this case we can use the API offered by Windows to extract what information we want from a program and/or the system.

More precisely, we are interested in the Process Status API from Windows. This offers basic functionality to extract information from the system and its processes.

We will only briefly go over the functions in the Task, so feel free to try more of the functionality of the API.

Tasks

The tasks can be found for the Windows sessions can be found here:

01. [10p] Task Manager

Go through the Task Manager introduction and/or watch the video below.

:!: Which process is constantly reading or writing to your hard disk?

  1. Open Task Manager and select the Details tab
  2. Right-click on the column header and click “Select columns”
  3. Enable the “I/O Read Bytes” and “I/O Write Bytes” checkboxes
  4. Sort the list by clicking on a column header to see the most read-intensive and/or write-intensive processes
  5. Identify the process and mention if and why the I/O traffic is justified

  • I/O Read Bytes is the number of bytes read in input/output operations generated by a process, including file, network, and device I/Os.
  • I/O Write Bytes is the number of bytes written in input/output operations by a process, including file, network, and device I/Os.
  • I/O Read Bytes and I/O Write Bytes directed to console are not counted.

02. [20p] Windows Performance Recorder & Analyzer

For this task we will use Logs.exe and GoodLog.exe inside the Task-02 folder.

We want to capture the activity of both Logs.exe and GoodLog.exe. To do this we will start WPR and record an execution of Logs.exe and GoodLog.exe afterwards. We will then analyze the results using WPA, concentrating on the Disk activity and compare the results with Task Manager.

:!: To do this you can follow the steps below or the video.

  1. Open Windows Performance Recorder by searching for it in the search bar
  2. Click “More Options” and choose the following from Resource Analysis: CPU usage, Disk I/O activity, File I/O activity, Registry I/O activity, Networking I/O activity, Heap usage, Pool usage, and VirtualAlloc usage
  3. Using Command Prompt, navigate to the folders where the Logs/GoodLog executables are located
  4. Start the recording and run GoodLog & Logs afterwards
  5. Save the recording and open it with WPA
  6. Double click on Storage to analyze the storage operations
  7. Click on Utilization by Disk on the upper-left corner and select: Utilization by Process, Path Name, and Stack
  8. Run the executables again, this time observing them inside Task Manager

Can you guess why there is disk activity, but WPA does not show it?

Click to display ⇲

Click to hide ⇱

The way Windows Performance Recorder records activity is based on events generated by the Windows kernel. It registers to track the events, listens to them, and during the recording period it constantly samples which process uses which resource at the time of sampling. It sums up the number of times that a process was caught doing something.

In our case, the two processes want to write to the disk, but they are not the ones that get to do the actual writing. They tell the system that they want to write, and the System process schedules the writing.

The reason for this is targeting a more efficient disk writing, as the System process is trying to minimize the impact to the disk. This is why our process's writing is passed over to the System process.

03. [20p] Process Monitor

Task A - Checking Logging File

For this task we will use Logs.exe and GoodLog.exe inside the Task-03/Task-A folder.

We want to see what is the difference between Logs.exe and GoodLog.exe. After running both programs we can see that the output is identical. Still, Logs.exe takes much longer to complete compared to GoodLog.exe. We will use Process Monitor to determine the problem.

:!: Follow the steps to analyze the programs.

  1. Open Process Monitor and add a filter for Logs.exe (and GoodLog.exe afterwards). To do this create a filter of the format 'Process Name' 'is' 'Logs.exe'.
  2. Open the Command Prompt and run Logs.exe, while Process Monitor is running.
  3. Save the capture and reset all events (Ctrl + X).
  4. Repeat the first 3 steps for GoodLog.exe.
  5. Open both saved PML files and notice the differences.

Click to display ⇲

Click to hide ⇱

On the left-hand side it is shown the faster logging process, and on the right-hand side the slower one. Look in the red highlighted area to see the difference.

On the left-hand side the logging file is opened, followed by continuous writing. On the right-hand side the file is opened and closed for every writing operation which explains the significant slowdown.

We now know why one program is faster than the other.

If we want to go more in depth to find what code generated an event we can still use Process Monitor.

  1. Go to Options → Configure Symbols. Here you can configure the path for the symbols and the source code.
  2. Now we can double click on events like CreateFile and check the code by going to the Stack tab and clicking on an entry from our program.

Task B - Investigating a Handle Leak

For this task we will use bad.exe and good.exe inside the Task-03/Task-B folder.

Both executables have the same outcome. The difference is their running time.

:!: Identify the problem. Hint: Process Monitor

Click to display ⇲

Click to hide ⇱

A handle leak consists of a process that opens files and does not close them. On modern computers if this action is performed millions of times, the system may become unresponsive and will either experience an overall slowdown or the application that causes this will eventually crash.

You may think that millions of handles are impossible to reach, so it is not worth paying attention to this problem. However, imagine that there are services running on servers for years. As an example, having a handle leak every 2 seconds amounts for over 10 million handle leaks in a year.

How should such problems be investigated?

Open up a terminal and run bad.exe. Check out the Details tab in Task Manager after adding the Handles column.

04. [10p] Process Explorer

For this task we will use HandleLeak.exe inside the Task-04 folder.

We want to check what is wrong with this HandleLeak.exe program (ignore its name :-)). For Task Manager, on the Details tab add the Handles column.

Run the program, search for it inside Task Manager, and look at the Handles column. The number of handles keeps growing. This is clearly a problem.

:!: How do we investigate it? What is wrong with this file?

  1. Run Process Explorer as administrator. It is similar to Task Manager.
  2. Select the process you are interested in (HandleLeak) and press Ctrl + H. This will display all open Handles of the process.
  3. Notice that the leak happens on the leak.txt file.
  4. To find out more exact information about what causes this we will have to use Process Monitor with a filter for HandleLeak.exe.

05. [20p] Windows API

For this task we will use the project skeleton inside the Task-05 folder.

:!: We want to create a simple Console Task Manager with functions from PSAPI.

This simple Task Manager will print two CSV tables. The first will contain system information provided by PSAPI. The second will print memory information for any accessible process and their name.

This list of functions from PSAPI should catch your eye:

Follow the hints inside the skeleton to see how you will put them all together.

If everything goes smoothly you should see these two CSV tables in your console:

System Data
 CommitTotal, CommitLimit, CommitPeak, PhysicalTotal, PhysicalAvailable, SystemCache, KernelTotal, KernelPaged, KernelNonpaged, PageSize, HandleCount, ProcessCount, ThreadCount
 783370, 3211133, 876957, 1572733, 905287, 917590, 110538, 65666, 44872, 4096, 55952, 150, 1401
Process Data
 ProcessName, PageFaultCount, PeakWorkingSetSize, WorkingSetSize, QuotaPeakPagedPoolUsage, QuotaPagedPoolUsage, QuotaPeakNonPagedPoolUsage, QuotaNonPagedPoolUsage, PagefileUsage, PeakPagefileUsage
 sihost.exe, 11355, 24620, 14116, 250, 235, 19, 17, 5812, 6180
 svchost.exe, 17407, 16264, 8996, 160, 160, 14, 14, 5104, 5480
 svchost.exe, 17702, 29644, 15852, 278, 264, 26, 20, 6448, 7312
 taskhostw.exe, 9978, 16236, 9260, 183, 179, 30, 28, 6580, 7140
 explorer.exe, 363502, 143464, 116820, 1189, 1073, 124, 102, 62260, 85940
 svchost.exe, 13452, 20052, 15104, 238, 235, 16, 16, 3956, 4112
 StartMenuExperienceHost.exe, 24412, 65908, 22660, 574, 536, 32, 27, 19304, 23456
 RuntimeBroker.exe, 12519, 25888, 6088, 254, 213, 20, 16, 5320, 7440
 SearchUI.exe, 219211, 230984, 196976, 1070, 989, 119, 107, 140180, 155912
 RuntimeBroker.exe, 54907, 48524, 45580, 533, 440, 42, 32, 19536, 22592
 SkypeApp.exe, 13196, 42504, 7800, 453, 453, 37, 37, 14156, 14216
 SkypeBackgroundHost.exe, 3378, 11908, 1320, 124, 124, 8, 8, 2004, 2176
 RuntimeBroker.exe, 15080, 20336, 11592, 256, 193, 20, 15, 3340, 4936
 SecurityHealthSystray.exe, 2742, 8808, 2256, 149, 144, 10, 9, 1736, 1896
 vmtoolsd.exe, 66129, 39368, 28588, 316, 302, 30, 29, 19000, 30292
 OneDrive.exe, 26632, 73076, 23408, 527, 506, 67, 48, 21124, 34376
 RuntimeBroker.exe, 5798, 16256, 3384, 173, 162, 12, 11, 2440, 3088
 svchost.exe, 3619, 12040, 3236, 153, 152, 15, 14, 2940, 3304
 MyTaskManager.exe, 893, 3256, 3104, 26, 26, 5, 4, 1336, 1336
 msvsmon.exe, 5289, 20004, 20000, 464, 464, 14, 14, 141756, 141756
 ...

06. [20p] CHKDSK & CrystalDiskMark

For this task we will need to install CrystalDiskMark.

For this task we want to see and estimate the health and speed of our storage. This can vary considerably and needs to be inspected regularly to see if we need to replace our HDD/SSD.

CHKDSK Health Checks

CHKDSK is a standard tool inside Windows that allows us to get basic information about our disks. It tells us how much memory we have, how much is free, how much we used. It also tells us if the disk has bad sectors, which can be a sign of a failing HDD/SSD.

:!: After running you should see something similar to the image below.

  1. Open CMD as Administrator
  2. Run CHKDSK
  3. Discuss with your assistant if the program detected Bad Sectors

CHKDSK can also try to fix bad sectors on start. Bad sectors appear due to external alteration of the storage medium. In order to try fixing bad sectors you can use /f, but this requires a restart so do it on your free time.

CrystalDiskMark Benchmarking

CrystalDiskMark is a simple, yet very efficient tool for testing how fast our HDD/SSD actually is. It features several testing suites, but, for example purposes, we will use only the default one. We should use this tool at least once every 1-2 years to see if there is any significant slowdown in our system. You can also use this tool to check if the Second-hand HDD you bought for 2$ actually works. ;)

Press “All” to start running the tests.

:!: After running you should see something similar to the image below.

:!: Explain what each test benchmarked and why the values differ by answering the next questions.

  1. Why are Sequential Tests faster than Random Tests?
  2. How does the Block Size influence speed?
  3. When do Worker Queues/Threads influence speed?
  4. What is a real-life sequential task? What is a real-life random task?
  5. When do we know we should replace our SSD/HDD?
  6. Would defragmenting our HDD help with Random Tests?

07. [10p] Feedback

:!: Please take a minute to fill in the feedback form for this lab.