Differences
This shows you the differences between two versions of the page.
|
ep:labs:09 [2021/12/04 19:20] andrei.mirciu [03. Network Monitoring] |
ep:labs:09 [2025/05/15 21:17] (current) andrei.mirciu [Resources] |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== Lab 09 - Memory, CPU and Network Monitoring (Windows) ====== | + | ====== Lab 09 - Machine Learning ====== |
| ===== Objectives ===== | ===== Objectives ===== | ||
| - | * **Find the root cause if the system runs out of memory** with Task Manager, Windows Performance Recorder, Windows Performance Analyzer, Visual Studio, VMMap. | + | * Understand basic concepts of machine learning |
| - | * **Monitor the CPU usage** with Task Manager, Windows Performance Recorder, Windows Performance Analyzer. | + | * Remember examples of real-world problems that can be solved with machine learning |
| - | * **Check the amount of network traffic generated by processes** with Task Manager, Windows Performance Recorder, Microsoft Network Monitoring, Wireshark. | + | * Learn the most common performance evaluation metrics for machine learning models |
| + | * Analyse the behaviour of typical machine learning algorithms using the most popular techniques | ||
| + | * Be able to compare multiple machine learning models | ||
| - | <note important> | + | ===== Resources ===== |
| - | You can download the **Windows 10 VM** via [[https://ctipub-my.sharepoint.com/:u:/g/personal/radu_mantu_upb_ro/EXSrHQMCkWBEpGYseFEmnnABCA1hyb1oGWMUhnnHx8LIdQ?e=I0pxHg | OneDrive]]. | + | In this lab, we will study basic performance evaluation techniques used in machine learning, covering elementary concepts such as classification, regression, data fitting, clustering and much more. |
| - | If you need to use VirtualBox, you can use this //.ovf// version to import the VM (just on OneDrive) | + | You will work in an environment that is easy to use, and provides a couple of tools like manipulating data and visualizing results. We will use a **Jupyter Notebook** hosted on **Google Colab**, which comes with a variety of useful tools already installed. |
| - | [[https://ctipub-my.sharepoint.com/:u:/g/personal/cezar_craciunoiu_upb_ro/EZYR_YFyHx5GiHf5yBNuiyYB-zXhIaTNzJ8o8Ri2M8l5Mw?e=9qxrde | OneDrive]]. | + | |
| - | There is also the option to download as a torrent {{:ep:labs:ep_win10_vm.7z.torrent.txt}}. | + | The exercises will be solved in Python, using popular libraries that are usually integrated in machine learning projects: |
| - | DokuWiki is not configured to accept //.torrent// files so remove the //.txt// extension. | + | |
| - | After that, you know what to do... | + | |
| - | Alternatively, you can install the following on your own Windows machine: | + | * [[https://scikit-learn.org/stable/documentation.html|Scikit-Learn]]: fast model development, performance metrics, pipelines, dataset splitting |
| - | * **[[https://go.microsoft.com/fwlink/?linkid=2120254 | ADK]]** - make sure to check //**Windows Performance Analyser**// and //**Windows Performance Recorder**//. | + | * [[https://pandas.pydata.org/pandas-docs/stable/|Pandas]]: data frames, csv parser, data analysis |
| - | * **[[https://visualstudio.microsoft.com/downloads/ | Visual Studio Community Edition]]** - select //C++ development//. | + | * [[https://numpy.org/doc/|NumPy]]: scientific computation |
| - | * **[[https://docs.microsoft.com/en-us/sysinternals/downloads/sysinternals-suite | Sysinternals suite]]** | + | * [[https://matplotlib.org/3.1.1/users/index.html|Matplotlib]]: data plotting |
| - | </note> | + | As datasets, we will use some public corpora provided by the Kaggle community: |
| - | <note important> | + | * [[https://www.kaggle.com/uciml/pima-indians-diabetes-database/data|Classification Dataset]] |
| - | If Visual Studio prompts you with an "Expired" message, you will have to log in with your (university) account. | + | * [[https://www.kaggle.com/zaraavagyan/weathercsv|Regression dataset]] |
| - | </note> | + | |
| - | ===== Proof of Work ===== | + | You can also check out these cheet sheets for fast reference to the most common libraries: |
| - | Before you start, create a [[http://docs.google.com/|Google Doc]]. Here, you will add screenshots / code snippets / comments for each exercise. Whatever you decide to include, it must prove that you managed to solve the given task (so don't show just the output, but how you obtained it and what conclusion can be drawn from it). If you decide to complete the feedback for bonus points, include a screenshot with the form submission confirmation, but not with its contents. | + | **Cheat sheets:** |
| - | When done, export the document as a //pdf// and upload in the appropriate assignment on Moodle. Remember, the cut-off time is 15 minutes after the lab ends. | + | * [[https://perso.limsi.fr/pointal/_media/python:cours:mementopython3-english.pdf)|python]] |
| + | * [[https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Numpy_Python_Cheat_Sheet.pdf|numpy]] | ||
| + | * [[https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Python_Matplotlib_Cheat_Sheet.pdf|matplotlib]] | ||
| + | * [[https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Scikit_Learn_Cheat_Sheet_Python.pdf|sklearn]] | ||
| + | * [[https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdf|pandas]] | ||
| - | ===== Contents ===== | + | <solution -hidden> |
| - | {{page>:ep:labs:09:meta:nav&nofooter&noeditbutton}} | + | [[https://colab.research.google.com/drive/1aeV9PGF_uxBA3FoKNMEzsiXMxjVSCcm4?usp=sharing|Solution]] |
| + | </solution> | ||
| - | ===== Introduction ===== | + | ===== Tasks ===== |
| - | The Windows operating system contains plenty of built-in tools to analyze its resource usage. The most famous one is probably the Windows Task Manager, as it highlights resource usage of individual processes and gives admins and users the option to kill any misbehaving ones. | + | ==== Google Colab Notebook ==== |
| - | The Performance Monitor and Resource Monitor are two additional tools that admins and experienced Windows users may use to analyze performance or any resources related issues on Windows machines. | + | For this lab, we will use Google Colab for exploring performance evaluation in machine learning. Please solve your tasks [[https://github.com/vladastefanescu/machine-learning-introduction/blob/main/Machine_Learning_Introduction.ipynb|here]] by clicking "**Open in Colaboratory**". |
| - | ==== 01. RAM Monitoring ==== | + | You can then export this python notebook as a PDF (**File -> Print**) and upload it to **Moodle**. |
| - | Processes are dynamically allocating and using memory. Thus, it is possible to have memory allocation spikes. **Task Manager** shows you the amount of memory allocated to a process, by checking the **Peak Working Set** value in the Details tab. We can notice that the **Peak Working Set** can be sometimes significantly greater than the **Working Set**. | + | ===== Feedback ===== |
| - | ** What can we do if the system runs out of memory, but we do not know what causes this? ** | + | Please take a minute to fill in the **[[https://forms.gle/NpSRnoEh9NLYowFr5 | feedback form]]** for this lab. |
| - | Let's imagine a situation when a system encounters issues if some conditions are met, and these conditions can be reproduced. To find out the root cause of this problem, we can start the **Windows Performance Recorder** tool. We need to select the following profiles for performance recording: **Heap usage**, **Pool usage** and **VirtualAlloc usage**. Moreover, we need to make sure that the detail level is set to verbose and the logging mode selected is for memory. | ||
| - | Following the steps below, we can highlight how such a situation should be treated: | ||
| - | ^ Part 1: Emphasize memory usage for each process ^^ | ||
| - | | **1.** Run, in parallel with the Windows Performance Recorder application, a program that allocates memory for a while and then stops. || | ||
| - | | **2.** Save the capture and open it in Windows Performance Analyzer. || | ||
| - | | **3.** Right click on the Memory section and select the option to **Add all Memory graphs to Analysis View**. || | ||
| - | | **4.** Scroll down until you find the **Virtual Memory Snapshots** graph, which shows you the memory usage for all processes. || | ||
| - | Going further, let's assume that we did not find any memory leaks for our program and the memory used by it gets successfully freed at the end. | ||
| - | ** How can we determine which part of the program is responsible for generating a memory spike? ** | ||
| - | A solution to this question can be achieved by using another tool from the Sysinternals suite, more precisely [[https://technet.microsoft.com/en-us/sysinternals/vmmap.aspx | VMMap]]. With this tool, we can view a process's memory allocations and usage. | ||
| - | ^ Part 2: Emphasize memory spike of a process ^^ | ||
| - | | **1.** Go to Options -> Configure Symbols and make sure the paths are set correctly to point to the Microsoft Symbol Server and to the program's source files. || | ||
| - | | **2.** Click OK and navigate to File -> Select Process. || | ||
| - | | **3.** Select the **Launch and trace a new process** tab, choose the corresponding executable application and the directory where it will run. || | ||
| - | | **4.** Click OK and let the program run. || | ||
| - | | **5.** To view the latest memory allocations, you need to click on the **Heap** row in the upper-part of the VMMap tool, and hit F5 (refresh) from time to time. || | ||
| - | | **6.** Select one memory allocation from the bottom-part of the VMMap tool and press the **Heap Allocations** button, located in the bottom right corner. || | ||
| - | | **7.** Press the **Stack** button to observe where the allocation occurred. By clicking on the **Source** button, we can view the actual code for the allocation. || | ||
| - | ==== 02. CPU Monitoring ==== | ||
| - | Let's consider the same scenario as the one presented in the previous section. | ||
| - | Monitoring the CPU usage presents similar issues to the ones encountered when monitoring the memory usage. Task Manager can help us to find out the current CPU usage for a process. | ||
| - | **How can we catch an event when the CPU usage briefly spikes up and then goes back to normal?** | ||
| - | Using Task Manager, we would need to continuously have someone watching what is happening, to catch the moment when the spike occurs. To overcome this limitation, we can use Windows Performance Recorder, by selecting the **CPU usage** profile. | ||
| - | |||
| - | Following the steps below, we can highlight how such a situation should be treated: | ||
| - | |||
| - | ^ Emphasize the CPU usage of a program ^^ | ||
| - | | **1.** Run, in parallel with the Windows Performance Recorder application, a program that generates CPU usage for a while and then stops. || | ||
| - | | **2.** Save the capture and open it in Windows Performance Analyzer. || | ||
| - | | **3.** Right click on the Computation section and select the option to **Add all Computation graphs to Analysis View**. || | ||
| - | | **4.** The **CPU Usage** graph shows the impact of our program and helps us to determine who is generating this load. || | ||
| - | |||
| - | <note> | ||
| - | To further debug this situation, as in the previous case of the memory, if the program that runs the problematic process was not written by us, we need to check whether it is useful or not. If the answer is no, we should definitely stop it. In case it is useful, but the program is not ours, we can try to find an update to fix the problem, or report the issue to the producer. If the program is written by us (this course - Performance Evaluation - targets such processes developed by us), then it is important to determine what causes this problem. Unfortunately, unlike in the case of monitoring the memory usage, there is no tool that can show us the entire stack, so we need to create one. | ||
| - | </note> | ||
| - | |||
| - | ==== 03. Network Monitoring ==== | ||
| - | |||
| - | As you probably already know, the easiest and fastest way to monitor the amount of network traffic generated by a process on a Windows operating system is through using Task Manager. | ||
| - | |||
| - | However, if the process generating the network activity is unknown, we can use Windows Performance Recorder, by selecting the **Networking I/O activity** profile. After we save the capture and open it in Windows Performance Analyzer, we will observe that the information offered is for the total use of the network, rather than per process statistics. | ||
| - | |||
| - | Due to this reason, we are calling upon another tool developed by Microsoft, more precisely **[[https://www.microsoft.com/en-us/download/details.aspx?id=4865 | Microsoft Network Monitor]]**. After we install it, start it using “Run as administrator” and select the network interface through which the traffic is expected to pass (cable, wifi, etc.), we will be able to capture the frames. | ||
| - | |||
| - | As in the case of the CPU, inspecting the events taking place on the network involves some amount of work for the analyst. However, since this is a simple case, we can just expand the view on the traffic generated by Winhttp.exe, and notice the request for putty.exe. If it is still not clear why some requests are there or why they last so long, we can integrate the application that we wish to investigate with Process Monitor. This way, we can insert logging elements to find out what request are made and how long they take. Furthermore, the part with timing the requests and traffic can be determined straight from Network Monitor, by considering the times of the packets. | ||
| - | |||
| - | Last but not least, another well-known tool that you should already be familiar with from the Linux network monitoring lab is [[https://www.wireshark.org/download.html | Wireshark]]. This can be used for displaying all traffic generated on a HTTP connection (it can also be HTTPS, as long as we control the server, but this is not in the scope of this tutorial). | ||
| - | |||
| - | <note tip> | ||
| - | When installing **Wireshark**, mare sure to select the **64-bit** version and accept the default settings. After that, you can start it and select the interface that you want to listen to. | ||
| - | </note> | ||
| - | |||
| - | ===== Tasks ===== | ||
| - | <note warning> | ||
| - | The password for {{:ep:laboratoare:logs2.zip | log2.zip}} and {{:ep:labs:build.zip | build.zip}} is: //parola// | ||
| - | </note> | ||
| - | {{namespace>:ep:labs:09:contents:tasks&nofooter&noeditbutton}} | ||
