Differences

This shows you the differences between two versions of the page.

Link to this comparison view

ep:labs:03:contents:tasks:ex1 [2020/07/30 21:21]
gheorghe.petre2608 [03. [5p] Iotop]
ep:labs:03:contents:tasks:ex1 [2025/02/11 23:27] (current)
cezar.craciunoiu
Line 1: Line 1:
-==== 01. [5pRotational delay - IOPS calculations ​====+==== 01. [30pVmstat ​==== 
 +The **vmstat** utility provides a good low-overhead view of system performance. Since **vmstat** is such a low-overhead tool, it is practical to have it running even on heavily loaded servers when it is needed to monitor the system’s health.
  
-=== Good to know ===+=== [10p] Task A - Monitoring stress ​=== 
 +Run **vmstat** on your machine with a 1 second delay between updates. Notice the CPU utilisation (info about the output columns [[https://​medium.com/​@damianmyerscough/​vmstat-explained-83b3e87493b3|here]]).
  
-Every disk in your storage system has a maximum theoretical IOPS value that is based on a formulaDisk performance and IOPS is based on three key factors:+In another terminal, use the **stress** command to start N CPU workers, where N is the number of cores on your system. 
 +Do not pass the number directly. Instead, use command substitution.
  
-  * **Rotational speed**. Measured in RPM, mostly 7,200, 10,000 or 15,000 RPM. A higher rotational speed is associated with a higher performing disk. +Note: if you are trying ​to solve the lab on //fep// and you don't have **stress** installed, try cloning and compiling [[https://github.com/ColinIanKing/stress-ng|stress-ng]].
-  * **Average latency**. The time it takes for the sector of the disk being accessed ​to rotate into position under a read/write head. +
-  * **Average seek time**. The time (in ms) it takes for the hard drive’s read/write head to position itself over the track being read or written. +
-  * **Average IOPS**: Divide 1 by the sum of the average latency in ms and the average seek time in ms (1 (average latency in ms + average seek time in ms). +
- +
-To calculate the IOPS range, use this formula: Average IOPS: Divide 1 by the sum of the average latency in ms and the average seek time in ms (1 / (average latency in ms + average seek time in ms). +
- +
-Let's calculate the Rotational Delay - RD for a 10K RPM drive: +
- +
-  ​Divide 10000 RPM by 60 seconds: ​**''​10000/​60 = 166 RPS''​*+
-  * Convert 1 of 166 to decimal**''​1/166 = 0.006 seconds per Rotation''​** +
-  * Multiply the seconds per rotation by 1000 milliseconds (6 MS per rotation). +
-  * Divide the total in half (RD is considered half a revolution around a disk): **''​6/2 = 3 MS''​** +
-  * Add an average of 3 MS for seek time: **''​3 MS + 3 MS = 6 MS''​** +
-  * Add 2 MS for latency (internal transfer): **''​6 MS + 2 MS = 8 MS''​** +
-  * Divide 1000 MS by 8 MS per I/O: **''​1000/​8 = 125 IOPS''​** +
- +
-=== IOPS calculations === +
- +
-Calculate the rotational delay (RD) for a 5400 RPM drive.+
  
 <​solution -hidden> <​solution -hidden>
-As shown in the //"​Calculating IOs Per Second"//​ section+**Task A:**\\ 
-<​code>​ +<​code ​bash
-5400 / 60 = 90 RPS +$ vmstat -w -n 
-1/90 = 0.011 seconds per Rotation +$ stress -c $(nproc)
-0.011 * 1000 = 11ms per Rotation +
-11 / 2 = 5.5ms RD (Rotational Delay = half a revolution around a disk) +
-add approx. 3ms seek time => 8.5ms +
-add 2ms latency => 10.5ms +
-Divide 1000ms by 10.5ms per I/O => approx. 95 IOPS+
 </​code>​ </​code>​
 </​solution>​ </​solution>​
-==== 02. [5p] Iostat ==== 
  
-=== Good to know ===+=== [10p] Task B - How does it work? === 
 +Let us look at how **vmstat** works under the hood. We can assume that all these statistics (memory, swap, etc.) can not be normally gathered in userspace. So how does **vmstat** get these values from the kernel? Or rather, how does any process interact with the kernel? Most obvious answer: //**system calls**//.
  
-<​code>​ +<​code ​bash
-#iostat [ -x for extended statistics, -d to display device stastistics only, -m for displaying r/w in MB/s ] +$ strace vmstat
-iostat -xdm +
- +
-#iostat with -p for specific device statistics +
-iostat -xdm -p sda+
 </​code>​ </​code>​
  
-=== Monitoring the behaviour ===+//"All well and good. But what am I looking at?"//
  
-  * Run //iostat -x 1 5//+What you //should// be looking at are the system calls after the two **write**that display ​the output header ​(hint: it has to do with **/proc/** file system). So, what are these files that **vmstat** opens?
-  * Considering ​the last two outputs provided by the previous command, calculate the efficiency of IOPS for each of them. Does the amount of data written per I/O increase or decrease? +
- +
-<​note>​ +
-How to's +
- +
-  ​Divide the kilobytes read (//rkB/s//) and written (//wkB/s//) per second by the reads per second ​(//r/s//) and the writes per second (//w/s//). +
-  ​If you happen to have quite a few [[https://​en.wikipedia.org/​wiki/​Loop_device|loop devices]] in your **iostat** output, find out what they are exactly:+
  
 <code bash> <code bash>
-df -kh /dev/loop* +file /proc/meminfo 
-</code> +$ cat /proc/meminfo
-</note>+
  
- +$ man 5 proc
-<​solution -hidden>​ +
-The way to calculate the efficiency of IOPS is to divide the reads per second //(r/s)// and writes per second //(w/s)// by the kilobytes read //(rkB/s)// and written //(wkB/s)// per second. +
- +
-Example: the amount of data written per I/O for ///​dev/​sda//​ increases during each iteration:​ +
- +
-{{ :​ep:​labs:​ep2017_l3_ex01.png?​700 |}} +
- +
-<​code>​ +
-53040/105 = 505KB per I/O +
-71152/102 = 697KB per I/O+
 </​code>​ </​code>​
  
-If everything is zero in iostat - perform some I/O operations... +The manual should contain enough information about what these kernel interfaces can provide. However, if you are interested ​in //how// the kernel generates the statistics in **/​proc/​meminfo** (for example), a good place to start would be [[https://​elixir.bootlin.com/​linux/​v4.15/source/​fs/​proc/​meminfo.c|meminfo.c]] (but first, ​[[https://​ocw.cs.pub.ro/​courses/​so2|SO2 wiki]]).
-</solution>​ +
-==== 03. [5pIotop ====+
  
-=== Good to know ===+=== [10p] Task C - USO flashbacks (1) ===
  
-**Iotop** is an utility similar ​to top command, that interfaces with the kernel to provide per-thread/​process I/O usage statistics.+Write a one-liner that uses **vmstat** to report complete **disk ​statistics** and sort the output in **descending** order based on **total reads** column.
  
-<code+<note tip
-Debian/​Ubuntu Linux install iotop +You can eliminate the first two header ​lines from the **vmstat** output ​using ''​tail -n +3''​
-$ sudo apt-get install iotop +</note>
- +
-How to use iotop command +
-$ sudo iotop OR $ iotop +
-</​code>​ +
- +
-Supported options by iotop command: +
- +
-| **Options** | **Description** ^^ +
-| --version | show program’s version number and exit || +
-| -h, --help | show this help message and exit || +
-| -o, --only | only show processes or threads actually doing I/O || +
-| -b, --batch | non-interactive mode || +
-| -n NUM, --iter=NUM | number of iterations before ending [infinite] || +
-| -d SEC, --delay=SEC | delay between iterations [1 second] || +
-| -p PID, --pid=PID | processes/​threads to monitor [all] || +
-| -u USER, --user=USER | users to monitor [all] || +
-| -P, --processes | only show processes, not all threads || +
-| -a, --accumulated | show accumulated I/O instead of bandwidth || +
-| -k, --kilobytes | use kilobytes instead of a human friendly unit || +
-| -t, --time | add a timestamp on each line (implies –batch) || +
-| -q, --quiet | suppress some lines of header (implies –batch) || +
- +
-=== Monitoring ​the behaviour === +
- +
-Datafile: {{:​ep:​laboratoare:​dummy.sh|dummy.sh}}. +
- +
-  ​Run iotop (install it if you do not already have it) in a separate shell showing only processes or threads actually doing I/O. +
-  ​Inspect the script code (**dummy.sh**) to see what it does. +
-  * Monitor the behaviour of the system with iotop while running the script. +
-  * Identify the PID and PPID of the process running the dummy script and kill the process ​using command line from another shell (sending SIGINT signal to both parent & child processes)+
-  * Hint - [[https://​superuser.com/​questions/​150117/​how-to-get-parent-pid-of-a-given-process-in-gnu-linux-from-command-line|How to get parent PID of a given process in GNU/Linux from command line?]]+
  
 <​solution -hidden> <​solution -hidden>
-{{:​ep:​laboratoare:​lab3-ex4.png?​600}} +<code bash
- +vmstat -wdn tail -+3 | sort -nrk 2
-Find PPID from PID: ps -o ppid= -p PID +
- +
-Send SIGINT signal: kill -SIGINT PID,PPID +
-</solution+
-==== 04. [20p] Monitor I/O with vmstat ​and iostat ==== +
-We said in the beginning that the disk I/O subsystems are the slowest part of any system. This is why the I/O monitoring is so important, maximizing the performance of the slowest part of a system resulting in an improvement of the performance of the entire system. +
- +
-=== [10p] Task A Script === +
- +
-Write a script that reads the data into memory and generates a text file 500 times larger, by concatenating the contents of the following novel {{:​ep:​labs:​olivertwist.txt|olivertwist.txt}} to itself. +
- +
-<​solution ​-hidden>​ +
-<​code>​ +
-if __name__ == '​__main__':​ +
-    text_file1 = open("​OliverTwist.txt",​ "​r"​) +
-    text_file2 = open("​OliverTwistLarge.txt",​ "w+") +
-    lines_file1 = text_file1.readlines() +
-    for x in range(0, 500): +
-    text_file2.writelines(lines_file1)+
 </​code>​ </​code>​
 </​solution>​ </​solution>​
- 
-=== [10p] Task B - Monitoring behaviour === 
- 
-Now we want to analyze what is happening with the I/O subsystem during an expensive operation. Monitor the behavior of the system while running your script using **vmstat** and **iostat**. 
- 
-<note tip> 
-Understanding vmstat IO section: 
-  * **bi** - column reports the number of blocks received (or “blocks in”) from a disk per second. 
- 
-  * **bi** - column reports the number of blocks sent (“blocks out”) to a disk per second. 
-</​note>​ 
- 
- 
ep/labs/03/contents/tasks/ex1.1596133279.txt.gz · Last modified: 2020/07/30 21:21 by gheorghe.petre2608
CC Attribution-Share Alike 3.0 Unported
www.chimeric.de Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0