Lab 04 - Network Monitoring (Linux)

Objectives

  • Dive into the inner workings of previously studied traffic monitoring / filtering tools
  • Discuss methods of path discovery
  • Provide an introduction to protocol options

Contents

Proof of Work

Before you start, create a Google Doc. Here, you will add screenshots / code snippets / comments for each exercise. Whatever you decide to include, it must prove that you managed to solve the given task (so don't show just the output, but how you obtained it and what conclusion can be drawn from it). If you decide to complete the feedback for bonus points, include a screenshot with the form submission confirmation, but not with its contents.

When done, export the document as a pdf and upload in the appropriate assignment on moodle. Remember, the cut-off time is 15m after the lab ends.

Tasks

01. [40p] Primer / Reminder

Pro tip #1: since you'll be using man a lot in this exercise, add this to your .bashrc or .zshrc:

# color schemes for man pages
man() {
    LESS_TERMCAP_mb=$'\e[1;34m'   \
    LESS_TERMCAP_md=$'\e[1;32m'   \
    LESS_TERMCAP_so=$'\e[1;33m'   \
    LESS_TERMCAP_us=$'\e[1;4;31m' \
    LESS_TERMCAP_me=$'\e[0m'      \
    LESS_TERMCAP_se=$'\e[0m'      \
    LESS_TERMCAP_ue=$'\e[0m'      \
    command man "$@"
}

Source the file and test that it works.

[20p] Task A - tcpdump

tcpdump is a network traffic monitoring tool. At its core, it uses libpcap which in turn uses a technology called Extended Berkley Packet Filter (eBPF).

BPF was first proposed around 1995 when filtering mechanisms (and firewalls) were still a novel concept and were based on interpreters. BPF (now referred to as Classic BPF - cBPF) was the initial version of a Motorola inspired virtual ISA (i.e.: had no hardware implementation – think CHIP-8). eBPF is basically still BPF but more compatible with 64-bit architectures so that Just In Time (JIT) translators have an easier time running the code.

At first, the whole idea was to compile packet filtering programs and attach them to sockets in kernelspace. These programs would filter out packets that userspace processes would not be interested in. Consequently, this would reduce the quantity of data copied over the kernelspace/userspace boundary, only to ultimately be discarded.

Today, eBPF is used heavily for system profiling by companies such as Netflix and Facebook. Linux has had a kernel VM capable of running and statically analyzing eBPF code since around 2006. tcpdump is one of the few examples that still use it for its original purpose.

The Task

Use tcpdump to output outgoing NTP queries and incoming http(s) responses. Use the -d flag to see an objdump of the filter program's code.

Complete the tcpdump command in order to satisfy the following formatting requirements:

  • print the packet number
  • print the elapsed time (in nanoseconds) since the first packet was captured
  • print the content of each packet (without l2 header) in hex and ASCII
  • do not resolve IP addresses to names

How to test:

$ ntpdate -q ro.pool.ntp.org
$ curl ocw.cs.pub.ro

tpcdump can list the available interfaces if run with -D. In addition to your network interfaces, you may also see a Bluetooth device or the dbus-system (depending on your desktop).

If you don't specify the interface with -i, the first entry in the printed list will be used by default. This may not always be your active network interface but in stead, your docker bridge (for example).

[20p] Task B - iptables

iptables is a configuration tool for the kernel packet filter.

The system as a whole provides many functionalities that are grouped by tables: filter, nat, mangle, raw, security. If you want to alter a packet header, you place a rule in the mangle table. If you want to mask the private IP address of an internal host with the external IP address of the default gateway, you place a rule in the nat table. Depending on the table you choose, you will gain or lose access to some chains. If not specified, the default is the filter table.

Chains are basically lists of rules. The five built-in chains are PREROUTING, FORWARD, POSTROUTING, INPUT, OUTPUT. Each of these corresponds to certain locations in the network stack where packets trigger Netfilter hooks (here is the PREROUTING kernel hook as an example – not that hard to add one, right?) For a selected chain, the order in which the rules are evaluated is determined primarily by the priority of their tables and secondarily by the user's discretionary arrangement (i.e.: order in which rules are inserted).

Figure 1: Netfilter hooks; each has a subset of associated tables. Tables categorize the actions (i.e.: targets) taken when a match occurs. E.g: NAT cannot be performed on the FORWARD chain. When multiple rules exist on the same chain, their processing order is primarily determined by the priority of the table they are defined in.

A rule consists of two entities: a sequence of match criteria and a jump target.

The jump target represents an action to be taken. You are most likely familiar with the built-in actions such as ACCEPT or DROP. These actions decide the ultimate fate of the packet and are final (i.e.: rule iteration stops when these are invoked). However, there are also extended actions (see man iptables-extensions(8)) that are not terminal verdicts and can be used for various tasks such as auditing, forced checksum recalculation or removal of Explicit Congestion Notification (ECN) bits.

The match criteria of every rule are checked to determine if the jump target is applied. The way this is designed is very elegant: every type of feature (e.g.: Layer 3 IP address vs Layer 4 port) that you can check has a match callback function defined in the kernel. If you want, you can write your own such function in a Linux Kernel Module (LKM) and thus extend the functionality of iptables (Writing Netfilter Modules with code example). However, you will need to implement a userspace shared library counterpart. When you start an iptables process, it searches in /usr/lib/xtables/ and automatically loads certain shared libraries (note: this path can be overwritten or extended using the XTABLES_LIBDIR environment variable). Each library there must do three things:

  • define iptables flags for the new criteria that you want to include.
  • define help messages for when iptables --help is called (its help message is an amalgamation of each library's help snippet).
  • provide an initialization function for the structure containing the rule parameters; this structure will end up in the kernel's rule chain.

So when you want to test the efficiency of the iptables rule evaluation process, keep in mind that each rule may imply the invocation of multiple callbacks such as this.

The Task (1)

Write an iptables rule according to the following specifications:

  • chain: OUTPUT
  • match rule: TCP packets originating from ephemeral ports bound to a socket created by root
  • target: enable kernel logging of matched packets with the “EP: ” prefix

How to test:

$ sudo curl www.google.com
$ sudo dmesg

multiport, owner modules

$ man 8 iptables-extensions

The Task (2)

Write an iptables rule according to the following specifications:

  • chain: OUTPUT
  • match rule: BPF program that filters UDP traffic to port 53 (try bash command substitution)
  • target: set TTL to 1 (initially)

Continue appending the same rule with incremented TTL value until the DNS request goes through.

How to test:

$ dig +short fep.grid.pub.ro @8.8.8.8

bpf module

$ man 8 iptables-extensions nfbpf_compile

If you are working on Ubuntu, there is a chance that nfbpf_compile did not come with the iptables package (oh Canonical… maybe there's something in the Universe repos?).
Anyway, you can still install it manually:

$ sudo apt install libpcap-dev
$ wget https://raw.githubusercontent.com/netgroup-polito/iptables/master/utils/nfbpf_compile.c
$ gcc -o nfbpf_compile nfbpf_compile.c -lpcap

Also, use this man page rather than installing it separately.

The Task (3)

Give an example when iptables is unable to catch a packet.

02. [30p] Network Exploration

[10p] Task A - ARP vs ICMP

The Address Resolution Protocol (ARP) resolves layer 2 addresses (MAC) from layer 3 addresses (e.g.: IP). Normally, all hosts are compelled to reply to ARP requests, but this can be fiddled with using tools such as arptables. You can show the currently known neighbors using iproute2.

$ ip -c neigh show

Pro tip #2: yes, ip can also generate color output. Most people don't know this and still use ifconfig, even though it's already deprecated at this point. Add this as an alias to your .bashrc or .zshrc and source it.

# alias for iproute2 color output
alias ip='ip -c'

The Internet Control Message Protocol (ICMP) is an ancillary protocol meant mainly to report errors between hosts. Sometimes it can also be used to perform measurements (ping) or to inform network participants of better routes (Redirect Messages). There are many ICMP functionalities, most of which are now deprecated. Note that some network equipment may not be capable of understanding new and officially recognized protocols, while other may not even recognize experimental ICMP codepoints (i.e.: type=253,254) and simply drop the packet. Because ICMP can be used to stage attacks in a network, some operating systems (e.g.: Windows ≥7) went so far as to disable Echo Replies by default.

The Task(s)

Use arp-scan to scan your local network while monitoring ARP traffic with wireshark to get a sense of what's going on. After that, use the following script to identify hosts discoverable via ARP but not ICMP.

Click to display ⇲

Click to hide ⇱

Hint: click on the file name to download the snippet below.

localnet-ping.sh
#!/bin/bash
 
# localnet-ping.sh - performs differential ARP / ICMP scan
#   $1 : [required] interface name  
 
if [ "$#" -ne 1 ]; then
    echo "Usage: ./localnet-ping.sh <interface>"
    exit 1
fi
 
# generate list of IPs and hostnames in local network for given interface
localnet_hosts=$(sudo arp-scan                                      \
                    --interface=$1           `# scanned network`    \
                    --localnet               `# only local network` \
                | head -n -3                 `# hide footer lines`  \
                | tail -n +3                 `# hide header lines`  \
                | awk '{$2=""; print $0}'    `# hide MAC address`   \
                )
 
# process generated list, one item at a time
while read -r it; do
    # separate IP from hostname
    current_ip=$(awk '{print $1}' <<< $it)
    current_host=$(awk '{$1=""; print $0}' <<< $it)
 
    printf '\033[1;33m%15s   %-35s \033[0;33m==>  \033[0m' \
        $current_ip "$current_host"
 
    # ping current host
    ping -c 1           `# only one ping` \
         -W 1           `# 1s timeout`    \
         $current_ip    `# target host`   \
         1>/dev/null 2>&1
 
    # evaluate ping success
    if [ $? -eq 0 ]; then
        printf '\033[1;32mok\n\033[0m'
    else
        printf '\033[1;31mfail\n\033[0m'
    fi
done <<< "$localnet_hosts"

[20p] Task B - nmap vs traceroute

nmap is a network exploration tool and a port scanner. Today, we will look only at a specific functionality that it shares with the traceroute utility.

Route discovery is simple in principle: IPv4 packets have a Time to Live (TTL) field that is decremented by 1 with each hop, thus ensuring a limited packet lifespan (imagine routing loops without TTL). Even if the TTL is 0, the layer 3 network equipment must process the received packet (the destination host can accept a packet with TTL=0). Routers may check the TTL field only if they are to forward the packet. If the TTL is already 0, the packet is dropped and a ICMP Time-To-Live Exceeded message is issued to the source IP. By sending packets with incrementally larger TTL values, it is possible to obtain the IP of each router on the path (at least in theory).

The Task(s)

With 8.8.8.8 as a target, use wireshark to view the traffic generated by both nmap and traceroute. What differences can you find in their default mode of operation?

$ sudo nmap                            \
    -sn     `# disable port scan`      \
    -Pn     `# disable host discovery` \
    -tr     `# perform traceroute`     \
    8.8.8.8
$ traceroute 8.8.8.8

Troubleshooting:

  • permission denied : make sure that nmap is not installed as a snap; you have two choices:
    • reinstall nmap with apt : sudo snap remove nmap && sudo apt install nmap
    • grant nmap permissions : snap connect nmap:network-control

If we do allow for a port scan by removing -sn (default is a TCP-based scan; use -sU for a UDP scan), this will take place before the actual traceroute. What changes does this bring?

Optional Task (... no, really)

When doing the TCP scan with nmap, you may have noticed a weird field in the TCP header: Options. Generate some TCP traffic with curl and look at the SYN packet in wireshark. What options do you see there?

Here is a quick break down of the more common TCP options and how they are used to overcome protocol limitations and improve throughput. Take a quick look if you want, then move on. We'll dive deeper into protocol options in the next task.

03. [30p] Packets, where are you?

Earlier in Ex. 1, we mentioned that eBPF is used for more than traffic filtering. Some of you may have heard of the eXpress Data Path (XDP) or the more recent eXpress Resubmission Path (XRP). Both of these are eBPF-powered shunts of kernel data paths that are used to optimize the system for very specific types of workloads. We'll return to these in a future lecture (and maybe a lab as well) since they can be considered advanced topics. For now, we'll focus on the third purpose eBPF can serve: execution tracing.

pwru is a tool created by Cilium to help trace network packets in the kernel's network stack and debug network connectivity issues. It does this by attaching simple eBPF programs to certain function entry points. These programs can report back to a userspace process different kinds of information, including the function that was reached, the arguments that were passed, and a CPU clock timestamp. The method used for instrumenting kernel code is based on kprobes. Ask your assistant for more information.

[10p] Task A - A packet's journey

Install pwru on your system. Check that the minimum requirements stated on the Github page are met. Note that this tool is already provided by some public package repos (e.g.: pacman: extra/).

Now, trace all outgoing DNS queries to the Google DNS (i.e.: 8.8.8.8) and perform one using dig. Add relative timestamps to the individual trace entries, to get an idea of the computational cost of each operation.

Finally, insert an iptables rule on the OUTPUT chain that drops DNS queries to 8.8.8.8 and redo the experiment. Check where the packet's path is cut short (the reason should be obvious :p).

Be careful of local DNS caches, especially on Ubuntu.

[20p] Task B - Interpreting the call path

Analyze the call path in the kernel network stack for the first scenario (when the packet actually made it out). Explain each step of the packet's journey.

Check out this map of the kernel subsystems, but note that the best source of information is always RTFS.

04. [10p] Bonus - Protocol Options

Click to display ⇲

Click to hide ⇱

As you've probably already seen in the previous exercise, TCP uses protocol extensions (called options) in order to negotiate session parameters and improve overall performance. Note that this mechanism has existed since the very inception of the protocol approx. 30 years ago (RFC793 - Transmission Control Protocol), with many being added post-factum.

While the same could be said for IP options (RFC791 - Internet Protocol), there have always been… issues. In 2005 it was decided that IP Options are not an option. Middleboxes (i.e.: network equipment in the Internet – routers, NATs, firewalls, etc.) would sometimes implement abridged versions of the protocol specifications. For IP options, this meant that the IHL field would be ignored and the header would always be considered to be 20 bytes in length. As a result, if a packet carried IP options, cheap network equipment would wrongly assume that the layer 4 header would start at a 20 byte offset and drop it due to erroneously perceived malformations.

The authors of the 2005 report discovered that only a fraction (15%) of edge Autonomous Systems (AS – huge, ISP-grade networks with a unified routing policy) were responsible for most packet drops. This made them optimistic towards a speedy resolution of this issue, but things haven't changed much in the past 15 or so years. Today IP options usually work just fine in local networks. As for the wider Internet, there are specific paths and networks where IP options can pass unthwarted, but only if the layer 4 protocol is ICMP. The logic may be that most IP options are used for path measurement anyway, so why use it in conjunction with anything other than ICMP (a pretty dumb argument, I admit… but it's not mine).

Luckily, one of these compliant networks is RoEduNet. If you're not working from the university's network, then start a VM instance on OpenStack. ISPs like RDS tend to drop IP Options. As a target, we will use DigitalOcean. Based on some personal tests, I can guarantee that they don't drop any options, regardless of region.

Figure 2: Layout of a layer 3 (IP) header with options. Their presence is confirmed by an Internet Header Length (IHL) value strictly larger than 5. Most options are Type-Length-Value (TLV) encoded. When the length is constant or there is no information to be included (e.g.: No Operation, End of Options List), deviations from this format can be accepted for the purpose of saving space.

Overarching Goal

The first task is to modify outgoing traffic and include a Record Route option in an ICMP Echo Request. Check its description in RFC791 to understand what it does (and no, not “Loose/Strict Source and Record Route”… keep hitting that Find key). The packet structure will resemble that in the picture above. Don't worry; for this step, we'll give you a little help ;)

Your second task will be to write a bash script that extracts the IP addresses from the ICMP Echo Response's Record Route option from a packet capture and perform an AS Lookup. In other words, you will determine the names of all the networks that the packet traverses on its way from the university to DigialOcean and back.

But let's take it step-by-step.

Task A - Injecting IP Options

Remember talking about iptables extensions earlier? Netfilter Queue is one of them and will be relevant for this task. What it is, is an iptables target. What it does, it redirects each matched packet to a userspace process for evaluation and optionally, modification.

The userspace process receives each packet by polling a Unix Domain Socket. After obtaining one, it can perform any type of analysis that it wants (e.g.: deep packet inspection) in order to reach a verdict. The verdict can be the already known built-ins (i.e.: ACCEPT, DROP, etc.) or it can redirect the packet to another queue, with another process listening. When setting the verdict, a modified packet can be provided to replace the original on its datapath through the kernel's network stack.

Enter ops-inject. This is a tool (that's still under development, mind you) that allows the annotation of matched packets with IP/TCP/UDP options. Why is this tool simple to use: all you have to do is provide a sequence of bytes representing the codepoints of the options that you want to append. This byte stream is passed to an internal decoder that expands each byte into a fully-fledged option, albeit in accordance to an arbitrary implementation. Once you clone the repo, you should look over two sources in particular:

  • src/main.cpp : here, just understand what libnetfilterqueue library calls are made in order to set up the Unix socket, to receive the packet and to set its verdict.
  • src/ops_ip.c : this is where all available IP Options are implemented; take a look at the ip_decoders vtable at the bottom to associate options and codepoints.

First of all, let's fetch the tool and compile it.

$ git clone https://github.com/RaduMantu/ops-inject.git
$ cd !$:t:r
$ make -j $(nproc)

Pro tip #3: Bash Modifiers and Word Designators

  • !$ is substituted with the final argument of previously executed command
  • :t removes the leading pathname components, leaving ops-inject.git
  • :r removes exactly one trailing suffix, leaving ops-inject

Troubleshooting:

  • IPPROTO_ETHERNET error : if your netinet/in.h is missing this definition, you can delete line 36 from src/str_proto.c.
  • libnetfilter-queue missing : consider installing libnetfilter-queue-dev and libnetfilter-queue1.

Next, let's insert an iptables rule that matches all outgoing ICMP packets. Take note of --queue-num 0 for when we'll need to tell the userspace process which Netfilter Queue to subscribe. Also, --queue-bypass tells the iptables module to disable enqueuing packets if there's no process listening. Otherwise, the queue's buffer will fill up and overflowing packets will be dropped by default until some space is created.

$ sudo iptables -I OUTPUT -p icmp -j NFQUEUE --queue-num 0 --queue-bypass

Finally, run ops-inject while telling it to append the Record Route option (0x07). Because the tool takes a file as input, and because we give it the PseudoTerminal Slave of a subshell, things can get messy if we simply run it with sudo. As a result, it's easier to just switch to root. To get a feel of what the other options do, just run ops-inject --help once.

$ sudo su
# ./bin/ops-inject -p ip -q 0 -w <(printf '\x07')

Having completed the setup, let's generate some traffic!

$ ping -c 3 $(dig +short digitalocean.com | head -n 1)
    PING 104.16.182.15 (104.16.182.15) 56(84) bytes of data.
    64 bytes from 104.16.182.15: icmp_seq=1 ttl=57 time=46.7 ms
    RR:     141.85.13.15
            37.128.225.226
            37.128.232.178
            37.128.232.177
            80.97.248.33
            162.158.16.1
            104.16.182.15
            104.16.182.15
            162.158.16.1
 
    64 bytes from 104.16.182.15: icmp_seq=2 ttl=57 time=14.2 ms     (same route)
    64 bytes from 104.16.182.15: icmp_seq=3 ttl=57 time=18.2 ms     (same route)

Normally, we would need wireshark or tcpdump to see the result but fortunately, ping is able to understand the Record Route option. The reason for this is that it can generate it itself (see -R option). Should it have wondered that it received a Record Route option in response to a normal ICMP Echo Request? Apparently not…

If your ISP is blocking IP options, try ping-ing your default gateway.

Normally, that should work, but there is really no guarantee. A TP-Link router usually runs Linux 2.6 (at least) and does its job well. A Tenda router, however, most likely runs some garbage proprietary firmware and won't even reply to an ICMP Echo Request with IP options.

From this point onward, it's all you! :)

Task B - Traffic capture

Run the same experiment with ICMP Echo Request again, but this time capture the traffic using tcpdump and write it to a pcap capture file.

Consider using the -U option in tcpdump to avoid buffering packets if you plan to suddenly stop it with Ctrl^C.

If you had reachability problems at task A and IP options just can't get through, use this pcap starting with Task C. It contains pretty much what you were supposed to get.

As for Task B, show us that you can capture traffic correctly by targeting the default gateway again.

Task C - Route extraction

Use tshark to extract the Record Route payload of ICMP Echo Replies from the created pcap.
You only need the IPs of the intermediary hops; these will be further processed in your script at Task D.

  • Check out the -Y option in man tshark(1).
  • Look for the appropriate IPv4 Display Filter.
  • Test filter expressions in wireshark before applying them in tshark.

Task D - AS lookup

Write a bash script starting from your tshark command. The script must perform an AS lookup and display information about each registered hop (e.g.: IP, AS name, etc.), for all packets in the pcap. Run the script. What do you notice?

The output should look something like this ± a few info.

Figure 3: Packet sent from 104.16.181.15 to 172.19.7.205. In green are enumerated middleboxes (e.g.: routers) that added the IP of their outgoing interface to the buffer of the RR option. Not all may do so, depending on the implementation of their networking stack! In blue we have part of the publicly available information regarding the Autonomous System that said IP addresses belong to.

In order to get the required information, use the whois tool.

$ whois ${SOME_IP}
$ whois -h whois.cymru.com -- -v ${SOME_IP}

Want to make your script's output look pretty? Remember that you have ANSI color escape codes :)

Task E - AS lookup (part II)

As you might have noticed during the previous task, even DigitalOcean uses CloudFlare. This archive contains a pcap with ICMP Echo Request/Replies sent from the university to four VMs hosted on DigitalOcean. Run your script again, on this pcap and see if you can spot any interesting organization names. Then, look them up.

Not really relevant, but here are the IP addresses of the VMs involved:

New York           134.122.28.219
Frankfurt          46.101.222.105
Singapore          178.128.213.179
Toronto            165.22.239.70
UPB (localhost)    10.5.0.1

05. [10p] Feedback

Please take a minute to fill in the feedback form for this lab.

ep/labs/04.txt · Last modified: 2021/10/22 17:25 by radu.mantu
CC Attribution-Share Alike 3.0 Unported
www.chimeric.de Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0