Categories
Network Technologies Python

Tips on using scapy for custom IP packets

Intro
scapy is an IP packet customization tool that keeps coming up in my searches so I could no longer avoid it. I was unnecessarily intimidated because it was built around python and the documentation is a little strange. But I’m warming up to it now…

The details
Download and install
CentOS
Just go to scapy.net and it will propose to you to download the .zip file. I got scapy-2.3.1.zip. Then you can unzip it; change directory to the scapy-2.3.1 sub-directory and run

$ sudo python setup.py install

Debian systems such as Raspberry Pi
Simple. It’s just:

$ sudo apt-get install python-scapy

Usage modes
scapy can be called from within python, but if you’re afraid to do that like I am, you can run it from the command line which simply throws you into a python shell. I’m finding that a lot more comfortable as I slowly learn python syntax and some useful shortcuts.

Example 1
The background
Let’s cut to the chase and do something hard first. Remember how we got those Cisco Jabber packets with DSCP set, causing Cisco Jabber to not work for some users? The long-term solution according to that post is to turn off the DSCP flag for all packets on the Internet router. So we want to be able to generate packets under our control with that flag set so we can see if we’ve managed to turn it off correctly.

DSCP value occupies the first 6 bits of the 8-bit tos field. The packets we got from Cisco had DSCP of 0x2e which is Expedited Forwarding (EF), and if you do the math that corresponds to tos of 0xb8 which in decimal is 184.

$ sudo scapy
>>> sr(IP(dst="50.17.188.196",tos=184)/TCP(dport=80,sport=4025))

Begin emission:
....Finished to send 1 packets.
.*
Received 6 packets, got 1 answers, remaining 0 packets
(<Results: TCP:1 UDP:0 ICMP:0 Other:0>, <Unanswered: TCP:0 UDP:0 ICMP:0 Other:0>)
>>>

Instead of the call to sr you can simply use send. Breaking this down, I’m testing against my drjohns server with IP 50.17.188.196. tos is a property of an IP packet so it’s included as a keyword argument to the IP function. The “/” following the IP function is funny syntax but it somehow says that more properties at different layers are coming. So in the TCP section I used keyword arguments and set source port of 4025 and destination port of 80. What I observed is that this will send a SYN packet even though I didn’t explicitly identify that.

Want to have a random source port like “real” packets? Then use this:

$ >>> sr(IP(dst="50.17.188.196",tos=184)/TCP(dport=80,sport=RandShort()))

Look for it
I know tcpdump better so I look for my packet with that tool like this:

$ sudo tcpdump -v -n -i eth0 host 71.2.39.115 and port 80

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
19:33:29.749170 IP (tos 0xb8, ttl 39, id 1, offset 0, flags [none], proto TCP (6), length 44)
    71.2.39.115.partimage > 10.185.21.116.http: Flags [S], cksum 0xd97b (correct), seq 0, win 8192, options [mss 1460], length 0
19:33:29.749217 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 44)
    10.185.21.116.http > 71.2.39.115.partimage: Flags [S.], cksum 0x3e39 (correct), seq 3026513916, ack 1, win 5840, options [mss 1460], length 0
19:33:29.781193 IP (tos 0x0, ttl 41, id 19578, offset 0, flags [DF], proto TCP (6), length 40)

Interpretation
Our tos was wiped clean by the time our generated packet was received by Amazon AWS. This was a packet I sent from my home using my Raspberry Pi. So likely my ISP CenturyLink is removing QOS from packets its residential customers send out. With some ISPs and business class service I have seen the tos field preserved exactly. When sent from Amazon AWS I saw the field value altered, but not set to 0!

Example 2, ping
>>> sr(IP(dst="8.8.8.8")/ICMP())

Begin emission:
Finished to send 1 packets.
.*
Received 2 packets, got 1 answers, remaining 0 packets
(<Results: TCP:0 UDP:0 ICMP:1 Other:0>, <Unanswered: TCP:0 UDP:0 ICMP:0 Other:0>)

Getting info on return packet
$ >>> sr1(IP(dst="drjohnstechtalk.com",tos=184)/TCP(dport=80,sport=RandShort()))

Begin emission:
...............................................................................................Finished to send 1 packets.
...........................................*
Received 139 packets, got 1 answers, remaining 0 packets
<IP  version=4L ihl=5L tos=0x0 len=44 id=0 flags=DF frag=0L ttl=25 proto=tcp chksum=0xe1d7 
src=50.17.188.196 dst=144.29.1.2 options=[] |<TCP  sport=http dport=17176 seq=3590570804 ack=1 dataofs=6L reserved=0L flags=SA window=5840 chksum=0x24b0 urgptr=0 options=[('MSS', 1460)] |<Padding  load='\x00\x00' |>>>

Note that this tells me about the return packet, which is a SYN ACK. So it tells me my SYN packet must have been sent from port 17176 (it changes every time because I’ve included sport=RandShort()). Each “.” in the response indicates a packet hitting the interface. I guess it’s promiscuously listening on the interface.

Hitting a closed port

$ >>> sr1(IP(dst="drjohnstechtalk.com",tos=184)/TCP(dport=81,sport=RandShort()))

Begin emission:
....................................................................................
.........................................................................................................Finished to send 1 packets.
...............................................................................
.................................................................................
.........................................................................................

Basically those dots are going to keep going forever until you type -C, because there will be no return packet if something like a firewall is dropping your packet, or the returned packet.

Useful shortcuts
The scapy commands look pretty daunting at first, right? And too much trouble to type in, right? Just get it right once and you’re set. In typical networking debugging you’ll be running such test packets multiple times. Because it’s basically a python shell, you can use the up arrow key to recall the previous thing, or hit it multiple times to scroll through your previously typed commands. And even if you exit and return, it still remembers your command history so you can hit the up-arrow to get back to your commands from previous sesisons and previous days.

References and related
This scapy for dummies guide is very well written.
I’m finding this python tutorial really helpful.
DSCP and explanation of Cisco Jabber not working is described here.
A simpler tool which is fine for most things is nmap. I provide some real-world examples in this blog post.

Categories
Admin Linux Network Technologies

The IT Detetecive Agency: the case of the unreliable FTP

Intro
So one of my power users complains that his FTPs to a particular site fail frequently, but not always. I rolled up my sleeves and set to work. The thing I do best is find the essence of a problem – what is the bare minimum sequence of events that reproduces it. I’m still getting my head around this one and I haven’t cracked the case yet, but I’ve learned about a few obscure packet generation tools.

The details
I may flesh this out later. The essence of the thing is that a packet trace (using tcpdump) shows that randomly no SYN-ACK packet is returned for our SYN packet to the FTP server on port 21. The FTP server resides on the Amazon cloud, but on the West Coast. We are on the East Coast. Not that that matters.

So I learned to reproduce the problem myself with the built-in ftp client. But I wanted even more control.

Packet generation tools
My trajectory went kind of like this:

ftp -> ping -> nmap -> hping3 -> mausezahn (-> scapy)

I had to compile mausezahn but I did manage to get it to work. I guess the developer has passed away. It doesn’t offer complete control over tcp packet generation, but nearly so.

I just discovered scapy. It appears to offer complete control over packet generation, including the tcp options such as mss, but the proof is in the pudding and I haven’t had time to check it out.

See the references fo links to further information about where to find these packages.

Preliminary findings
I began to see that with nmap and hping3 I was getting SYN-ACKs back consistently. What’s the difference between their SYN packets and ftp’s? They don’t use any options whereas my ftp client does.
And that is the essence of the problem. A tcp SYN packet which sets options like SACK, wscale and MSS is not being responded to around 30% of the time. No options set? SYN-ACKs come back 100% of the time. Pings are answered 99 – 100% of the time. mausezahn (mz) allows to set the window size. The window size is irrelevant.

Is it one particular tcp option that is the culprit, or just the fact of using any of them? Unfortunately that’s where you reach the limits of mz. mz only allows you to turn on or off all options. scapy promises to be more granular. So at least with mz by itself I can turn of/off the problem at will. That is getting to the essence of the problem.

Another wrinkle? Only certain source IPs have the problem! I have an identical system using a different ISP and it works all day long.

Conclusion
A lot of work and only modest progress to show for it. I need cooperation of the ftp administrator to do a simultaneous trace. Either the packet never gets to him, or his infrastructure discards it, or he responds but I never see the response. A two-sided trace will narrow down which of these three things is happening.

But I did learn that fine-control packet generation is a bit difficult to come by, which comes as a surprise in this day and age. You have to do some work to get full control over your packets.

I have nos stomach for writing my own C++ code to have total control.

It’s still an open case.

References
nmap.org talks about nmap. nmap is a pretty standard package available for major distributions. But it is not sufficiently configurable.
I’ve written about hping3 before, showing how to compile it.
I used this site for mausezahn source code and documentation.
scapy is well-documented here.

Categories
Admin

Nmap: Swiss Army Knife of network utilities

Intro
I just wanted to put in a plug for nmap. It’s a very useful tool for any network specialist. I show a use case that came up today.

The details
While cleaning up DNS entries I came across a network segment that didn’t seem to have any active network devices, at least not after I cleaned up the old DNS entries for inactive devices.

So I wanted to see if I could tell the networking tech that this subnet is unused and could be allocated for some other purpose.

I remembered using nmap years ago, and that it was a powerful tool for this kind of thing. What I had in mind was to ping every IP on this segment to see if there were any undocumented hosts.

As it turns out I didn’t even have it installed, but it was very easy to get:

On SLES:
$ zypper install nmap

On CentOS:
$ yum install nmap

It doesn’t get easier than that!

A quick review of the man page showed that what I wanted was indeed possible. Here’s the syntax for a systematic PING sweep through a subnet:

$ nmap −sP 10.101.192.0/24

Starting Nmap 4.75 ( http://nmap.org ) at 2012-11-08 10:21 EST
Host 10.101.192.5 appears to be up.
Host 10.101.192.10 appears to be up.
Host 10.101.192.151 appears to be up.
Host 10.101.192.152 appears to be up.
Host 10.101.192.153 appears to be up.
Nmap done: 256 IP addresses (5 hosts up) scanned in 1.28 seconds

Now I know that subnet has rogue or at least undocumented hosts and is not unused!

The original usage for nmap, at least for me, was to fingerprint an unknown host:

$ nmap −A −T4 ossim.drj.com

Interesting ports on ossim.drj.com (10.22.235.19):
Not shown: 996 filtered ports
PORT    STATE  SERVICE  VERSION
22/tcp  open   ssh       (protocol 2.0)
80/tcp  open   http     Apache httpd
|_ HTML title: 302 Found
443/tcp open   ssl/http Apache httpd
|_ HTML title: Site doesn't have a title.
514/tcp closed shell
1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at http://www.insecure.org/cgi-bin/servicefp-submit.cgi :
SF-Port22-TCP:V=4.75%I=7%D=4/25%Time=51793070%P=x86_64-suse-linux-gnu%r(NU
SF:LL,29,"SSH-2\.0-OpenSSH_5\.5p1\x20Debian-6\+squeeze2\r\n");
Device type: WAP|general purpose|PBX
Running (JUST GUESSING) : Linux 2.6.X|2.4.X (93%), Vodavi embedded (85%)
Aggressive OS guesses: OpenWrt 7.09 (Linux 2.6.22) (93%), OpenWrt 0.9 - 7.09 (Linux 2.4.30 - 2.4.34) (92%), Linux 2.6.20.6 (89%), Linux 2.6.21 (Slackware 12.0) (88%), OpenWrt 7.09 (Linux 2.6.17 - 2.6.21) (88%), Linux 2.6.19 - 2.6.21 (88%), Linux 2.6.22 (Fedora 7) (88%), Vodavi XTS-IP PBX (85%), Linux 2.6.22 (85%)
No exact OS matches for host (test conditions non-ideal).
 
TRACEROUTE (using port 21/tcp)
HOP RTT    ADDRESS
1   0.87   10.202...
2   0.38   ...
3   0.57   ...
4   6.10   ...
5   114.64 ...
6   119.79 ...
7   103.43 ossim.drj.com (10.22.235.19)
 
OS and Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 35.02 seconds

Now that was kind of an unusual example in which nmap wasn’t too sure about the OS. Usually you get a positive ID of some sort. That’s a chatty server and I’m still not sure what it is.

Nmap can be used for nasty things and in an impolite way, network-wise. So be careful to tone it down. Target your hosts and protocols with care. It can guess what OS a host is running, what ports are open, all kinds of amazing stuff.

I checked PING and did not see a built-in capability to do a PING sweep, though it would have been easy enough to script it. That was my backup option.

Once I had to check on a single UDP port being open on port 80 for a webcast client called Kontiki (they call this protocol KDP). No other ports were open, necessitating the -PN switch.

Single UDP port check
$ nmap −PN −sU -p 80 29.239.11.4

Starting Nmap 4.75 ( http://nmap.org ) at 2013-07-23 13:59 EDT
Interesting ports on 29.239.11.4:
PORT   STATE         SERVICE
80/udp open|filtered http
 
Nmap done: 1 IP address (1 host up) scanned in 2.15 seconds

Three TCP ports checked
$ nmap ‐PN ‐sS ‐p 445,28080,28443 12.92.96.37

Results of that scan

Starting Nmap 5.51 ( http://nmap.org ) at 2017-04-13 09:18 EDT
Nmap scan report for 12.92.96.37
Host is up.
PORT      STATE    SERVICE
445/tcp   filtered microsoft-ds
28080/tcp filtered unknown
28443/tcp filtered unknown
 
Nmap done: 1 IP address (1 host up) scanned in 3.06 seconds

filtered” means there were no reply packets to my SYN packets, usually a sign of an intervening firewall dropping packets. I’m not sure why it describes the host as “up” when actually it is down or behind a firewall. A state of closed indicates that a RST packet was received in reply, indicating that the port is closed on the host itself and it wasn’t a firewall that prevented the test from succeeding. the third possible state is open, which of couse means that it replied with a SYN-ACK to that probe on that port.

To fix the source port add a -g to the above command. E.g., some firewalls have trouble with permitting inbound UDP packets from port 53 so to test for that you throw in a -g 53 and try some random high destination port.

I needed to spoof another host’s IP address and send a simple PING (ICMP request) to diagnose what was going wrong with the reply. Here’s how I did that:

$ nmap −PE −e eth0 −S 10.42.48.1 10.1.145.10

But then I realized what I really needed to do to emulate the problem is to send a single TCP SYN packet to port 8081, without the accompanying ICMP probes that nmap is wont to throw in there first. Here’s how I built up that probe:

$ nmap −PN −sS −p 8081 −−max-retries 0 −e eth0 −S 10.42.48.1 10.1.145.10

Check if a web server is running
$ nmap −PN −p T:80,443 drjohnstechtalk.com
This will check both ports 80 and 443. It doesn’t execute any HTTP protocol. It’s just a quick and dirty test.

don’t have nmap but have something like netcat instead? A good tcp port check with netcat is
netcat -vzw5 <host> <port>. Here’s an actual example.

$ netcat ‐vzw5 drjohnstechtalk.com 443

DNS mismatch
drjohnstechtalk.com [50.17.188.196] 443 (https) open

Conclusion
Nmap is a great network tool that every IT network tech should be familiar with.

References and related
A more capable and complicated packet generation tool is scapy. I describe it in this blog post.

A simpler network for Windows (simpler than nmap for Windows) is PortQry. It was created by Microsoft.