netstat – Dr John's Tech Talk

Intro
I love the Linux command line. I don’t know how to categorize the tips I have collected so I’m dumping them here for the time being.

The newer netstat
In the old days I used to constantly do

$ netstat -an|grep LISTEN|grep <PORT>

to see what tcp ports I had processes listening on. While that is very helpful, it doesn’t tell you the listening process. A better approach:

$ lsof -i tcp:<PORT>

where is 22, 80 or whatever you want to look for. lsof does show the process.

Repeat last argument
I am constantly doing things like

$ grep <string> <big-file> > /tmp/results

and then I want to have a look at that file but I don’t want to type in

$ more /tmp/results

so what do I do?

I type this to give me the last argument:

$ more !$

My fingers have memorized that pattern and so I do it without conscious thought, and you’re holding down the SHIFT key so it can be typed very quickly.

But sometimes I can’t wait to type even that. I want to collect the grep results into a file in case there were a lot of matches, but I also want to see the matching results on my screen as soon as they are available. What do I do? Use tee, as in:

$ grep <string> <big-filez> |tee /tmp/results|more

Cool, huh?

More Obscure stuff
A cool blog with lots of good, obscure tips, generally more technical than the ones above, is https://die-computerhilfe.de/blog

To be continued…

Conclusion

The IT Detective Agency: the case of the messages from mars

Intro
Today we got a “funny” message on our SLES 11 server in the /var/log/warn file. You might think that Martians have landed!

The Details
Specifically this:

Nov 9 10:54:19 drjohn24 kernel: [72397.088297] martian source 10.120.2.24 from 10.0.0.3, on dev eth1
Nov 9 10:54:19 drjohn24 kernel: [72397.088300] ll header: 78:e7:d1:7b:25:32:00:a0:8e:a8:8e:b3:08:00

Every time I pinged 10.120.2.24 (drjohn24) from 10.0.0.3 it would produce those two lines in the warn and messages file. More worrisome, I could not ssh from one host to the other. I could ssh from a host on the local network to drjohn24. We observed this behaviour even with the firewall disabled. Strange, right?

One more thing to note: drjohn24 has two network interfaces and various routes defined.

The Solution
It didn’t take too long to get to the bottom of this. We set up the routes wrong. We meant to create a default route out of eth0, which was right, and a net-10 route for eth1, which we specified incorrectly. Do

netstat -rn

to show all routes. I had this:

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.120.2.0      0.0.0.0         255.255.255.128 U         0 0          0 eth1
10.120.3.0      0.0.0.0         255.255.255.0   U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
10.0.0.0        10.120.2.1      255.255.255.128 UG        0 0          0 eth1
128.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
0.0.0.0         10.120.3.1      0.0.0.0         UG        0 0          0 eth0

Do you see the error? We put the mask on the 10.0.0.0 the same as we put on the interface and that’s not what we wanted.

The corrected version looks like this:

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
...
10.0.0.0        10.120.2.1      255.0.0.0       UG        0 0          0 eth1
...

Conclusion
So what was happening is that the inbound packet from 10.0.0.3 was arriving at eth0 as we intended. But SLES 11 is now clever enough to realize, based on its routing table, that that is not the expected interface where a packet with that source IP should arrive. It should have arrived at eth1 because of the default route. No other static route was more specific for 10.0.0.3 due to our error. And apparently even with firewall turned off, SLES gets very defensive at this point. I’m not sure if it was sending return packets out of eth1 or not, because I kept looking for them out of eth0!

Once we corrected the routes the inbound packet arrived at eth0 and was returned with an answer packet from eth0 and the martian messages went away.

The martian message thing is a little obscure, and at the time more a distraction than anything else as we had to research what that meant. I guess for the future we’ll instantly know. It’s very similar to defining network topology on your firewalls in an anti-spoofing defense.

Case closed!