Categories
Admin DNS Internet Mail

The IT Detective Agency: can’t get email from one sender

Intro
For this article to make any sense whatsoever you have to understand that I enforce SPF in my mail system, which I described in SPF – not all it’s cracked up to be.

The details
Well, some domain admins boldly eliminated their SOFTFAIL conditions – but didn’t quite manage to pull it off correctly! Today I ran into this example. A sender from the domain pclnet.net sent me email from IP 64.8.71.112 which I didn’t get – my SPF protection rejected it. The sender got an error:

550 IP Authorization check failed - psmtp

Let’s look at his SPF record with this DNS query:

$ dig txt pclnet.net

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.2 <<>> txt pclnet.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42145
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
 
;; QUESTION SECTION:
;pclnet.net.                    IN      TXT
 
;; ANSWER SECTION:
pclnet.net.             300     IN      TXT     "v=spf1 mx ip4:24.214.64.230 ip4:24.214.64.231 ip4:64.8.71.110 ipv4:64.8.71.111 ipv4:64.8.71.112 -all"

That IP, 64.8.71.112 is right there at the end. So what’s the deal?

Well, Google/Postini was called in for help. They apparently still have people who are on the ball because they noticed something funny about this SPF record, namely, that it isn’t correct. Notice that the first few IPs are prefixed with an ip4? Well the last IPs are prefixed with an ipv4! They are not both valid. In fact the ipv4 is not valid syntax and so those IPs are not considered by programs which evaluate SPF records, hence the rejection!

My recourse in this case was to remove SPF enforcement on an exception basis for this one domain.

Case closed!

Conclusion
It’s now a few months after my original post about SPF. I’m sticking with it and hope to increase its adoption more broadly. It has worked well, and the exceptions, such as today’s, have been few and far between. It’s a good tool in the fight against spam.

Categories
Admin DNS

The IT Detective Agency: is the guy rebooting our Linux server from France?

Intro
Sometimes you get a case that turns out to be silly and puts a smile on your face. This one is hardly worth the effort to write it up, but as I want to portray the life of an IT professional in all its glory and mundacity it’s worth sharing this anecdote.

The details
Yesterday I was migrating some SLES 11 VMs from an old VMWare server to a new platform. A shutdown was required, which I did. I noticed these particular systems had been barely used, hadn’t been booted in over a half year (and hence hadn’t been patched in that long). I guess my enterprise monitoring system actually works because not many minutes after the migration I get a call from the sysadmin, Fautt. He noticed the reboot, but he also noticed something else – a strange IP address associated with the reboot.

He says, over the phone, that the IP is 2.6.32.54 plus some other numbers. Of course when you’re listening over the phone it’s hard to both concentrate on the numbers and understand the concept at the same time. So he threw a giant red herring my way and said that that IP is blah-blah-.wanadoo.fr. Wanadoo.fr caught my ear. I recognize that as an ISP in France, with, like most ISPs, a somewhat dodgy reputation (in my opinion) for sending spam. So you see he led me down this path and I took the bait. I said I would look into it. The obvious underlying yet unspoken concern is this: if someone from France is rebooting my servers we have been completely compromised. So this was potentially very serious. Yet I knew I ahd been the one to reboot so I wasn’t actually panicked.

I ran the commands that I knew he must have run to see what he had seen. They look like this:

> last

reboot   system boot  2.6.32.54-0.3-de Thu Oct 11 14:03          (18:49)
drjohns  pts/0        sys1234.drjohns. Thu Oct 11 13:43 - down   (00:06)
root     pts/0        fauttsys.drjohns Wed Oct 10 11:16 - 11:32  (00:16)
...
root     :0           console          Wed Feb 15 15:13 - 15:55  (00:41)
root     :0                            Wed Feb 15 15:13 - 15:13  (00:00)
root     tty7         :0               Wed Feb 15 15:13 - 15:54  (00:41)
reboot   system boot  2.6.32.54-0.3-de Wed Feb 15 15:07         (33+19:43)
fauttdr  pts/1        fauttsys.drjohns Wed Feb 15 14:38 - down   (00:21)

Actually he told me he had just done a last reboot, but this presentation above makes the illusion more dramatic!

So the last entries contain the IP of the system where the admin logged in from, or domain names if the IP could be reverse looked up in DNS.

But, much like a magician who draws your attention to where he wants it to go, it is all a clever illusion. If you run

> dig -x 2.6.32.54

as Fautt surely must have, you get

; <<>> DiG 9.6-ESV-R5-P1 <<>> -x 2.6.32.54
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7990
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
 
;; QUESTION SECTION:
;54.32.6.2.in-addr.arpa.                IN      PTR
 
;; ANSWER SECTION:
54.32.6.2.in-addr.arpa. 42307   IN      PTR     ABordeaux-652-1-113-54.w2-6.abo.wanadoo.fr.
 
;; Query time: 3 msec
;; SERVER: 10.201.145.20#53(10.114.206.104)
;; WHEN: Fri Oct 12 09:02:21 2012
;; MSG SIZE  rcvd: 96

But I figured that if this was a hack, there might be some mention in Google concerning this notorious IP. I searched for 2.6.32.54. The first few matches talked about a linux kernel.

That was the duh moment! A quick

> uname -a

on the server to confirm:

Linux drjohns28 2.6.32.54-0.3-default #1 SMP 2012-01-27 17:38:56 +0100 x86_64 x86_64 x86_64 GNU/Linux

So clearly the reboot line in the last output was using a different convention than the other entries and was providing the kernel version!

I told Fautt and he was duly embarrassed but thought it was funny (there aren’t many good jokes in a sysadmin’s routine, apparently :)).

Case closed.

Conclusion
A good night’s rest is important. When you’re tired you’re much more prone to superficial thinking that allows you to be drawn into someone else’s completely wrong picture of events.