Intro
Sometimes you get a case that turns out to be silly and puts a smile on your face. This one is hardly worth the effort to write it up, but as I want to portray the life of an IT professional in all its glory and mundacity it’s worth sharing this anecdote.
The details
Yesterday I was migrating some SLES 11 VMs from an old VMWare server to a new platform. A shutdown was required, which I did. I noticed these particular systems had been barely used, hadn’t been booted in over a half year (and hence hadn’t been patched in that long). I guess my enterprise monitoring system actually works because not many minutes after the migration I get a call from the sysadmin, Fautt. He noticed the reboot, but he also noticed something else – a strange IP address associated with the reboot.
He says, over the phone, that the IP is 2.6.32.54 plus some other numbers. Of course when you’re listening over the phone it’s hard to both concentrate on the numbers and understand the concept at the same time. So he threw a giant red herring my way and said that that IP is blah-blah-.wanadoo.fr. Wanadoo.fr caught my ear. I recognize that as an ISP in France, with, like most ISPs, a somewhat dodgy reputation (in my opinion) for sending spam. So you see he led me down this path and I took the bait. I said I would look into it. The obvious underlying yet unspoken concern is this: if someone from France is rebooting my servers we have been completely compromised. So this was potentially very serious. Yet I knew I ahd been the one to reboot so I wasn’t actually panicked.
I ran the commands that I knew he must have run to see what he had seen. They look like this:
> last
reboot system boot 2.6.32.54-0.3-de Thu Oct 11 14:03 (18:49) drjohns pts/0 sys1234.drjohns. Thu Oct 11 13:43 - down (00:06) root pts/0 fauttsys.drjohns Wed Oct 10 11:16 - 11:32 (00:16) ... root :0 console Wed Feb 15 15:13 - 15:55 (00:41) root :0 Wed Feb 15 15:13 - 15:13 (00:00) root tty7 :0 Wed Feb 15 15:13 - 15:54 (00:41) reboot system boot 2.6.32.54-0.3-de Wed Feb 15 15:07 (33+19:43) fauttdr pts/1 fauttsys.drjohns Wed Feb 15 14:38 - down (00:21) |
Actually he told me he had just done a last reboot, but this presentation above makes the illusion more dramatic!
So the last entries contain the IP of the system where the admin logged in from, or domain names if the IP could be reverse looked up in DNS.
But, much like a magician who draws your attention to where he wants it to go, it is all a clever illusion. If you run
> dig -x 2.6.32.54
as Fautt surely must have, you get
; <<>> DiG 9.6-ESV-R5-P1 <<>> -x 2.6.32.54 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7990 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;54.32.6.2.in-addr.arpa. IN PTR ;; ANSWER SECTION: 54.32.6.2.in-addr.arpa. 42307 IN PTR ABordeaux-652-1-113-54.w2-6.abo.wanadoo.fr. ;; Query time: 3 msec ;; SERVER: 10.201.145.20#53(10.114.206.104) ;; WHEN: Fri Oct 12 09:02:21 2012 ;; MSG SIZE rcvd: 96 |
But I figured that if this was a hack, there might be some mention in Google concerning this notorious IP. I searched for 2.6.32.54. The first few matches talked about a linux kernel.
That was the duh moment! A quick
> uname -a
on the server to confirm:
Linux drjohns28 2.6.32.54-0.3-default #1 SMP 2012-01-27 17:38:56 +0100 x86_64 x86_64 x86_64 GNU/Linux |
So clearly the reboot line in the last output was using a different convention than the other entries and was providing the kernel version!
I told Fautt and he was duly embarrassed but thought it was funny (there aren’t many good jokes in a sysadmin’s routine, apparently :)).
Case closed.
Conclusion
A good night’s rest is important. When you’re tired you’re much more prone to superficial thinking that allows you to be drawn into someone else’s completely wrong picture of events.