Author: john

Categories
Admin Internet Mail Spam

Gmail: not as much a white-glove service as you thought

Intro
I have a pretty high regard for Google and their Gmail email service. They really seem to strive to keep its reputation sterling. But lately a persistent spam has been coming in to me from one of their users and no action is being taken, so I am beginning to wonder.

The details
It’s not that I don’t get spam from Gmail account holders. I do. That’s not shocking as I get hundreds of spam each day since my address is available from whois registrations on hundreds of domains, amongst other readily available sources.

How do I know it’s a real Gmail user and not someone spoofing the sender address? These two headers tell me:

Received-SPF: pass (google.com: domain of [email protected] designates 209.85.160.44 as permitted sender) client-ip=209.85.160.44;
Received: from mail-pb0-f44.google.com ([209.85.160.44]) (using TLSv1) by drjohnstechtalk.com, etc.

In other words one of the received headers provided by a trusted server gives me the IP of the sending server (209.85.160.44), which is in Google’s directly allocated IP space.
This can be confirmed at arin.net.

The email itself looks like this:

From: "Tom Zhu" <[email protected]>
 
Dear Sir/Madam,
 
We are owner of your_domain.mx and Presently we would like to know if you have an
interest in buying it. We are looking to sell this domain for 2,000 Euro.
It has been listed on Sedo.com. You can buy it through the following link:
 
https://sedo.com/search/details.php4?domain=your_domain.mx
...

But instead of your_domain the email mentions a specific copyrighted domain name.

I’ve received it over 10 times from the same sender. The sender is a cybersquatter sending repeated, unsolicited spam. If that doesn’t constitute a violation of their Terms of Use then I don’t know what does. I’ve filed no fewer than five formal abuse complaints to Google over the course of the several months. The Gmail abuse link is in the references. But they keep coming in so I know Google has taken no action whatsoever. And of course I have never heard back from them.

I’ve filed lots of other abuse complaints about other Gmail senders as well, but those others seem to be one-off spams and I don’t get additional emails from them. Yes it takes time to fill out the abuse form, but I do it for the overall good of the Internet. We technical people have a responsibility to make our world better…

Conclusion
I am personally miffed and professionally concerned that Google Gmail may not be operating as clean a shop (white glove) as we all had thought. Here I’ve documented a specific case of documented abuse about which they have done nothing for months on end.

References
Gmail abuse link.
ARIN’s IP lookup service is here.
A detailed look at how enom has handled some spam/domain complaints is written up here.
My successful fight to conquer scads of Chinese language spam is documented here.

Categories
DNS Network Technologies

The IT Detective Agency: internal DNS queries getting clobbered after bind upgrade

Intro
We’ve upgraded BIND innumerable times over the years. There’s never really been an issue. The new version just picks up and behaves exactly like the old version and all is good. But this time, in upgrading from ISC’s BIND v 9.8.5-P2 to BIND v 9.9.5-P1 something was dramatically different.

The details
Look at these queries:

> dig ns 10.in-addr.arpa

; <<>> DiG 9.9.2-P2 <<>> ns 10.in-addr.arpa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60248
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;10.in-addr.arpa.               IN      NS
 
;; ANSWER SECTION:
10.in-addr.arpa.        0       IN      NS      10.IN-ADDR.ARPA.
 
;; Query time: 0 msec
;; SERVER: blah, blah
;; WHEN: Wed Jun 25 09:49:30 2014
;; MSG SIZE  rcvd: 73

> dig -x 10.100.208.10

; <<>> DiG 9.9.2-P2 <<>> -x 10.100.208.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 6088
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;10.208.100.10.in-addr.arpa.   IN      PTR
 
;; AUTHORITY SECTION:
10.IN-ADDR.ARPA.        86400   IN      SOA     10.IN-ADDR.ARPA. . 0 28800 7200 604800 86400
 
;; Query time: 0 msec
;; SERVER: blah, blah
;; WHEN: Wed Jun 25 09:49:56 2014
;; MSG SIZE  rcvd: 106

That is seriously bad and wrong – for us! This is a cache-only server and there are indeed RFC-1918 addresses defined on internal nameservers, such as that 10.100.200.10.

An email relay which relies on reverse lookups started to fail.

A DuckDuckGo search did not show anything relevant. Maybe Google would have.

I ultimately registered for an account at the knowledge base at isc.org, kb.isc.org, and quickly found my answer.

In fact they were crystal clear in explaining this very problem, so I hesitated to document it here, but I figure others might leap first and then read the documentation later like myself so it might do someone else some good.

They say:

...
Although this will be effective as a workaround, administrators are urged not to just specify empty-zones-enable no;
 
It is much better to use one or more disable-empty-zone option declarations to disable only the RFC 1918 empty zones that are in use internally.  
...

That empty-zones-enable no; by the way is a configuration option you can toss in your main configuration file in the options section.

Case closed.

Conclusion
Our reverse lookups on the Intranet began to fail after an innocuous upgrade of the ISC bind nameserver to version 9.9. A simple addition of an extra configuration statement resolved the matter. I guess it really is a good idea to RTFM.

References
ISC’s site is www.isc.org.
A different type of DNS clobbering was described in this case.
A word about Google’s awesome public DNS service.

Categories
Internet Mail

Exchange Online Protection is currently broken. Resolved.

Intro
6/24th, 3:20 PM
A lot of my outbound emails are currently stuck with this status:

Deferred: 421 4.3.2 The maximum number of concurrent server connections has exceeded a limit, closing transmission channel

For instance, to belcorp.biz.

The MX record:

> dig +short mx belcorp.biz
10 belcorp-biz.mail.eo.outlook.com.

So it ends in outlook.com.

Same for emails to accenture.com. The MX record:

> dig +short mx accenture.com
10 accenture-com.mail.protection.outlook.com.

Also ends in outlook.com.

sheraton.com is another one I see.

So all these domains we can’t currently email to have an MX record ending in outlook.com and so I conclude are using Exchange Online Protection.

The situation has persisted for a couple hours so this doesn’t seem to be a 99.9999% uptime type of service.

Conclusion
Something went seriously wrong with Microsoft’s Exchange Online Protection service today.

6/25/14 update
Apparently this affected lots of Outlook users as well. It was finally fixed last night.

Categories
Admin Linux Network Technologies

The IT Detetecive Agency: the case of the unreliable FTP

Intro
So one of my power users complains that his FTPs to a particular site fail frequently, but not always. I rolled up my sleeves and set to work. The thing I do best is find the essence of a problem – what is the bare minimum sequence of events that reproduces it. I’m still getting my head around this one and I haven’t cracked the case yet, but I’ve learned about a few obscure packet generation tools.

The details
I may flesh this out later. The essence of the thing is that a packet trace (using tcpdump) shows that randomly no SYN-ACK packet is returned for our SYN packet to the FTP server on port 21. The FTP server resides on the Amazon cloud, but on the West Coast. We are on the East Coast. Not that that matters.

So I learned to reproduce the problem myself with the built-in ftp client. But I wanted even more control.

Packet generation tools
My trajectory went kind of like this:

ftp -> ping -> nmap -> hping3 -> mausezahn (-> scapy)

I had to compile mausezahn but I did manage to get it to work. I guess the developer has passed away. It doesn’t offer complete control over tcp packet generation, but nearly so.

I just discovered scapy. It appears to offer complete control over packet generation, including the tcp options such as mss, but the proof is in the pudding and I haven’t had time to check it out.

See the references fo links to further information about where to find these packages.

Preliminary findings
I began to see that with nmap and hping3 I was getting SYN-ACKs back consistently. What’s the difference between their SYN packets and ftp’s? They don’t use any options whereas my ftp client does.
And that is the essence of the problem. A tcp SYN packet which sets options like SACK, wscale and MSS is not being responded to around 30% of the time. No options set? SYN-ACKs come back 100% of the time. Pings are answered 99 – 100% of the time. mausezahn (mz) allows to set the window size. The window size is irrelevant.

Is it one particular tcp option that is the culprit, or just the fact of using any of them? Unfortunately that’s where you reach the limits of mz. mz only allows you to turn on or off all options. scapy promises to be more granular. So at least with mz by itself I can turn of/off the problem at will. That is getting to the essence of the problem.

Another wrinkle? Only certain source IPs have the problem! I have an identical system using a different ISP and it works all day long.

Conclusion
A lot of work and only modest progress to show for it. I need cooperation of the ftp administrator to do a simultaneous trace. Either the packet never gets to him, or his infrastructure discards it, or he responds but I never see the response. A two-sided trace will narrow down which of these three things is happening.

But I did learn that fine-control packet generation is a bit difficult to come by, which comes as a surprise in this day and age. You have to do some work to get full control over your packets.

I have nos stomach for writing my own C++ code to have total control.

It’s still an open case.

References
nmap.org talks about nmap. nmap is a pretty standard package available for major distributions. But it is not sufficiently configurable.
I’ve written about hping3 before, showing how to compile it.
I used this site for mausezahn source code and documentation.
scapy is well-documented here.

Categories
Internet Mail Spam

enom is the source of recent spam campaigns

Intro
I’m still watching over spam. The latest trend are spam campaigns which have a few characteristics in common perhaps the most interesting of which is that the domains have all been registered at enom.com.

The details
Some other things in common. These recent campaigns fell into two main categories. One set uses domains which are semi-pronounceable. The other are domains which incorporate sensible english words. Both categories have these other features in common.

– brevity (no HTML, for instance)
– valid SPF records (!)
– domains were used for spam almost immediately after having been registered (new domains)

Today’s example

From:        Patriot Survival Plan <[email protected]> 
To:        <[email protected]> 
Date:        05/22/2014 04:22 AM 
Subject:        REVEALED: The Coming Collapse 
 
 
 
--------------------------------------------------------------------------------
 
 
 
 
[email protected]
 
Since I exposed this I'm getting a lot of comments. 
 
People are terrified and they are asking me to spread the word even more...
 
So don't miss this because it might be too late for you and your family!
 
Obama's done a lot of stupid things so far, but this one will freeze the blood in your veins!
 
He's been trying hard to keep this from American Patriots... but now his betrayal has finally come to light.
 
And he'll have to pay through the nose for this.
 
But here's a Warning: the effects of Obama's actions will hit you and your family by the end of this year.
 
And they'll hit you like nothing you've ever seen before...
 
So watch this revealing video to know what to expect...
and how to protect against it.
 
-> Watch Blacklisted video now, before it's too late -->                 http://check.best-survival-plan-types.com
 
 
 
 
 
 
 
No_longer_receive_this _Warning :   http://exit.best-survival-plan-types.com
Patriot Survival Plan _405 W. Fairmont Dr. _Tempe, AZ 85282
 
 
 
 
 
First off, there's nothing special 22409526 in the Ironbound. Food in quantity, 22409526not quality. It's amazing how many people 22409526 rate these establishments as excellent. This said, I've always had fun going to these places, 22409526 as long as your dining expectations are gauged accordingly. Therefore, 22409526 my rating reflects those reduced expectations. :)
 
Being a steakhouse, 22409526 one would expect a thorough steak menu such as those at Gallagher's, Luger's, or even Del Frisco's. However, you're not getting true steakhouse fare here; 22409526 it's the Ironbound after all. So, you're getting a less than Prime cut of beef, 22409526 sometimes cooked to your liking.

Whois lookup of best-survival-plan-types.com shows this:

Domain Name: BEST-SURVIVAL-PLAN-TYPES.COM
Registry Domain ID: 1859701370_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.enom.com
Registrar URL: www.enom.com
Updated Date: 2014-05-21 17:26:19Z
Creation Date: 2014-05-22 00:26:00Z
Registrar Registration Expiration Date: 2015-05-22 00:26:00Z
Registrar: ENOM, INC.
Registrar IANA ID: 48
Registrar Abuse Contact Email: [email protected]
Registrar Abuse Contact Phone: +1.4252744500
Reseller: NAMECHEAP.COM
Domain Status: clientTransferProhibited
Registry Registrant ID:
Registrant Name: DONI FOSTER
Registrant Organization: NONE
Registrant Street: 841-4 SPARKLEBERRY LN
Registrant City: COLUMBIA
Registrant State/Province: SC
Registrant Postal Code: 29229
Registrant Country: US
Registrant Phone: +1.8037886966
Registrant Phone Ext:
Registrant Fax: +1.5555555555
Registrant Fax Ext:
Registrant Email: [email protected]
Registry Admin ID:
Admin Name: DONI FOSTER
Admin Organization: NONE
Admin Street: 841-4 SPARKLEBERRY LN
Admin City: COLUMBIA
Admin State/Province: SC
Admin Postal Code: 29229
Admin Country: US
Admin Phone: +1.8037886966
Admin Phone Ext:
Admin Fax: +1.5555555555
Admin Fax Ext:
Admin Email: [email protected]
Registry Tech ID:
Tech Name: DONI FOSTER
Tech Organization: NONE
Tech Street: 841-4 SPARKLEBERRY LN
Tech City: COLUMBIA
Tech State/Province: SC
Tech Postal Code: 29229
Tech Country: US
Tech Phone: +1.8037886966
Tech Phone Ext:
Tech Fax: +1.5555555555
Tech Fax Ext:
Tech Email: [email protected]
Name Server: DNS1.REGISTRAR-SERVERS.COM
Name Server: DNS2.REGISTRAR-SERVERS.COM
Name Server: DNS3.REGISTRAR-SERVERS.COM
Name Server: DNS4.REGISTRAR-SERVERS.COM
Name Server: DNS5.REGISTRAR-SERVERS.COM

See 1) that it was registered yesterday at 17:26:19 Universal Time, and 2) that the registrar is enom?

And the SPF record:

> dig +short txt best-survival-plan-types.com

"v=spf1 a mx ptr ~all"

Actually this domain is a small aberration insofar as it does not have a SPF record with a -all at the end – the others I checked do.

What to do, what to do
Well, I reported the spam to Postini, but I don’t think that has any effect as they are winding down their business.

I am pinning greater hopes on filling out enom’s abuse form. Of course I have no idea what actions, if any, they take. But they claim to take abuse seriously so I am willing to give them their chance to prove that.

enom’s culpability
I don’t feel enom is complicit in this spam. I’m not even sure they can easily stop these rogue operators. But they have to try. Their reputation is at stake. On the Internet there are complaints like this from years ago, that enom domains are spamming.

Every one that comes across my desk I am reporting to them. The time it takes for me to report any individual one isn’t worth the effort compared to the ease of hitting DELETE, but I am hoping to help lead enom to find a pattern in all these goings-on so they can stop these registrations before new ones cause harm – that is why I feel my actions are for the greater good.

Other recently deployed enom domains

Domain

First spam seen

First registered

onlinetncresults.us

8/22

8/21

checkdnconlinesystems.us

8/20

8/20

extremeconcretecoating.com

8/8

8/8

woodsurface.com

8/7

8/7

shorttermloanspecial.com

7/24

7/23

heartattackfighter1.com

6/19

3/2

handle-unsafe-parasites.me

6/10

6/9

best-survivalplan-learn.com

5/28

5/28

survival-plan-days.com

5/27

5/26

only-survival-plan.com

5/20

5/19

local-vehicle-clearance.us

5/19

5/19

ghiused.com

5/14

5/14

pastutmy.com

5/14

5/14

lekabamow.com

5/14

5/14

etc – there are plenty more!

Finally we hear back
Weeks later, on June 14th, I finally received a formal response concerning only-survival-plan.com and local-vehicle-clearance.us.

From: [email protected]
Subject: [~OOQ-128-23745]: FW: eNom - Report Abuse - Reference #ABUSE-11116
 
Hello, 
 
Thank you for your email. While the domain name(s) reported is registered with Namecheap, it is hosted with another company. So we cannot check the logs for the domain(s) and confirm if it is involved in sending unsolicited bulk emails. We can only take an action if a report is confirmed by blacklists of trusted anti-spam organizations like SpamHaus or SURBL.
 
Thus, we have initiated a case regarding the following domain(s) blacklisted by trusted anti-spam organizations:
only-survival-plan.com
In case the listing is not removed, the domain(s) will be suspended.
 
The following domain(s) has already been suspended:
local-vehicle-clearance.us
 
Let us also suggest you addressing the issue to the hosting company which servers were involved in email transmission for help with investigating the incident of spam. You may find their IP address in the headers. To find their contact details, please whois this IP address. You may use any public Whois tool like https://www.domaintools.com/ 
 
Kindly let us know if you have any question.
 
-------------------------------
Regards,
Alexander XXX.
Legal & Abuse Department
Namecheap Group
http://www.namecheapgroup.com

Analysis of their response
Reading between the lines, here’s my analysis. There’s some not-well-documented relationship between enom and namecheap.com. I reported the abuse to enom and got a response from namecheap.com. I kind of agree that suspending a domain is a BIG DEAL and a registrar has to be on firm footing to do so. As I write this one Jun 16th, the domains do not yet appear to be suspended. Are you really going to trust Spamhaus to render your judgement? That’s basically one of those extortionist enterprises purportedly offering a take-it-or-leave-it service. If the author of that email was a lawyer, well, their English isn’t the best. That doesn’t provide a lot of confidence in their handling of the matter. And wasn’t my complaint by itself good enough for them to initiate action? I do have to concede the point that the sending of the spam was probably out of their control and probably did come from another hosting company. But it is glib advice to suppose it is that easy to track them down the way they describe. Since they are part of the problem and have the evidence why don’t they follow up with the hosting provider themselves?? There was no mention of my other eight or so formal complaints. So this still seems to be getting an ad hoc one-by-one case treatment and not the, Whoa, we got a problem on our hands and there’s something systemically wrong with what we’re doing here reaction I had hoped to provoke.

Actually I got two responses but with slightly different wording. So they were crafted by hand from some boilerplate text, and yet the person stitching together the boilerplate was sufficiently mindless of the task as to forget they had already just sent me the first email??

So their response is better than a blackhole, but perhaps could be characterized as close to the bare minimum.

I have gotten several other responses from some of my other complaints as well, all saying pretty much the same thing. In August the responses started to look different however.

August responses
Here’s one I received this morning about woodsurface.com, 19 days after my initial complaint:

Hello,
 
This is to inform you that woodsurface.com domain was suspended. It is now pointed to non-resolving nameservers and will be nullrouted once the propagation is over. The domain is locked for modifications in our system.
 
Thank you for letting us know about the issue. 
 
------------------
Regards,
Alexander T.
Legal & Abuse Department
Namecheap.com

Conclusion
I hope my actions spur enom into some action of their own in figuring out where there domain registration requirements are too lax that spammers are taking wholesale advantage of the situation and sullying their reputation.

June, 2014 Update
The storm of spam from enom has subsided. I’m basically not seeing any. Oops. Spoke too soon! New enom-registered domains popped up and created more spam storms (documented in the table above), but not as severe as in the past. I don’t know if our anti-spam filter got better or enom stepped up to the plate and improved their scrutiny of domain registrants. If another spam storm hits us I’ll report back…

August, 2014
enom-generated spam is back!

References
My most popular spam-fighting article describes how to defeat Chinese-language spam.
A new type of spam that uses Google search results for link laundering is described here.

Categories
Uncategorized

Keyboard macro programming

Intro
I had the perfect motivation to learn a new tool. I had hundreds of repetitive keyboard clicks in my future. So my strong motivation was to save time.

The details
I needed to cancel about 300 changes in the Helpdesk system AHD (Advanced Helpdesk) from CA. First I turned to the team with the power tools and showed them what I needed. It looked readily doable. Move the mouse over here. Click. New Window pops up, in focus. Click on this drop-down menu item. Chance status to Cancel and Save. But for whatever reason they couldn’t do more than two, and then they lost their motivation.

Meanwhile my friend Nix told me about alternatives I could try on my own.

Workstation Macro Recorder from Automation Anywhere
First I tried this one. You only get a 30-day free trial. It was supposed to record my clicks and other keystrokes and then play them back. It didn’t work. Running it seemed to interfere with the browser window and the Javascript, which normally displays a menu during a mouse-over event was not behaving that way: no menu was popping up as long as this program was running.

AutoHotkey
So Nix suggested AutoHotkey. He warned it was a little buggy and cantankerous, but probably suitable for the task at hand.

There’s learning a language theoretically and learning a language to accomplish a very specific task. I’m not too good at the former, just doesn’t interest me and motivate me. But as I was saying I had great motivation for the latter. It wasn’t obvious at first how to proceed. But I started with a test script, which of course initially didn’t do anything at all. And then I slowly built up in complexity for the task at hand. It probably took several hours in total time spent over a few days.

The actual, working script, tested and used on my Windows 7 laptop, test.ahk

; launch by holding down Windows key, then pressing z key.
#z::
; loop five times
loop, 5 {
; Rt-arrow key
send {Right}
sleep, 100
; cancel out my AHD changes. u is for update
send u
sleep, 6000 ; 6 seconds
; now we need 3 tabs
loop, 3 {
send {tab}
sleep, 300
}
send Change is being cancelled.
;
loop, 6 {
sleep, 100
send {tab}
}
sleep 200
; c is for cancel
send c
; save it
send !s
sleep, 5000
; make next change the active one
send {tab}
sleep, 100
send {tab}
sleep, 100
}


Can you use AHD without mouse movement?
The problem I was solving is two-fold. AutoHotkey is not really great at mouse movement. So I needed to understand how to interact with AHD without using the mouse at all. It tuns out to be possible, at least mostly and sufficiently for my purposes. And I think it is important to mention because most users never get to this level: most programs do offer you a keyboard alternative to mouse movements and in fact it’s probably worth the time to find out what those are as it will almost always be faster than dragging that mouse pointer around to the precise locale of your next click.

I never did get the script to perfection, but that wasn’t the point. The point was to save my mind and my time. And it did work for that! There were always special conditions, etc which I did not code for and wasn’t motivated to code for. So I plowed through cancelling five changes at a time.

Conclusion
AutoHotkey proved to be quite helpful. Did I mention it is free? I recommend to use it to save on keystrokes for repetitive things.

References
Auto HotKey web site.

Categories
Admin Linux Network Technologies

Querying AD via LDAP – reference documentation

Intro
Suppose you managed to stuff the user’s ID into the description field of every computer object. Then independently the name of the computer object appears in a log such as a web server log and you want to know the user associated with that.

These examples show how to get at that description field from the computer object name.

The details

ldapsearch is a useful tool. I have two versions of it installed on Windows 7 and different Unix/Linux versions. The syntax is slightly different in all cases. Let’s assume the AD domain DRJOHNSAD is mapped to DNS domain drjohnsad.drjohns.net, and the user is drj. Then we have:

Linux
> ldapsearch -h drjohnsad.drjohns.net -b dc=drjohnsad,dc=drjohns,dc=net -D ‘drjohnsad\drj’ -W cn=computerName description

The -W switch prompts for the password. That is a nice switch, and not available in all versions of ldapsearch. If not, use -w password instead. drjohnsad\drj needs the single quote to prevent the “\” character from being treated as a special character by the shell. Windows doesn’t need that.

Windows 7 CMD Window

Oracle-provided ldapsearch

> ldapsearch -h drjohnsad.drjohns.net -b dc=drjohnsad,dc=drjohns,dc=net -D drjohnsad\drj -q cn=computerName description

So -q is used to prompt for a password instead of Linux’s -W.

Lotus Notes ldapsearch

> ldapsearch -h drjohnsad.drjohns.net -b dc=drjohnsad,dc=drjohns,dc=net -D drjohnsad\drj -w password cn=computerName description

You gotta put in the password on the command line.

Of course Windows also has applications which can be used for ldap queries in a GUI, but I don’t use them.

Conclusion
The syntax for a simple ldap query against an AD domain controller is shown.

Categories
Admin Web Site Technologies

Getting WebDav to work through Basic Authentication and HTTP

Intro
What I’m about to describe is not a recommended setup, but if like me you’re dealing with legacy infrastructure, well, sometimes you just gotta make things work as they are.

So imagine on your Intranet you have a WebDAV server running HTTP, not HTTPS, and using Basic Authentication. And you want to work with the files on a Windows 7 workstation. Read on to see how we got this combination to play nice together.

The details
First I gave myself access to a WebDAV resource on the server.

Being a Unixy type of guy, I then tried a Linux command-line program. I chose cadaver. See the webdav.org site for other clients.

I had to compile and install cadaver but that was no problem. Here is my test session:

> cadaver http://iwwwd.drj.com/webdav.ear/
Authentication required for CORP LDAP webDAV on server `iwwwd.drj.com’:
Username: drj
Password:
dav:/webdav.ear/> ls
Listing collection `/webdav.ear/’: succeeded.
Coll: drJ Application.ear 0 Sep 17 2012
Coll: drj app Application.ear 0 Nov 30 2013
…
dav:/webdav.ear/>

So, in other words, it worked! cadaver is kind of nice because it puts you into a shell and has commands similar to FTP, so it is a (mostly) familiar environment.

Now why was my Windows 7 PC giving me such a hard time?

Windows 7 by default only allows for Basic authentication against HTTPS webDAV servers.

To enable Basic authentication on the client computer, follow these steps:
1) Click Start , type regedit in the Start Search box, and then click regedit.exe in the Programs list.
2) Locate and then click the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters
3) Double-click the BasicAuthLevel registry key.
4) In the Value data box, type 2, and then click OK.
5) Exit Registry Editor, and then restart the computer.

More info can be found in this Microsoft knowledge base article: http://support.microsoft.com/kb/841215

http-webdav

Conclusion
Though not recommended for security reasons, it’s good to know there is a way to map a webdav drive when just HTTP (not HTTPS) is being used.

Categories
Network Technologies Proxy

The IT Detective Agency: The case of the purloined packet

Intro
Someone in the org notices that laptops can’t connect to our VPN concentrator when using Verizon MiFi. An investigation ensues and culprits are found. Read on for the details…

The details

Some background
The laptops are configured to use an explicit PAC file (proxy auto-config) for proxy access to the Internet. It works great when they’re on the Intranet. There were some initial bumps in the road, so we found the best results (as determined by using normal ISPs such as CenturyLink) for successful sign-on to VPN and establishing the tunnel was to create a dummy service on the Internet for the PAC file server. The main thing that does is send a RESET packet every time someone requests a PAC file. In that way the laptop knows that it should quickly give up on retrieving the PAC file and just use direct Internet connections for its initial connection to the VPN concentrator. OK?

So with that background, this current situation will make more sense. sometime after we developed that approach someone from the organization came along and tried to connect to VPN using a Verizon MiFi device. Although in the JunOS client it shows Connected in actual fact no tunnel is ever established so it does not work.

Time for to try some of our methodical testing to get to the bottom of this. Fortunately my friend has a Verizon MiFi jetpack, so I hooked myself up. Yup, same problem. It shows me connected but I haven’t been issued a tunnel IP (as ipconfig shows) and I am not on the internal network.

Next test – no PAC file
Then I turned off use of a PAC file and tried again. I connected and brought up a tunnel just fine!

Next test – PAC file restored, use of normal ISP
Then I restored the PAC file setting and tried my regular ISP, CenturyLink. I connected just fine!

Next test – try with a Verizon Hotspot
I tried my Verizon phone’s Hotspot. It also did not work!

I performed these tests several times to make sure there weren’t any flukes.

I also took traces of all these tests. To save myself time I won’t share the traces here like I normally do, but describe the relevant bits that I feel go into the heart of this case.

The heart of the matter
When the laptop is bringing up the VPN tunnel there is always at least one and usually several requests for the PAC file (assuming we’re in a mode where PAC file is in use). Remember I said above that the PAC file on the Internet is “served” by a dummy server that all it does is respond to every request (at TCP level every SYN) with a RST (reset)?

Well, that’s exactly what I saw on the laptop trace when using CenturyLink. But is not what I saw when using either Verizon MiFi or HotSpot. No, in the Verizon case the laptop is sent a (SYN,ACK). So the laptop in that case finishes the TCP handshake with an ACK and then makes a full HTTP request for the PAC file (which of course it never receives).

Well, that is just wrong because now the laptop thinks there’s a chance that it might get the PAC file if it just waits around long enough, or something like that.

The main point is that Verizon – probably with the best intentions – has messed with our TCP packets and the laptop’s behavior is so different as a result that it breaks this application.

I’m still thinking about what to do. I doubt Verizon, being a massive organization, is likely to change their ways, but we’ll see. UPDATE. After a few weeks and poor customer support from Verizon, they want to try some more tests. Verizon refuses to engage in a discussion at a technical level with us and ignores all our technical points in this open case. They don’t agree or disagree, but simply ignore it. Then they cherry pick some facts which support their preconceived notion of the conclusion. “It work with the PAC file turned off so the only change is on your end” is pretty much an exact quote from their “support.”

We also asked Verizon for all their ranges so we could respond differently to Verizon users, but they also ignored that request.

In particular What Verizon is doing amounts to WAN optimization.

UPDATE – a test with Sprint
I got a chance to try yet a different Telecom ISP: Sprint. I expected it to work because we don’t have any complaints, and indeed it did. I tested with a Sprint aircard someone lent me. But the surprising thing – indeed amazing thing – is how it worked. Sprint also clobbers those RST packets from when the laptop if fetching the PAC file. They also turn them into SYN,ACKs. But they carry the deceit one step further. When the laptop then does the GET request, they go so far as to fake the server response! They respond with a short 503 server fail status code! The laptop makes a few attempts to get the PAC file, always getting these 503 responses and then it gives up and it is happy to establish the VPN tunnelled connection at that point! So this has got me thinking…

April, 2014 update
Verizon actually did continue to work with us. They asked us the IP of our external PAC file server, never revealing what they were going to do with that information. Then we provided them phone numbers of Verizon devices we wished to test with. As an aside did you know that a Verizon MiFi Jetpack has a phone number? It does. Just log into it at my.jetpack and it will be displayed.

So I tried the Jetpack after they did their secret things, and, yes, VPN now works. But we are not helpless. I also traced that session, and in particular the packets to and from the PAC file webserver. Yes, they are coming back to my laptop as RSTs. I tested a second time, because once can be a fluke. Same thing: successful connection and RST packets as we wanted them to be.

Now we just need the fix to be generalized to all Verizon devices.

May 2014 update
Finally, finally, on May 6th we got the word that the fix has been rolled out. They are “bypassing” the IPs of our PAC file webservers, which means we get to see the RST packets. I tested an aircard and a Verizon hotspot and all looked good. VPN was established and RST packets were observed.

Case finally closed.

Conclusion
Verizon has done some “optimizations” which clobber our RST packets. It is probably pre-answering SYNs, with ACKs, which under ordinary circumstances probably helps performance. But it is fatal to creating an SSL VPN tunnel with the JunOS client configured with a PAC file. Sprint optimizes as well, but in such a way that things actually work. Verizon actually worked with us, at their own slow pace and secretive methodology, and changed how they handled those packets and things began to work correctly.

References
The next two links are closely related to this current issue.
The IT Detective Agency: How We Neutralized Nasty DNS Clobbering Before it Could Bite Us.
The IT Detective Agency: Browsing Stopped Working on Internet-Connected Enterprise Laptops

Categories
Admin Network Technologies Raspberry Pi Security

Bridging with the Raspberry Pi

Intro
Now I’m into playing with networking stuff on the Pi. So for a small investment I got a USB to ethernet adapter – $25 from Amazon. My first few experiments with it – turning it into a bridge – were largely successful.

The details
You need the bridge-utils package:

$ sudo apt-get install bridge-utils

For me it was easiest to connect to my Pi via the Wifi adapter I have on it (see this post describing how I used that to make a router). Then I could blow up the wired ethernet without disrupting my connection.

The first thing I noticed after plugging in the adapter is that it was automagically recognized and, for instance, the ifconfig -a command now shows an eth1 device. So no device driver needed to be installed, which was pretty sweet.

Listing the USB devices now looks like this:

$ lsusb

Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp.
Bus 001 Device 004: ID 0b95:1780 ASIX Electronics Corp. AX88178
Bus 001 Device 005: ID 7392:7811 Edimax Technology Co., Ltd EW-7811Un 802.11n Wireless Adapter [Realtek RTL8188CUS]

In short I want to take my two wired ports, eth0 and eth1, bridge them and send real traffic through them to see what happens. Bridging here means connecting two separate networks at a layer-2 level, so MAC addresses are proxied through the bridge and layer-3 communications occur transparently over the bridge as if it weren’t even there.

So…

$ sudo brctl addbr br0

creates a bridge named br0.

$ sudo brctl addif br0 eth0
$ sudo brctl addif br0 eth1

Adds those two physical ports to the bridge.

By the way I got all this from this reliable Linux Foundation source.

Now if I got it right, I should be able to unplug any wired device and put the Pi in between the switch and the device, no matter what its network is!

So I plugged my PC into one port, the other port into the switch the PC had been plugged into, and…

Well, for instance, showmacs output looks like this:

$ sudo brctl showmacs br0

port no mac addr                is local?       ageing timer
  1     00:00:00:0c:c8:1f       no                 1.30
  1     00:01:00:02:c8:1f       no               197.23
  2     00:02:cf:80:cc:99       no                 0.07
  2     00:90:a9:bb:3d:76       no                 0.94
  1     5e:00:00:fb:c8:1f       no               134.11
  1     5e:00:00:fd:c8:1f       no                57.49
  1     5e:7f:ff:fa:c8:1f       no                29.55
  2     64:66:b3:3b:bd:51       no                12.34
  1     8c:ae:4c:ff:27:69       yes                0.00
  2     b8:27:eb:dd:21:03       yes                0.00
  1     c8:1f:66:00:63:fe       no                 0.15

00:02:cf:80:cc:99 belongs to my upstream router, which is plugged into eth0, so I conclude that port 2 is eth0 and port 1 must be eth1. Maybe the port number was determined by the order in which I added the interface to the bridge?

Then I ran speedtest on my PC – there was no measurable slowdown in speed.

I loaded up the Pi with a cpu-intensive job:

$ yes > /dev/null

and re-ran speedtest. Still no hit to performance.

A brief review
I don’t think everyone will have appreciated what we’ve accomplished so let me review. We have created a passive, stealth man-in-the-middle (MITM) device with a management interface!

There are lots of uses for that, some not so noble. A more nefarious usage would be to connect up to a device and record all traffic passing back and forth to it (tcpdump -i br0). But there are also lots of good things we can achieve as well. For instance, Intrusion Detection System or Intrusion Prevention System (IDS/IPS). Firewall. Transparent proxy filter. IDS/IPS may be possible with Snort. I have to ask one of my security pals if that’s still the popular open source choice for IPS. Five years ago it was the package of choice.

What’s on my plate
Now I’d like to tinker with the Pi so that I can in fact slow down traffic and emulate slower connections! But I don’t know how yet…

I need to show how to make the bridging permanent using /etc/network/interfaces file.

To be continued…