Categories
Exchange Online Internet Mail

Web forms: creating today’s Open Relays

Intro
Maybe it’s just me, but I’ve been receiving a lot of non-delivery reports in the past couple weeks. I happen to receive email addressed to webmaster@<large company>.com and as well I am knowledgeable about how email works. This puts me in the rare position of knowing how to interpret the meaning of the many NDRs I started to see.

The details
This picture shows the actual state of my inbox if I look for NDRs by searching for the word undeliverable:

NDRs-2016-09-16_14-22-17

Included is the preview panel showing the contents of one of those messages from Wells Fargo.

Actions
Personally, I can live with this level of clutter in my inbox – I actually get hundreds of messages a day like many IT folks. But I put this into a larger context. As a responsible Netizen (not sure if anyone really uses that term, but it was a useful one!) I feel a responsibility to keep the Internet running in an orderly way and stop the abusers as quickly as possible. After all I owe my job to a well-functioning Internet. So I decided rather than to do the easy thing – write rules to shunt these aside, or let clutter go to work, or junk email them, I was going to try to go after the source. Actually sources because you may have noticed that there are multiple domains involved.

First analysis
I thought about how these emails could have originated. I thought, OK, someone spoofs the email address webmaster@<large company>.com, they find a reputable mail server somewhere on the Internet; this mail server has to be an open relay. So the first few mail servers I checked out on Mxtoolbox didn’t show any problems or complaints whatsoever – sterling reputations. Turns out that way of thinking is soooo 20th century. Excusable in my case because I was already running sendmail in the 20th century and in those days that was some of the biggest worries – running an open relay.

Upon further reflection
Someone mentioned it’s probably a web form and I thought, Yes. Did you ever see those forms, like, Share this web page, that allow you to enter your email address and send something to a friend – which they will receive as apparently coming from you? What if in addition you could add your own custom message? Well, that’s the modern way of turning a mail server into an open relay.

The fatal recipe
A web form with the following properties is an open relay enabler:

– permits setting sender address
– permits setting recipient address
– has a comment field which will get sent along to the recipient
– does not employ captcha technology

Only visible from its side effects
The thing is, I am only receiving the NDRs, in other words those emails which had a spoofed sender address (my webmaster address) and a recipient address which for whatever reason decided not to accept the message. The “open relay,” failing to send the message, sends the failure notice to the spoofed sender: my webmaster account. But this NDR does not contain the original message, just vestigial hints about the original message, such as what server it was sent from, who was the recipient and sender, what the subject line was (sometimes), and when the receiving system complained. If a spam email was successfully sent in my name I never get to see it!

But I was able to actually find one of these dangerous forms that is being abused. Here it is:

open-relay-web-form-2016-09-16_14-41-28

Of course some of these forms are more restrictive than others. And almost all share the characteristic that they always put certain words into the subject and perhaps the body as well, which is beyond the control of the abuser. But that free-form message field is gold for the abuser and allows them to put their spammy or malicious message into the body of the email.

So after checking out a few of these domains for open relay and coming up empty, I do think all the abuse was done through too-lenient web pages. So I guess that is the current method of creating a de facto open relay.

Results
I’ve written very well-informed emails, initially trying to send to abuse@<domain>.com. But I also got more creative, in some cases tracing the domain to an ISP and looking up how to contact that ISP’s abuse department. I’m kind of disappointed that Wells Fargo hasn’t responded. Many other ISPs did. I believe that some have already corrected the problem. Meanwhile new ones crop up.

Over the weeks I’ve worked – successfully – with several offenders. Each one represented unique aspects and I had to do some IT detective work to track down someone who wuold be likely to respond. The ones who haven’t cooperated at all are overseas. Here is the wall of shame:

ecritel.fr (handles emails for marieclaire.fr web site)
ifastfinancial.com
RTHK.com.hk

Update
After about a week Wells Fargo did give a brief reply. They did not ask for any details. But the spam from their server did stop as far as I can tell.

If it turns into a never-ending battle I will give up, except perhaps to spend a few minutes a day.

Permanent fix
I don’t know the best way to fix this. I used to be a fan of SPF, but its limitations make it impossible for some large companies who need to have too many third parties sending email on their behalf. I guess Google is pushing DMARC, but I haven’t had time to think through if it’s feasible for large enterprises.

Conclusion
Poorly constructed web forms are the new open relay enablers. Be very careful before creating such a form and at a minimum use good captcha technology so its usage can not be automated.

This is speculation but I would not be surprised to learn that there is a marketplace for a listing of all the poorly constructed web forms out there – that information could be very valuable to spammers who have been increasingly shut out of our inboxes by improved anti-spam detection.

References and related
I found this site helpful in finding valid contacts for a domain: abuse.net. You enter the domain and spits back a couple of valid abuse contact addresses.
I only reluctantly use Mxtoolbox. It’s like a necessary evil. So i don’t want to give out a link for it. Probably nearly as good to check out a mail server’s reputation is senderbase.org. They’re not trying to sell you anything.
DMARC – perhaps the email authentication mechanism of the future.
My old post advocating SPF, which just never caught on…
PHPMailer remote code execution explanation, which takes advantages of web forms used to send email.

Categories
Admin Exchange Online Internet Mail

PowerShell and Proxy server

Intro
I’ve used Windows PowerShell for all of a few hours so far. But, still, I think I have something to contribute to the community. The documentation on how to send commands through a standard http proxy is pretty miserable so I’d like to make that more clear. I plan to use PowerShell to administer Exchange online.

The details
Microsoft has some pretty good documentation on PowerShell in general. in particular for my desire to connect to Exchange Online I found this very helpful article. But that article says not a whit about sending your connection through an explicit proxy, which I found bewildering.

But I found some key documentation pages on a few related commands (TBD) which I eventually realized could be chained together to achieve what I wanted.

First I set up a credentials object:

$credential = Get-Credential

This pops up an authentication window so be prepared with your Microsoft administrator credentials.

cap-Get-Cred-popup

Next I make sure Internet Explorer has the correct proxy settings. Then I inherit them from IE like this:

$drj = New-PSSessionOption -ProxyAccessType IEConfig

I refer to this options object in the next command:

$exchangeSession = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/powershell-liveid/" -Credential $credential -Authentication "Basic" -AllowRedirection -SessionOption $drj

One more command to get things going:

Import-PSSession $exchangeSession

and I’m ready to issue real get/set commands!

Conclusion
Hopefully this posting helps to clear up what to do to make certain commands in PowerShell work through a standard http proxy. PowerShell, for a guy who’s only done BASH scripts, is actually pretty cool.

References
The basic idea of connecting to Exchange Online is contained here in this helpful Microsoft article, but you will find no mention of proxy whatsoever on that page. That part I figured out.

Categories
Admin Internet Mail Scams

The latest trend – Google search engine spam

Intro
I’ve been seeing an uptick in brief spams which provide links to a very legitimate site: the Google search engine!

The details
I’ve been getting a lot – several per day – that look like this one:

From: noc@me02.ru
To: webmaster@drj.com
Subject: Legal drugs forum
 
Legalize!!! Read about strongest legal drugs in the world, and buy it online: https://www.google.com/url?q=http%3A%2F%2F%77%77%77.le%67a%6C%69z%65r%2EDRJinfo%2F&sa=D&usg=AFQjCNG0coaOvXJMkOn0nEMvP-dl11XKnQ
 
Attention: MDMB(N)-BZ-F is not allowed now!

Here’s another example which appears to be a different spam campaign using the same technique which I received several weeks after initially posting this article:

From: zaria3946044.6159f@e-technik.uni-rostock.de
Subject: Turn your bedroom into paradise of satisfaction
 
https://www.google.com/url?q=http%3A%2F%2F%73lip.h%65al%69DRJn%67%73%65%63%75re%65%73hop.%65%75%2F&sa=D&usg=AFQjCNFeP_XevUiXV-m-DtxAJVi3SMRtVQ

I’ve changed the links slightly so no one gets in trouble by actually following it.

The link is changed each time and so is the sender.

How to report this?
I have been reporting these to Google directly on their page to Report malicious software, https://www.google.com/safebrowsing/report_badware/.

I have reported five to then of these and have never received a response from Google. It seems the best we can hope for is that Google engineers are sufficiently annoyed by my reports that they begin to agree hey there’s a problem here and maybe people will think less of us if we continue to do nothing.

Why this is particularly devastating
Because the malware link uses this combination:

– https (which encrypts everything)
– a very legitimate web site, www.google.com
– malware

It is very tricky to defeat. Many URL filters, e.g., those used on explicit proxies, cannot peer into https traffic and so have to make a single judgment for a whole site, even one as complicated as www.google.com. Either it is all good, or it is all bad. Who would have the courage to categorize Google as a source of malware and hence block all users from it?

So these perpetrators have engaged in what amounts to link laundering. Some of the URI is encoded in hex, I suppose to help avoid detection and create many valid patterns that are hard for Google to stamp out.

This started over a month ago and is stronger than ever today, so we know at press time Google, in spite of all its advanced technology, does not have a handle on it.

If you see something similar I suggest to report it directly to Google. They may need a little more motivation than I can single-handedly provide them.

Conclusion
Link laundering is now an avenue to sneak spam through. It uses links that point to the Google search engine itself. It seems to have eluded them or been under their radar in spite of many reports. Let’s hope the bad guys don’t have the upper hand permanently.

Appendix
If you are interested in how the URL looks decoded I figured there would be decoders available on the Internet and indeed there are. For instance at http://meyerweb.com/eric/tools/dencoder/

So the URL mentioned above decodes as (again just slightly obfuscated to not make good people do bad things by mistake:

https://www.google.com/url?q=http://www.legalizerDRJ.info/&sa=D&usg=AFQjCNG0coaOvXJMkOn0nEMvP-dl11XKnQ

References
enom-originated spam is discussed here.

Categories
Admin Internet Mail Spam

Gmail: not as much a white-glove service as you thought

Intro
I have a pretty high regard for Google and their Gmail email service. They really seem to strive to keep its reputation sterling. But lately a persistent spam has been coming in to me from one of their users and no action is being taken, so I am beginning to wonder.

The details

It’s not that I don’t get spam from Gmail account holders. I do. That’s not shocking as I get hundreds of spam each day since my address is available from whois registrations on hundreds of domains, amongst other readily available sources.

How do I know it’s a real Gmail user and not someone spoofing the sender address? These two headers tell me:

Received-SPF: pass (google.com: domain of zhuxumeitom@gmail.com designates 209.85.160.44 as permitted sender) client-ip=209.85.160.44;
Received: from mail-pb0-f44.google.com ([209.85.160.44]) (using TLSv1) by drjohnstechtalk.com, etc.

In other words one of the received headers provided by a trusted server gives me the IP of the sending server (209.85.160.44), which is in Google’s directly allocated IP space.
This can be confirmed at arin.net.

The email itself looks like this:

From: "Tom Zhu" <zhuxumeitom@gmail.com>
 
Dear Sir/Madam,
 
We are owner of your_domain.mx and Presently we would like to know if you have an
interest in buying it. We are looking to sell this domain for 2,000 Euro.
It has been listed on Sedo.com. You can buy it through the following link:
 
https://sedo.com/search/details.php4?domain=your_domain.mx
...

But instead of your_domain the email mentions a specific copyrighted domain name.

I’ve received it over 10 times from the same sender. The sender is a cybersquatter sending repeated, unsolicited spam. If that doesn’t constitute a violation of their Terms of Use then I don’t know what does. I’ve filed no fewer than five formal abuse complaints to Google over the course of the several months. The Gmail abuse link is in the references. But they keep coming in so I know Google has taken no action whatsoever. And of course I have never heard back from them.

I’ve filed lots of other abuse complaints about other Gmail senders as well, but those others seem to be one-off spams and I don’t get additional emails from them. Yes it takes time to fill out the abuse form, but I do it for the overall good of the Internet. We technical people have a responsibility to make our world better…

Conclusion
I am personally miffed and professionally concerned that Google Gmail may not be operating as clean a shop (white glove) as we all had thought. Here I’ve documented a specific case of documented abuse about which they have done nothing for months on end.

References
Gmail abuse link.
ARIN’s IP lookup service is here.
A detailed look at how enom has handled some spam/domain complaints is written up here.
My successful fight to conquer scads of Chinese language spam is documented here.

Categories
Internet Mail

Exchange Online Protection is currently broken. Resolved.

Intro
6/24th, 3:20 PM
A lot of my outbound emails are currently stuck with this status:

Deferred: 421 4.3.2 The maximum number of concurrent server connections has exceeded a limit, closing transmission channel

For instance, to belcorp.biz.

The MX record:

> dig +short mx belcorp.biz
10 belcorp-biz.mail.eo.outlook.com.

So it ends in outlook.com.

Same for emails to accenture.com. The MX record:

> dig +short mx accenture.com
10 accenture-com.mail.protection.outlook.com.

Also ends in outlook.com.

sheraton.com is another one I see.

So all these domains we can’t currently email to have an MX record ending in outlook.com and so I conclude are using Exchange Online Protection.

The situation has persisted for a couple hours so this doesn’t seem to be a 99.9999% uptime type of service.

Conclusion
Something went seriously wrong with Microsoft’s Exchange Online Protection service today.

6/25/14 update
Apparently this affected lots of Outlook users as well. It was finally fixed last night.

Categories
Internet Mail Spam

enom is the source of recent spam campaigns

Intro
I’m still watching over spam. The latest trend are spam campaigns which have a few characteristics in common perhaps the most interesting of which is that the domains have all been registered at enom.com.

The details
Some other things in common. These recent campaigns fell into two main categories. One set uses domains which are semi-pronounceable. The other are domains which incorporate sensible english words. Both categories have these other features in common.

– brevity (no HTML, for instance)
– valid SPF records (!)
– domains were used for spam almost immediately after having been registered (new domains)

Today’s example

From:        Patriot Survival Plan <Patriot_Survival_Plan@best-survival-plan-types.com> 
To:        <drj@drj.com> 
Date:        05/22/2014 04:22 AM 
Subject:        REVEALED: The Coming Collapse 
 
 
 
--------------------------------------------------------------------------------
 
 
 
 
drj@drj.com
 
Since I exposed this I'm getting a lot of comments. 
 
People are terrified and they are asking me to spread the word even more...
 
So don't miss this because it might be too late for you and your family!
 
Obama's done a lot of stupid things so far, but this one will freeze the blood in your veins!
 
He's been trying hard to keep this from American Patriots... but now his betrayal has finally come to light.
 
And he'll have to pay through the nose for this.
 
But here's a Warning: the effects of Obama's actions will hit you and your family by the end of this year.
 
And they'll hit you like nothing you've ever seen before...
 
So watch this revealing video to know what to expect...
and how to protect against it.
 
-> Watch Blacklisted video now, before it's too late -->                 http://check.best-survival-plan-types.com
 
 
 
 
 
 
 
No_longer_receive_this _Warning :   http://exit.best-survival-plan-types.com
Patriot Survival Plan _405 W. Fairmont Dr. _Tempe, AZ 85282
 
 
 
 
 
First off, there's nothing special 22409526 in the Ironbound. Food in quantity, 22409526not quality. It's amazing how many people 22409526 rate these establishments as excellent. This said, I've always had fun going to these places, 22409526 as long as your dining expectations are gauged accordingly. Therefore, 22409526 my rating reflects those reduced expectations. :)
 
Being a steakhouse, 22409526 one would expect a thorough steak menu such as those at Gallagher's, Luger's, or even Del Frisco's. However, you're not getting true steakhouse fare here; 22409526 it's the Ironbound after all. So, you're getting a less than Prime cut of beef, 22409526 sometimes cooked to your liking.

Whois lookup of best-survival-plan-types.com shows this:

Domain Name: BEST-SURVIVAL-PLAN-TYPES.COM
Registry Domain ID: 1859701370_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.enom.com
Registrar URL: www.enom.com
Updated Date: 2014-05-21 17:26:19Z
Creation Date: 2014-05-22 00:26:00Z
Registrar Registration Expiration Date: 2015-05-22 00:26:00Z
Registrar: ENOM, INC.
Registrar IANA ID: 48
Registrar Abuse Contact Email: abuse@enom.com
Registrar Abuse Contact Phone: +1.4252744500
Reseller: NAMECHEAP.COM
Domain Status: clientTransferProhibited
Registry Registrant ID:
Registrant Name: DONI FOSTER
Registrant Organization: NONE
Registrant Street: 841-4 SPARKLEBERRY LN
Registrant City: COLUMBIA
Registrant State/Province: SC
Registrant Postal Code: 29229
Registrant Country: US
Registrant Phone: +1.8037886966
Registrant Phone Ext:
Registrant Fax: +1.5555555555
Registrant Fax Ext:
Registrant Email: DONIFOSTER73@GMAIL.COM
Registry Admin ID:
Admin Name: DONI FOSTER
Admin Organization: NONE
Admin Street: 841-4 SPARKLEBERRY LN
Admin City: COLUMBIA
Admin State/Province: SC
Admin Postal Code: 29229
Admin Country: US
Admin Phone: +1.8037886966
Admin Phone Ext:
Admin Fax: +1.5555555555
Admin Fax Ext:
Admin Email: DONIFOSTER73@GMAIL.COM
Registry Tech ID:
Tech Name: DONI FOSTER
Tech Organization: NONE
Tech Street: 841-4 SPARKLEBERRY LN
Tech City: COLUMBIA
Tech State/Province: SC
Tech Postal Code: 29229
Tech Country: US
Tech Phone: +1.8037886966
Tech Phone Ext:
Tech Fax: +1.5555555555
Tech Fax Ext:
Tech Email: DONIFOSTER73@GMAIL.COM
Name Server: DNS1.REGISTRAR-SERVERS.COM
Name Server: DNS2.REGISTRAR-SERVERS.COM
Name Server: DNS3.REGISTRAR-SERVERS.COM
Name Server: DNS4.REGISTRAR-SERVERS.COM
Name Server: DNS5.REGISTRAR-SERVERS.COM

See 1) that it was registered yesterday at 17:26:19 Universal Time, and 2) that the registrar is enom?

And the SPF record:

> dig +short txt best-survival-plan-types.com

"v=spf1 a mx ptr ~all"

Actually this domain is a small aberration insofar as it does not have a SPF record with a -all at the end – the others I checked do.

What to do, what to do
Well, I reported the spam to Postini, but I don’t think that has any effect as they are winding down their business.

I am pinning greater hopes on filling out enom’s abuse form. Of course I have no idea what actions, if any, they take. But they claim to take abuse seriously so I am willing to give them their chance to prove that.

enom’s culpability
I don’t feel enom is complicit in this spam. I’m not even sure they can easily stop these rogue operators. But they have to try. Their reputation is at stake. On the Internet there are complaints like this from years ago, that enom domains are spamming.

Every one that comes across my desk I am reporting to them. The time it takes for me to report any individual one isn’t worth the effort compared to the ease of hitting DELETE, but I am hoping to help lead enom to find a pattern in all these goings-on so they can stop these registrations before new ones cause harm – that is why I feel my actions are for the greater good.

Other recently deployed enom domains

Domain

First spam seen

First registered

onlinetncresults.us

8/22

8/21

checkdnconlinesystems.us

8/20

8/20

extremeconcretecoating.com

8/8

8/8

woodsurface.com

8/7

8/7

shorttermloanspecial.com

7/24

7/23

heartattackfighter1.com

6/19

3/2

handle-unsafe-parasites.me

6/10

6/9

best-survivalplan-learn.com

5/28

5/28

survival-plan-days.com

5/27

5/26

only-survival-plan.com

5/20

5/19

local-vehicle-clearance.us

5/19

5/19

ghiused.com

5/14

5/14

pastutmy.com

5/14

5/14

lekabamow.com

5/14

5/14

etc – there are plenty more!

Finally we hear back
Weeks later, on June 14th, I finally received a formal response concerning only-survival-plan.com and local-vehicle-clearance.us.

From: abuse@namecheap.com
Subject: [~OOQ-128-23745]: FW: eNom - Report Abuse - Reference #ABUSE-11116
 
Hello, 
 
Thank you for your email. While the domain name(s) reported is registered with Namecheap, it is hosted with another company. So we cannot check the logs for the domain(s) and confirm if it is involved in sending unsolicited bulk emails. We can only take an action if a report is confirmed by blacklists of trusted anti-spam organizations like SpamHaus or SURBL.
 
Thus, we have initiated a case regarding the following domain(s) blacklisted by trusted anti-spam organizations:
only-survival-plan.com
In case the listing is not removed, the domain(s) will be suspended.
 
The following domain(s) has already been suspended:
local-vehicle-clearance.us
 
Let us also suggest you addressing the issue to the hosting company which servers were involved in email transmission for help with investigating the incident of spam. You may find their IP address in the headers. To find their contact details, please whois this IP address. You may use any public Whois tool like https://www.domaintools.com/ 
 
Kindly let us know if you have any question.
 
-------------------------------
Regards,
Alexander XXX.
Legal & Abuse Department
Namecheap Group
http://www.namecheapgroup.com

Analysis of their response
Reading between the lines, here’s my analysis. There’s some not-well-documented relationship between enom and namecheap.com. I reported the abuse to enom and got a response from namecheap.com. I kind of agree that suspending a domain is a BIG DEAL and a registrar has to be on firm footing to do so. As I write this one Jun 16th, the domains do not yet appear to be suspended. Are you really going to trust Spamhaus to render your judgement? That’s basically one of those extortionist enterprises purportedly offering a take-it-or-leave-it service. If the author of that email was a lawyer, well, their English isn’t the best. That doesn’t provide a lot of confidence in their handling of the matter. And wasn’t my complaint by itself good enough for them to initiate action? I do have to concede the point that the sending of the spam was probably out of their control and probably did come from another hosting company. But it is glib advice to suppose it is that easy to track them down the way they describe. Since they are part of the problem and have the evidence why don’t they follow up with the hosting provider themselves?? There was no mention of my other eight or so formal complaints. So this still seems to be getting an ad hoc one-by-one case treatment and not the, Whoa, we got a problem on our hands and there’s something systemically wrong with what we’re doing here reaction I had hoped to provoke.

Actually I got two responses but with slightly different wording. So they were crafted by hand from some boilerplate text, and yet the person stitching together the boilerplate was sufficiently mindless of the task as to forget they had already just sent me the first email??

So their response is better than a blackhole, but perhaps could be characterized as close to the bare minimum.

I have gotten several other responses from some of my other complaints as well, all saying pretty much the same thing. In August the responses started to look different however.

August responses
Here’s one I received this morning about woodsurface.com, 19 days after my initial complaint:

Hello,
 
This is to inform you that woodsurface.com domain was suspended. It is now pointed to non-resolving nameservers and will be nullrouted once the propagation is over. The domain is locked for modifications in our system.
 
Thank you for letting us know about the issue. 
 
------------------
Regards,
Alexander T.
Legal & Abuse Department
Namecheap.com

Conclusion
I hope my actions spur enom into some action of their own in figuring out where there domain registration requirements are too lax that spammers are taking wholesale advantage of the situation and sullying their reputation.

June, 2014 Update
The storm of spam from enom has subsided. I’m basically not seeing any. Oops. Spoke too soon! New enom-registered domains popped up and created more spam storms (documented in the table above), but not as severe as in the past. I don’t know if our anti-spam filter got better or enom stepped up to the plate and improved their scrutiny of domain registrants. If another spam storm hits us I’ll report back…

August, 2014
enom-generated spam is back!

References
My most popular spam-fighting article describes how to defeat Chinese-language spam.
A new type of spam that uses Google search results for link laundering is described here.

Categories
Admin Internet Mail Linux

Sendmail: getting mailertable, smarttable and virtusertable to play nice together

Intro
Remember when I posted about some obscure but cool sendmail features? Well in that posting I mentioned having trouble with a catch-all mailertable entry co-existing with a smarttable. I recently got an incentive to work through that and I am sharing my results here. Plus use of virtusertable thrown in for good measure.

The setup
I have a Secure Mail Gateway (SMGW). Some of my users use it, most do not. So for the few who do I wish to divert my outgoing mail, if it sent by one of them, to the SMGW1. Everything else that’s outbound gets forwarded to an Internet-facing relay2. And for inbound mail I also want to divert their email over to it so it can do S/MIME or PGP decryption3. All other inbound emails should go to my native email system4.

So consider the outbound stream, requirement 1. Sender-based routing. That’s the normal thing in sendmail. That’s why you use smarttable, which Andrzej Filip has developed. See my references for more details on that. But my mailertable has a catch-all entry like:

.              relay:emgw.drj.com

I found if I got rid of this entry my smarttable entries began to work, but they stopped working if I put it back. I rolled up my sleeves and tried to understand ruleset 0. I became pretty convinced that mailertable runs before smarttable, and that if mailertable made a successful lookup of the recipie4nt domain that’s it. You’re done. Then I read a very brief comment in Andrzej’s writeup of smarttable. He bascially said mailertable runs first, and I guess form that brief comment you’re supposed to know that this setup I am trying will never work. And yet I have a colleague with a similar setup who says it does work, which got my competitive juices flowing – if the other guy can do it, then I know it can be done and I can do it as well! I knew I had to eliminate the catch-all entry, but still needed a catch-all feature. What to do? Fudge DNS to the point where all TLDs have a fake MX entry pointing at my external mail server?? Sounds too kludgy.

I was reading here and there in the Sendmail book and the cf/README file when SMARTHOST caught my eye. A smarthost can be defined to deliver all email to a specified relay. I always viewed it as an alternative to the mailertable, where you can specify much more specific delivery rules. But maybe they could co-exist? Mailertable for domain-specific delivery instructions, and smarthost for everything else? Yes, you can indeed do that! And with the catchall entry gone from mailertable does smarttable begin to work? Yes! It does! So our outbound stream is in good shape and all requirements are met. Each user of the SMGW is entered in the smarttable:

user1@drj.com    relay:smgw.drj.com
user2@drj.com    relay:smgw.drj.com
etc.

The mc configuration for smarthost looks like this:

dnl smarthost: to take care of the everything-else delivery case - DrJ 2/28/14
define(`SMART_HOST',`internet-facing-relay.drj.com')

Inbound
For inbound we can use virtusertable to rewrite the user domain to a fictional domain, then a mailertable entry which describes that this fictional domain should be routed over to the SMGW! Like this.

Virtusertable

user1@drj.com   user1@drj.com.smgw
user2@drj.com   user2@drj.com.smgw
etc

and

Mailertable

drj.com   relay:native-mail-system-gw.drj.com
.smgw     relay:smgw.drj.com

Although this sounded good on paper, I found it was not enough by itself. In addition I needed to throw in a virtuser domain file which included the domain drj.com.

The mc configuration for virtusertable and virtuserdomain look like this.

dnl
dnl add virtusertable to do recipient re-writing to accomodate SMGW routing -DrJ 3/3/14
FEATURE(`virtusertable',`hash -o /etc/mail/virtusertable')dnl
dnl virtuser domain file location - DrJ 2/27/14
VIRTUSER_DOMAIN_FILE(`/etc/mail/virtuserdomains')dnl

virtuserdomains is a standard text file which contains drj.com.

Testing
It is helpful to know how to debug this stuff. Smarttable is probably the hardest. Here’s how to see the before and after effect that smarttable has:

$ sendmail -Csendmail-test.cf -bt<<END
> 3,0 addr@gmail.com
> .Dfuser1@drj.com
> 3,0 addr@gmail.com
> END

The 3,0 lines show the result of running rulesets 3 and 0, I guess. The .Df…line defines the sender, so the last 3,0 line shows how delivery to the sender is now altered if the defined sender exists in the smarttable table.

Regular addresses can be tested with the same -bt switch if you want to see the gory details, or simply -bv:

$ sendmail -Csendmail-test.cf -bt< 3,0 regularaddr@gmail.com
> END

$ sendmail -Csendmail-test.cf -bv regularaddr@gmail.com

It’s assumed you put your sendmail config in sendmail-test.cf to not interfere with production.

Even if all the tests succeed, what I found is that smarthost did not take effect dynamically. I needed to re-start sendmail.

Conclusion
By digging into the innards of sendmail we learned enough to see how things should work together and found that it is indeed possible for smarttable, virtusertable and mailertable to peacefully co-exist, but only with a helping of smarthost and virtuserdomains!

References
I describe smarttable here.
Andrzej’s smarttable page is here.

Categories
Admin Internet Mail

Analysis of a spam campaign and how we managed to fight back for a few days

Intro
A long-running spam campaign has been bothering me lately. In this post I analyze it from a sendmail perspective and provide a simple script I wrote which helped me fight back.

The details
Let’s have a look see at the July 3rd variant of this spam. Although somewhat different from the previous campaigns in that this did not provide users with a carefully phished email to their inbox, from a sendmail perspective it had a lot of the same features.

So the July 3rd spam was a spoof of Marriott. Look at these from lines. They pretty much shout the pattern out:

Jul  3 14:12:20 drjemgw sm-mta[4707]: r63IA8dJ004707: from=existential840@email.marriottmail.com, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-m
ta, relay=eu1sysamx113.postini.com [217.226.243.182]
Jul  3 14:12:22 drjemgw sm-mta[7088]: r63ICDA7007088: from=chaldean4@complains.marriott.com, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, r
elay=eu1sysamx138.postini.com [217.226.243.227]
Jul  3 14:12:23 drjemgw sm-mta[7220]: r63ICIhL007220: from=dessertst8@complains.marriottmail.com, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-m
ta, relay=eu1sysamx103.postini.com [217.226.243.52]
Jul  3 14:12:24 drjemgw sm-mta[7119]: r63ICEp6007119: from=heraldingl48@complains.marriottmail.org, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm
-mta, relay=eu1sysamx112.postini.com [217.226.243.181]
Jul  3 14:12:33 drjemgw sm-mta[7346]: r63ICO8H007346: from=callouslyz@email.marriottmail.net, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta,
relay=eu1sysamx110.postini.com [217.226.243.59]
Jul  3 14:12:34 drjemgw sm-mta[7425]: r63ICTsI007425: from=clampdownrxm@customercare.marriott.net, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-
mta, relay=eu1sysamx107.postini.com [217.226.243.56]
Jul  3 14:12:35 drjemgw sm-mta[7387]: r63ICRMP007387: from=catechizesskz87@clients.marriottmail.net, size=0, class=0, nrcpts=1, proto=SMTP, daemon=s
m-mta, relay=eu1sysamx108.postini.com [217.226.243.57]
Jul  3 14:12:39 drjemgw sm-mta[1757]: r63I7dfa001757: from=nogginv939@inbound.marriott.net, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, re
lay=eu1sysamx138.postini.com [217.226.243.227]
Jul  3 14:12:40 drjemgw sm-mta[6643]: r63IBpYm006643: from=apostle95@ipn.marriott.biz, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, relay=e
u1sysamx120.postini.com [217.226.243.189]
Jul  3 14:12:42 drjemgw sm-mta[4894]: r63IAFug004894: from=splittinga27@mail.marriottmail.net, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta,
 relay=eu1sysamx110.postini.com [217.226.243.59]
Jul  3 14:12:43 drjemgw sm-mta[7573]: r63ICZJq007573: from=carina1@newsletters.marriottmail.com, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mt
a, relay=eu1sysamx140.postini.com [217.226.243.229]
Jul  3 14:12:45 drjemgw sm-mta[7698]: r63ICfP9007698: from=precariouslyjvf619@ema1lsrv32.marriott.org, size=0, class=0, nrcpts=1, proto=SMTP, daemon
=sm-mta, relay=eu1sysamx102.postini.com [217.226.243.51]
Jul  3 14:12:46 drjemgw sm-mta[7610]: r63ICblx007610: from=divertedjz54@customercare.marriott.net, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-
mta, relay=eu1sysamx109.postini.com [217.226.243.58]
Jul  3 14:12:50 drjemgw sm-mta[7792]: r63ICl6Y007792: from=vaporized08@news.marriott.net, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, rela
y=eu1sysamx112.postini.com [217.226.243.181]
Jul  3 14:12:51 drjemgw sm-mta[6072]: r63IBGCU006072: from=confederationh1511@newsletters.marriottmail.org, size=0, class=0, nrcpts=1, proto=SMTP, d
aemon=sm-mta, relay=eu1sysamx126.postini.com [217.226.243.195]
Jul  3 14:12:51 drjemgw sm-mta[7549]: r63ICYnm007549: from=piponj28@clients.marriott.org, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, rela
y=eu1sysamx115.postini.com [217.226.243.184]
Jul  3 14:12:55 drjemgw sm-mta[7882]: r63ICrUW007882: from=playbacksl634@m.marriottmail.net, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, r
elay=eu1sysamx139.postini.com [217.226.243.228]
Jul  3 14:12:57 drjemgw sm-mta[7925]: r63ICtav007925: from=palingkn7@marriottmail.com, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, relay=e
u1sysamx110.postini.com [217.226.243.59]
Jul  3 14:12:57 drjemgw sm-mta[7930]: r63ICu5c007930: from=abrogate2@newsletters.marriottmail.net, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-
mta, relay=eu1sysamx125.postini.com [217.226.243.194]
Jul  3 14:12:58 drjemgw sm-mta[7900]: r63ICsOE007900: from=ahabo86@marriottmail.com, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, relay=eu1
sysamx131.postini.com [217.226.243.220]
Jul  3 14:13:00 drjemgw sm-mta[7976]: r63ICwmu007976: from=camilleo3@mail.marriott.net, size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, relay=
eu1sysamx127.postini.com [217.226.243.196]

Of 2035 of these that were sent out, 192 were delivered, meaning got into the users inbox past Postini’s anti-spam defenses. So that’s a pretty high success rate as spam goes. And users get concerned.

Now look at sendmail’s access file which I created shortly after becoming aware of similar phishing of linkedin.com more recently on July 11th:

# 7/11/13
linkedinmail.com DISCARD
linkedinmail.net DISCARD
linkedinmail.org DISCARD
linkedinmail.biz DISCARD
linkedin.net DISCARD
linkedin.org DISCARD
linkedin.biz DISCARD
inbound.linkedin.com DISCARD
complains.linkedin.com DISCARD
emalsrv.linkedin.com DISCARD
clients.linkedin.com DISCARD
emlreq.linkedin.com DISCARD
customercare.linkedin.com DISCARD
m.linkedin.com DISCARD
enc.linkedin.com DISCARD
services.linkedin.com DISCARD
amc.linkedin.com DISCARD
news.linkedin.com DISCARD

You get the idea.

What I noticed in these campaigns is a wide variety of subdomains of the domain being phished, with and without “mail” attached to the domain. In particular some rather peculiar-looking subdomains such as complains and emalsrv. So I realized that instead of waiting for me to get the spam, I can constantly comb the log file for these peculiar subdomains. If I come across a new one, voila, it means a new spam campaign has just started! And I can send myself an alert so I can decide – by hand – how best to treat it, knowing it will generally follow the pattern of the recent campaigns.

Now here’s the script I wrote to catch this type of pattern early on:

#!/usr/bin/perl
# DrJ, 7/2013
# I keep my sendmail log file here in /maillog/stat.log and cut it daily
$sl = "/maillog/stat.log";
# 10000 lines occurs in about eight minutes
$DEBUG = 0;
$i = 0;
$lastlines = "-10000";
$access = "/etc/mail/access";
open(ACCESS,$access) || die "Cannot open $access!!\n";
@access = <ACCESS>;
open(SL,"/usr/bin/tail $lastlines $sl|") || die "cannot run tail $lastlines on $sl!!";
print "anti-spam domain: ";
while(<SL>){
  ($domain) = /from=\w{1,25}@(?:emalsrv|complains)\.([^\.]+)\./;
  if ($domain) {
# test if we already have it on our access table
    $seenit = 0;
    foreach $line (@access) {
# lazy, inaccurate match, but good enough...
      $seenit = 1 if $line =~ /$domain/;
      print "seenit, domain, line: $seenit, $domain, $line\n" if $DEBUG;
    }
    if (! $seenit) {
      $i++;
      print "$domain\n" if $i == 1;
    }
  }
}

I call the script spam-check.pl. I invoke spam-check.pl every couple minutes from HP SiteScope. There I have alerts set up which email me a brief message that includes the new domain that is being phished.

No sooner had I implemented this script than it went off and told me about that linkedin phishing spam campaign! That was sweet.

Recent campaigns
Here is a chronology of spam campaigns which follow the pattern documented above. They seem to cook them up one per day.

5/16
wallmart.com - their misspelling, not mine!
5/29
amazon.com
6/20
adp.com
date uncertain
ebay.com
7/9
eftps.com
7/10
visabusinessnews.com
7/11
linkedin.com
7/15
ups.com
7/16
twitter.com
7/17
marriott.com
mmm.com - this one changed up the pattern a bit
7/18
marriott.com - again
ups.com - with somewhat new pattern
7/22
AA.com
7/23-28
a bit more AA.com, a smattering of marriott.com and ebay.com
7/29
tapering off...
7/30 and later
spammer seems to have gone on hiatus, or finally been arrested
10/2, they're back
staples.com

One example spam
Here was my phishing spam from 3M which I got yesterday:

From: "3M" <3m_e-commerce@mmm.com>
To: DrJ
__________________________________________________
This is an automated e-mail.
PLEASE DO NOT RESPOND TO THIS EMAIL ACCOUNT.
This account is not reviewed for responses.
 
--------------------------------------------------------------------------------
 
 
This email is to confirm that on 07/17/2013, 3M's bank (JP Morgan) has debited $15,956.64 from your bank account.
 
If you have any questions, please visit the 3M EIPP Helpline at this link.

The HTML source for that last line looks like this:

If you have any questions, please visit the 3M EIPP=
 
		 		 Helpline <a href=3D"http://vlayaway.com/download/mmm.com.e-marketing.ht=
ml?help">at this link.</div>

When I checked Bluecoat’s K9 webfilter, which I even use at home, the URL in the link, vlayaway… was not rated. I submitted a suggest category, Malicious Sources, and they efficiently assigned it that category within minutes of my submission.

Also, note that the envelope sender of my email differs from the Sender header. The envelope sender was celestev@email.mmm-mail.net.

A word about DISCARD vs ERROR
While I’m waiting for more spam of this sort to come in as I write this on July 22nd, I had a brainstorm. Rather than DISCARDind these emails, which doesn’t tip the sender off, it’s probably better to send a 550 error code, which rattles the system a bit more. I think a sending IP with too many of these errors will be temporarily banned by Postini for all their users. So I changed all my DISCARDs. Here is the syntax for one example line:

linkedin-mail.com ERROR:"550 Sender banned. Please use legitimate domain to send email."

I originally wanted to put the message “No such user,” to try to get the spammer to take that specific recipient off their spam list, but it doesn’t really work in the right way: the error is reported in the context of the sender address, not the recipient address.

Here is the protocol which shows what I am talking about:

$ telnet drj.postini.com 25
Trying 217.136.247.13...
Connected to drj.postini.com..
Escape character is '^]'.
220 Postini ESMTP 133 y678_pstn_c6 ready.  CA Business and Professions Code Section 17538.45 forbids use of this system for unsolicited electronic mail advertisements.
helo localhost
250 Postini says hello back
mail from: test@marriott-mail.com
250 Ok
rcpt to: drj@drj.com
550 5.0.0 test@marriott-mail.com... Sender banned. Please use legitimate domain to send email. - on relay of: mail from: test@marriott-mail.com
quit
221 Catch you later

So that – on relay of: mail from: … is added by Postini so it really doesn’t make sense to say No such user in that context.

Conclusion
My satisfaction may be short-lived. But it is always sweet to be on top, even for a short while.

References
For a lighthearted discussion of HP SiteScope, read the comments from this post.

Sendmail is discussed in various posts of mine. For instance, Analyzing the sendmail log, and Obscure tips for sendmail admins.

Categories
Internet Mail

Analyzing the sendmail log

Intro
If you’ve read any of my posts you’ll see that I believe sendmail is a technical marvel, but that’ snot to say it’s without its flaws.

One of the annoying things is that the From line and To line are recorded spearately, in defiance of common logic. I present a simple program I wrote to join these lines.

The details

Without further ado, here is the program, which I call combine.pl:

#!/usr/bin/perl
# combine lines from stat.log
# Copyright work under the Artistic License, http://www.opensource.org/licenses/Artistic-2.0
# DrJ 6/2013 - make more readable based on this format:
# Date|Time|Size[Bytes]|Direction|Sender|Recipient|Relay-MTA
#
#
# From= usually has address surrounded by <>, but not always
#
# input of
#Jun 20 10:00:21 drjemgw sm-mta[24318]: r5KE0K1U024318: from=infatuatinggu3@adp.net, size=5331, class=0, nrcpts=1, msgid=<15936355.7941275772268.JavaMail.easypaynet@Z32C1GBSZ4AN2>, proto=SMTP, daemon=sm-mta, relay=amx139.postini.com [20.26.143.12]
#Jun 20 10:00:22 drjemgw sm-mta[24367]: r5KE0K1U024318: to=<dr.john@drj.com>, delay=00:00:02, xdelay=00:00:01, mailer=esmtp, pri=125331, relay=drjinternal.com [50.17.188.196], dsn=2.0.0, stat=Sent (r5KE0M6E027784 Message accepted for delivery)
# produces
#20.6.2013|10:00:21|5331|IN|infatuatinggu3@adp.net|dr.john@drj.com|amx139.postini.com
#
use Getopt::Std;
getopts('s:f'); # -s search_string -f for "full" version of output
$DEBUG = 0;
 
print "$relay{$ID}, $lines{$ID}, $sender{$ID}, $size{$ID}\n";
$year = `date +%Y`;
chomp($year);
while(<>) {
  chomp;
  print $_ ."\n" if $DEBUG;
#
# get ID
  ($ID) = /\[\d{2,10}\]:\s+(\w+):\s+/;
#print "ID: $ID\n";
  if ($lines{"$ID"} && / stat=Sent /) {
    if ($opt_f) {
      $lines{"$ID"} .= '**to**line**'.$_;
    } else {
      ($recip,$relay) = /:\sto=<(.+)>,\s.*\srelay=(\S+)\s/;
# there can be multiple recipients listed
      $recip =~ s/[\<\>]//g;
# disposition of email.  This needs customization for your situation, but it only determines IN vs OUT vs INTERNAL so it's not critical...
# In this example coding we get all our inbound email from postini.com, and outbound mail comes from drjinternal
      if ($relay{$ID} =~ /postini\.com/) {
        $disp = "IN";
      } else {
        $disp = $relay =~ /drjinternal/ ? "INTERNAL" : "OUT";
      }
      $lines = "$lines{$ID}|$size{$ID}|$disp|$sender{$ID}|$recip|$relay{$ID}";
      if ( ($lines =~ /$opt_s/ || ! $opt_s) && ($sender{$ID} || $recip) ) {
        $lines .= "|$ID" if $DEBUG;
#        push @lines, $lines; # why bother?  just spit it out immediately
         print "$lines\n";
      }
# save memory, hopefully? - can't do this. sometimes we have multiple To lines
#      undef $relay{$ID}, $lines{$ID}, $sender{$ID}, $size{$ID};
      print "$recip\n" if $DEBUG;
    }
  } else {
    if ($opt_f) {
      $lines{"$ID"} .= '**from**line**'.$_;
    } else {
      ($mon,$date,$time,$sender,$size,$relay) = /^(\w+)\s+(\d+)\s+([\d:]+)\s.+\sfrom=<?([^<][^,]*\w)>?,\ssize=(\d+).*relay=(\S+)/;
# convert month name to month number
      $monno = index('JanFebMarAprMayJunJulAugSepOctNovDec',$mon)/3 + 1;
# the year is faked - it's not actually recorded in the logs so we assume it's the current year...
      $lines{$ID} = "$date.$monno.$year|$time";
      $size{$ID} = $size;
      $sender{$ID} = $sender;
      $relay{$ID} = $relay;
 
      print "$mon,$date,$time,$sender,$size,$relay\n" if $DEBUG;
    }
  }
}
 
# now start matching
if ($opt_f) {
  foreach (@lines) {
    print $_."\n"
  }
}

What it does is combine the From and To lines based on the message ID which is unique to a message.

Usage
I usually use it to suck in an entire day’s log (I call my sendmail log stat.log) and grep the output to look for a particular string. For instance today there was a spam blast where ADP’s identity was phished. The sending domains all contained some variant of adp: adp.net, adp.org, adpmail.com, adp.biz, etc. So I wanted to find the answer to the question who’s received any of these ADP phishing emails today? Here’s how you use the program, to do that:

$ combine.pl<stat.log|grep adp.com|more

The input lines look like this:

Jun 20 10:00:21 drjemgw sm-mta[24318]: r5KE0K1U024318: from=infatuatinggu3@adp.net, size=5331, class=0, nrcpts=1, msgid=<15936355.7941275772268.JavaMail.easypaynet@Z32C1GBSZ4AN2>, proto=SMTP, daemon=sm-mta, relay=amx139.postini.com [27.16.14.22]
Jun 20 10:00:22 drjemgw sm-mta[24367]: r5KE0K1U024318: to=<dr.john@drj.com>, delay=00:00:02, xdelay=00:00:01, mailer=esmtp, pri=125331, relay=drjinternal.com. [50.17.188.196], dsn=2.0.0, stat=Sent (r5KE0M6E027784 Message accepted for delivery)

The output from combine.pl looks like this:

20.6.2013|10:00:21|5331|IN|infatuatinggu3@adp.net|dr.john@drj.com|amx139.postini.com

Yeah, I got that ADP spam by the way…

Conclusion
A useful Perl script has been presented which helps mail admins combine separate output lines into a single entry, preserving the most important meta-data from the mail.

Other interesting sendmail posts are also available here and here.

Categories
Admin Internet Mail SLES

The IT Detecive Agency: emails began piling up this week, no obvious cause

Intro
Today I had my choice of problems I could highlight, but I like this one the best. Our mail server delivers email to a wide variety of recipients. All was going well and it ran pretty much unattended until this week when it didn’t go so well. Most emails were getting delivered, but more and more were starting to pile up in the queues. This is the story of how we unraveled the mystery.

The details
It’s best to work from examples I think. I noticed emails to me.com were being refused delivery as well as emails to rnbdesign.com. The latter is a smaller company so we heard from them the usual story that we’re the only ones who can’t send to them.

So I forced delivery with verbose logging. I’m running sendmail, so that looks like this:

> sendmail -qRrnbdesign.com -Cconfig_file -v

That didn’t work out, producing a no route to host type of error. I did a DNS lookup by hand. That showed one set of results, while sendmail was connecting to an entirely different IP address. How could that be??

I was at a loss so I do what I do when I’m desperate: strace. That looks like this:

> strace -f sendmail -qRrnbdesign.com -Cconfig_file -v > /tmp/strace 2>&1

That produced 12,000 lines of output. All the system calls that the process and any of its forked processes invoke. Is that too much to comb through by hand? No, not at all, not when you begin to see the patterns.

I pored over the trace, not knowing what most of it meant, but looking for especially any activity regarding networking and DNS. Around line 6,000 I found it. There was mention of nscd.

For the unaware the use of nscd (nameserver caching daemon) might seem innocent enough, or even good-intentioned. What could be wrong with caching frequently used DNS results? The only issue is that it doesn’t work right! nscd derives from UC Berkeley Unix code and has never been supported. I didn’t even like it when I was running SunOS. It caches the DNS queries but ignores TTLs. This is fatal for mail servers or just about anything you can think of, especially on servers that are infrequently booted as mine are.

I stopped nscd right away:

> service nscd stop

and re-ran the sendmail queue runner (same command as above). The rnbdesign.com emails flowed out instantly! Soon hundreds of stuck emails were flushed out.

Of course for good measure nscd had to be removed from the startup sequence:

> chkconfig nscd off

An IT pro always keeps unsolved mysteries in his mind. This time I knew I also had in hand the solution an earlier-documented mystery about email to paladinny.com.

Conclusion
nscd might show up in your SLES or OpenSuse server. I strongly suggest to disable it before you wind up with old DNS values and an extremely hard-to-debug issue.

Case closed!