Categories
Internet Mail Linux Raspberry Pi

From Audio Recording to YouTube with two button clicks and a Raspberry Pi

Intro

This post builds on the success of previous posts and uses elements from them. I don’t honestly expect anyone to repeat all the ingredients I have assembled here. But I have created them in a fairly modular way so you can pick out those elements which will help your project.

But, it is true, I have gotten the user experience of recording audio from, e.g., a band practice, down to a click of the ENTER button to start the recording, another click to stop it, and a click of the UP ARROW button to process the audio recording – turn it into a video – and upload it to YouTube, mark it as UNLISTED, and send the link to me in an email. Pretty cool if I say so myself. I am refining things as I write this to make it more reliable.

This write-up is not terribly detailed. It presumes at least a medium skill level with linux.

Ingredients
  • RPi 3 or RPi 4
  • Raspberry OS desktop running Pixel desktop environment
  • tiger VNC, i.e., the package tigervnc-scraping-server
  • chromium-browser (but it comes with)
  • xdotool (apt-get install xdotool)
  • xsel (apt-get install xsel)
  • YouTube account
  • crontab entries – see below
  • you do not need an HDMI display, except for the OS setup
  • a vnc viewer such as Real VNC
  • exim4 and bsd-mailx packages
The scripts

recordswitch.sh

#!/bin/bash
# DrJ 8/2021
# Control the livestream of audio to youtube
# works in conjunction with an attached keyboard
# I use bash interpreter to give me access to RegEx matching
HOME=/home/pi
log=$HOME/audiocontrol.log
program=ffmpegwireless9.sh
##program=tst.sh # testing
PGM=$HOME/$program
# de-press ENTER button produces this:
matchE="1, 28, 0"
# up arrow
matchU="1, 103, 0"

epochsOld=0
cutoff=3 # seconds
DEBUG=1
ledtime=10
#
echo "$0 starting monitoring at "$(date)
# Note the use of script -q -c to avoid line buffering of the evread output
script  -q -c $HOME/evread.py /dev/null|while read line; do
[[ $DEBUG -eq 1 ]] && echo line is $line
# seconds since the epoch
epochs=$(date +%s)
elapsed=$((epochs-$epochsOld))
if [[ $elapsed -gt $cutoff ]]; then
  if [[ "$line" =~ $matchE ]]; then
# ENTER button section - recording
    echo "#################"
    echo We caught this input: $line at $(date)
# see if we are already running our recording program or not
    pgrep -f $program>/dev/null
# 0 means it's been found
    if [ $? -eq 0 ]; then
# kill it
      echo KILLING $program
      pkill -9 -f $program; pkill -9 arecord; pkill -9 ffmpeg
      pkill -9 -f blinkLED
      echo Shine the PWR LED
      $HOME/shineLED.sh
    else
# start it
      echo Blinking PWR LED
      $HOME/blinkLED.sh &
      echo STARTING $PGM
      $PGM > $PGM.log.$(date +%m-%d-%y:%H:%M) 2>&1 &
    fi
    epochsOld=$epochs
  elif [[ "$line" =~ $matchU ]]; then
# UP ARROW button section - processing
    echo "###########"
    echo processing commencing at $(date)
    $HOME/blinktwiceLED.sh &
    echo start processing of the recording
    $HOME/process.sh >> process.log 2>&1
    pkill -9 -f LED
    $HOME/shineLED.sh
    epochsOld=$epochs
  fi
[[ $DEBUG -eq 1 ]] && echo No action taken. Continue to listen
fi
done

ffmpegwireless9.sh

#!/bin/sh
ffmpeg \
-thread_queue_size 4096 \
-f alsa -i plughw:1,0 \
-thread_queue_size 64 \
-f lavfi -i color=color=darkgray \
-c:v libx264 -pix_fmt yuv420p -g 18  -x264opts no-scenecut -b:v 50k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 8 \
-b:a 128k \
-r 5 \
-s 480x320 \
-flush_packets 1 \
-f mp3 file:record-$(date +%m-%d-%y-%H-%M).mp3 \
< /dev/null

mp32flv.sh

#!/bin/sh
# DrJ 10/2021
#
# Note that ffmpeg runs at ~ 4 x real-time when it is producing this flv video file
#
line=$1
time=$(ffprobe -v error -show_entries format=duration   -of default=noprint_wrappers=1:nokey=1 file:${line}|tail -1)
echo recording time: $time s
echo $time > duration
video=$(echo ${line}|sed 's/mp3/flv/')
  ffmpeg \
 -i file:${line} \
 -f lavfi -i color=color=darkgray \
 -c:v libx264 -pix_fmt yuv420p -g 18  -x264opts no-scenecut -b:v 50k \
 -bufsize 512k \
 -acodec libmp3lame -ar 44100 \
 -threads 8 \
 -b:a 128k \
 -r 5 \
 -s 480x320 \
 -t $time \
 -f flv file:${video} \
 < /dev/null

auto-upload.sh

#!/bin/sh
# automate upload of YouTube videos
#
# define some functions
randomsleep(){
# sleep random amount between 1.5 to 2.5 seconds
t10=$(shuf -n1 -i 15-25)
t=$(echo $t10/10|bc -l)
sleep $t
}
drjtool(){
randomsleep
xdotool $1 $2 $3
randomsleep
}

echo Start video upload
echo set display to main display
export DISPLAY=:0
# launch chromium
echo launch chromium
chromium-browser --kiosk https://studio.youtube.com/ > /dev/null 2>&1 &
sleep 25
echo move to CREATE button
drjtool mousemove 579 19
echo click on CREATE button
drjtool click 1
echo move to Upload videos
drjtool mousemove 577 34
echo click Upload videos
drjtool click 1
echo move to SELECT FILES
drjtool mousemove 305 266
echo click on SELECT FILES
drjtool click 1
echo move mouse to Open button
drjtool mousemove 600 396
echo click open and pause a bit for video upload
drjtool click 1
sleep 20
secs=$(cat duration)
moretime=$(echo $secs/60|bc -l)
sleep $moretime
echo "mouse to NEXT button (accept defaults)"
drjtool mousemove 558 386
echo click on NEXT
drjtool click 1
echo move to radio button No it is not made for kids
drjtool mousemove  117 284
echo click radio button
drjtool click 1
echo back to NEXT button
drjtool mousemove 551 384
echo click NEXT
drjtool click 1
echo 'click NEXT again (then says no copyright issues found)'
drjtool click 1
echo click NEXT again
drjtool click 1
echo move to Unlisted visibility radio button
# [note that public would be drjtool mousemove 142 235, private is 142 181]
drjtool mousemove 142 208
echo click Unlisted
drjtool click 1
echo move to copy icon
drjtool mousemove 532 249
echo echo copy URL to clipboard
drjtool click 1
echo move to Save
drjtool mousemove 551 384
echo click Save
drjtool click 1
echo move to CLOSE
drjtool mousemove 434 304
echo click close
drjtool click 1

echo video URL
xsel -b|tee clipboard
echo '
kill chromium browser'
sleep 25
echo kill chromium
kill -9 %1
sleep 2
url=$(cat clipboard|xargs -0 echo)
echo url is $url

process.sh

#!/usr/bin/bash
HOME=/home/pi
sleeptime=5
cd $HOME
# loop over all mp3 files in home directory
ls -1 record*mp3|while read line;do
 echo working on $line at $(date)
 video=$(echo ${line}|sed 's/mp3/flv/')
 echo creating flv video file $video
# create the video first
 ./mp32flv.sh $line
 echo move $line to mp3 directory
 [[ -d mp3s ]] || mkdir mp3s
 mv $line mp3s
 echo mv flv to upload directory
 [[ -d 00uploads ]] || mkdir 00uploads
 mv $video 00uploads
 echo start the upload
 ./auto-upload.sh
 echo get the url to this video on YouTube
 url=$(cat clipboard|xargs -0 echo)
 echo test that it worked
 if [[ ! "$url" =~ "http" ]]; then
   echo FAIL. Try once again
   ./auto-upload.sh
 fi
 echo send mail to Drj
 ./announceit.sh
 echo move video $video to flvs directory
 mv ./00uploads/$video flvs
 echo sleep for a bit before starting the next one
 sleep $sleeptime
done
echo All done with processing at $(date)

blinkLED.sh

#!/bin/sh
# DrJ 8/30/2021
# https://www.jeffgeerling.com/blogs/jeff-geerling/controlling-pwr-act-leds-raspberry-pi
# put LED into GPIO mode
echo gpio | sudo tee /sys/class/leds/led1/trigger > /dev/null
# flash the bright RED PWR (power) LED quickly to signal whatever
while /bin/true; do
  echo 0|sudo tee /sys/class/leds/led1/brightness > /dev/null
  sleep 0.5
  echo 1|sudo tee /sys/class/leds/led1/brightness > /dev/null
  sleep 0.5
done

shineLED.sh

#!/bin/sh
# DrJ 8/30/2021
# https://www.jeffgeerling.com/blogs/jeff-geerling/controlling-pwr-act-leds-raspberry-pi
# put LED into GPIO mode
echo gpio | sudo tee /sys/class/leds/led1/trigger > /dev/null
# turn on the bright RED PWR (power) LED
echo 1|sudo tee /sys/class/leds/led1/brightness > /dev/null

blinktwiceLED.sh

#!/bin/sh
# DrJ 8/30/2021
# https://www.jeffgeerling.com/blogs/jeff-geerling/controlling-pwr-act-leds-raspberry-pi
# put LED into GPIO mode
echo gpio | sudo tee /sys/class/leds/led1/trigger > /dev/null
# flash the bright RED PWR (power) LED quickly to signal whatever
while /bin/true; do
  echo 0|sudo tee /sys/class/leds/led1/brightness > /dev/null
  sleep 3
  echo 1|sudo tee /sys/class/leds/led1/brightness > /dev/null
  sleep 0.35
  echo 0|sudo tee /sys/class/leds/led1/brightness > /dev/null
  sleep 0.35
  echo 1|sudo tee /sys/class/leds/led1/brightness > /dev/null
  sleep 0.35
done

announceit.sh

#!/bin/sh
url=$(cat clipboard|xargs -0 echo)
mailx -r [email protected] -s "New youtube video $url posted" [email protected]<<EOF
Check out our latest recording:

      $url

Regards,
Yourself
EOF

crontab entries

@reboot sleep 15; /home/pi/recordswitch.sh > recordswitch.log 2>&1
# launch vnc server on display 1
@reboot sleep 65;x0vncserver -passwordfile ~/.vnc/passwd -display :0 >  x0vncserver.log 2>&1

The idea

The recordswitch.sh script waits for input from the remote controller. It is programmed to kick off ffmpegwireless9.sh if the ENTER button is pushed, or process.sh if the UPLOAD button is pushed.

For testing purposes you may want to run process.sh by hand, i.e., ./process.sh, while you are viewing the display using a VNC viewer alongside the terminal screen.

The scripts are quite verbose and give lots of helpful output in their log files.

Upgrading from Raspberry Pi Lite to Raspberry Pi Desktop

I always like to start my RPi OS install with Raspberry Pi Lite. But to follow the upload parts of this post you really need Raspberry Pi Desktop. This article is a good write-up of how to upgrade to Desktop from Lite: How To Upgrade Raspbian Lite to Desktop (PIXEL, KDE, MATE, …) – RaspberryTips

Tips

Unfortunately the plugin I use inserts a blank line at the top. Those should all be removed.

After getting all the script, make them all executable in one go with a command such as chmod +x *sh

To read the input from the remote controller you need to set up evread.py and there may be some python work to do. This post has those details.

The chromium bowser needs to be run by hand one time over your VNC viewer. Its size has to be shrunk to 50% by running CTRL SHIFT – about four times. You need to log in to your YouTube or Gmail account so it remembers your credentials. And you need to og through the motions of uploading a video so it knows to use the 00uploads directory next time.

Don’t run a recording and an upload at the same time. I think the CPU would be taxed so I did not test that out. But you can record one day – even multiple recordings, and upload them a day or days later. That should work OK. It just processes the files one at a time, hopefully (untested).

announceit.sh is pretty dodgy. You have to understand SMTP mail somewhat to have a spitting chance for that to work. Fortunately I was an SMTP admin previously. So my ISP, Optimum, has a filter in place which prevents ordinary residential customers from sending out normal email to arbitrary SMTP addresses. However, to my surprise, they do run a mail relay server which you can connect to on the standard tcp port 25. I don’t really want to give it away but you can find it with the appropriate Internet search. I assume it is only for Optimum customers. Perhaps your ISP has something similar. So after you install exim4, you can configure a “smarthost” with the command dpkg-reconfigure exim4-config. But, again, you have to know a bit what you are doing. Suffice it to say that I got mine to work.

But for everyone else who can’t figure that out, just comment out this line in process.sh ./announceit.sh. put a # character in the front of the line to do that.

I have really only tested recordings of up to 45 minutes. I think an hour should be fine. I would suggest to break it up for longer.

The files can take a lot of space so you may need to clean up older files if you are a frequent user.

I’ve had about one failure during the upload out of about seven tests. So reliability is pretty good, but probably not perfect.

Why not just livestream? True, it’s sooo much easier. And I’ve covered how to do that previously. But, maybe it’s my WiFi, but its reliability was closer to 50% in my actual experience. I needed greater reliability and turns out I didn’t need the live aspect of the whole thing, just the recording for later critiqueing.

The recording approach I’ve taken uses ffmpeg to directly produce a mp3 file – it’s more compact than a WAV file. In and of itself the mp3 file may be useful to you, to, e.g., include as an attachment in email or whatever. For instance for a single song. All the mp3s are finally stored in a folder called mp3s, and all the videos are finally stored in a folder called flvs.

About that upload

The upload itself is super awesome to watch. I captured an actual automated upload with the script running on the right and the X Window display on the left in this YouTube video.

So the upload part was covered in this previous post.

Fixing recording which sounds like chipmunks

Somehow I managed to use some of these tools the other day and my mp3s ended up sounding like Alvin and the Chipmunks! I wondered if there was a way to recover them. I found there is, though I had to develop it a bit. It uses the new-ish rubberband filter of ffmpeg. I call this tiny script dechipmunk.sh:

#!/bin/sh
input=$1
#ffmpeg -thread_queue_size 2048 -i $input -y -ac 1 -filter:a rubberband='tempo=2' -loglevel warning stretched$input
#ffmpeg -thread_queue_size 2048 -i $input -y -ac 1 -filter:a rubberband='pitch=2' -loglevel warning stretched$input
ffmpeg -thread_queue_size 2048 -i $input -y -ac 1 -filter:a "rubberband=pitch=0.3333:tempo=0.3333" -loglevel warning stretched$input

In my case I had to slow things down and lower the pitch by the same factor: one third, hence the 0.3333.

How to pass multiple options to an ffmpeg filter

In doing the above I had to work out the syntax for passing more than one option to the new rubberband filter of ffmpeg. I wanted to specify both the pitch and the tempo options. So you see from above they had to be separated with a colon and the whole filter expression enclosed in quotes. Hence the funny-looking

“rubberband=pitch=0.3333:tempo=0.3333”

Future development

Well, I’m thinking of removing the chit-chat from the recording in an automated fashion. That may mean applying machine learning, or maybe something simpler if someone has covered this territory before for the RPi. But it might be a good excuse to do a shallow dive into machine learning.

Conclusion

I’m sure this method of YouTube upload is very flaky and will probably only work once or twice, if at all. But at least in my trials, it did work a few times. So perhaps it could be hardened and made more error-correcting. There are a lot of moving parts for it all to work. But it’s definitely cool to watch it go when it is working!

References and related

Rii infrared remote control – only $12: Amazon.com: Rii MX3 Multifunction 2.4G Fly Mouse Mini Wireless Keyboard & Infrared Remote Control & 3-Gyro + 3-Gsensor for Google Android TV/Box, IPTV, HTPC, Windows, MAC OS, PS3 : Electronics

Reading keyboard input.

How To Upgrade Raspbian Lite to Desktop (PIXEL, KDE, MATE, …) – RaspberryTips

YouTube Livestreaming with a click of a button on Raspberry Pi

Automated YouTube video uploading from Raspberry Pi without using the YouTube api

Categories
Exchange Online Internet Mail

Web forms: creating today’s Open Relays

Intro
Maybe it’s just me, but I’ve been receiving a lot of non-delivery reports in the past couple weeks. I happen to receive email addressed to webmaster@<large company>.com and as well I am knowledgeable about how email works. This puts me in the rare position of knowing how to interpret the meaning of the many NDRs I started to see.

The details
This picture shows the actual state of my inbox if I look for NDRs by searching for the word undeliverable:

NDRs-2016-09-16_14-22-17

Included is the preview panel showing the contents of one of those messages from Wells Fargo.

Actions
Personally, I can live with this level of clutter in my inbox – I actually get hundreds of messages a day like many IT folks. But I put this into a larger context. As a responsible Netizen (not sure if anyone really uses that term, but it was a useful one!) I feel a responsibility to keep the Internet running in an orderly way and stop the abusers as quickly as possible. After all I owe my job to a well-functioning Internet. So I decided rather than to do the easy thing – write rules to shunt these aside, or let clutter go to work, or junk email them, I was going to try to go after the source. Actually sources because you may have noticed that there are multiple domains involved.

First analysis
I thought about how these emails could have originated. I thought, OK, someone spoofs the email address webmaster@<large company>.com, they find a reputable mail server somewhere on the Internet; this mail server has to be an open relay. So the first few mail servers I checked out on Mxtoolbox didn’t show any problems or complaints whatsoever – sterling reputations. Turns out that way of thinking is soooo 20th century. Excusable in my case because I was already running sendmail in the 20th century and in those days that was some of the biggest worries – running an open relay.

Upon further reflection
Someone mentioned it’s probably a web form and I thought, Yes. Did you ever see those forms, like, Share this web page, that allow you to enter your email address and send something to a friend – which they will receive as apparently coming from you? What if in addition you could add your own custom message? Well, that’s the modern way of turning a mail server into an open relay.

The fatal recipe
A web form with the following properties is an open relay enabler:

– permits setting sender address
– permits setting recipient address
– has a comment field which will get sent along to the recipient
– does not employ captcha technology

Only visible from its side effects
The thing is, I am only receiving the NDRs, in other words those emails which had a spoofed sender address (my webmaster address) and a recipient address which for whatever reason decided not to accept the message. The “open relay,” failing to send the message, sends the failure notice to the spoofed sender: my webmaster account. But this NDR does not contain the original message, just vestigial hints about the original message, such as what server it was sent from, who was the recipient and sender, what the subject line was (sometimes), and when the receiving system complained. If a spam email was successfully sent in my name I never get to see it!

But I was able to actually find one of these dangerous forms that is being abused. Here it is:

open-relay-web-form-2016-09-16_14-41-28

Of course some of these forms are more restrictive than others. And almost all share the characteristic that they always put certain words into the subject and perhaps the body as well, which is beyond the control of the abuser. But that free-form message field is gold for the abuser and allows them to put their spammy or malicious message into the body of the email.

So after checking out a few of these domains for open relay and coming up empty, I do think all the abuse was done through too-lenient web pages. So I guess that is the current method of creating a de facto open relay.

Results
I’ve written very well-informed emails, initially trying to send to abuse@<domain>.com. But I also got more creative, in some cases tracing the domain to an ISP and looking up how to contact that ISP’s abuse department. I’m kind of disappointed that Wells Fargo hasn’t responded. Many other ISPs did. I believe that some have already corrected the problem. Meanwhile new ones crop up.

Over the weeks I’ve worked – successfully – with several offenders. Each one represented unique aspects and I had to do some IT detective work to track down someone who wuold be likely to respond. The ones who haven’t cooperated at all are overseas. Here is the wall of shame:

ecritel.fr (handles emails for marieclaire.fr web site)
ifastfinancial.com
RTHK.com.hk

Update
After about a week Wells Fargo did give a brief reply. They did not ask for any details. But the spam from their server did stop as far as I can tell.

If it turns into a never-ending battle I will give up, except perhaps to spend a few minutes a day.

Permanent fix
I don’t know the best way to fix this. I used to be a fan of SPF, but its limitations make it impossible for some large companies who need to have too many third parties sending email on their behalf. I guess Google is pushing DMARC, but I haven’t had time to think through if it’s feasible for large enterprises.

Conclusion
Poorly constructed web forms are the new open relay enablers. Be very careful before creating such a form and at a minimum use good captcha technology so its usage can not be automated.

This is speculation but I would not be surprised to learn that there is a marketplace for a listing of all the poorly constructed web forms out there – that information could be very valuable to spammers who have been increasingly shut out of our inboxes by improved anti-spam detection.

References and related
I found this site helpful in finding valid contacts for a domain: abuse.net. You enter the domain and spits back a couple of valid abuse contact addresses.
I only reluctantly use Mxtoolbox. It’s like a necessary evil. So i don’t want to give out a link for it. Probably nearly as good to check out a mail server’s reputation is senderbase.org. They’re not trying to sell you anything.
DMARC – perhaps the email authentication mechanism of the future.
My old post advocating SPF, which just never caught on…
PHPMailer remote code execution explanation, which takes advantages of web forms used to send email.

Categories
Admin Exchange Online Internet Mail

PowerShell and Proxy server

Intro
I’ve used Windows PowerShell for all of a few hours so far. But, still, I think I have something to contribute to the community. The documentation on how to send commands through a standard http proxy is pretty miserable so I’d like to make that more clear. I plan to use PowerShell to administer Exchange online.

The details
Microsoft has some pretty good documentation on PowerShell in general. in particular for my desire to connect to Exchange Online I found this very helpful article. But that article says not a whit about sending your connection through an explicit proxy, which I found bewildering.

But I found some key documentation pages on a few related commands (TBD) which I eventually realized could be chained together to achieve what I wanted.

First I set up a credentials object:

$credential = Get-Credential

This pops up an authentication window so be prepared with your Microsoft administrator credentials.

cap-Get-Cred-popup

Next I make sure Internet Explorer has the correct proxy settings. Then I inherit them from IE like this:

$drj = New-PSSessionOption -ProxyAccessType IEConfig

I refer to this options object in the next command:

$exchangeSession = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/powershell-liveid/" -Credential $credential -Authentication "Basic" -AllowRedirection -SessionOption $drj

One more command to get things going:

Import-PSSession $exchangeSession

and I’m ready to issue real get/set commands!

Conclusion
Hopefully this posting helps to clear up what to do to make certain commands in PowerShell work through a standard http proxy. PowerShell, for a guy who’s only done BASH scripts, is actually pretty cool.

References
The basic idea of connecting to Exchange Online is contained here in this helpful Microsoft article, but you will find no mention of proxy whatsoever on that page. That part I figured out.

Categories
Admin Internet Mail Scams

The latest trend – Google search engine spam

Intro
I’ve been seeing an uptick in brief spams which provide links to a very legitimate site: the Google search engine!

The details
I’ve been getting a lot – several per day – that look like this one:

From: [email protected]
To: [email protected]
Subject: Legal drugs forum
 
Legalize!!! Read about strongest legal drugs in the world, and buy it online: https://www.google.com/url?q=http%3A%2F%2F%77%77%77.le%67a%6C%69z%65r%2EDRJinfo%2F&sa=D&usg=AFQjCNG0coaOvXJMkOn0nEMvP-dl11XKnQ
 
Attention: MDMB(N)-BZ-F is not allowed now!

Here’s another example which appears to be a different spam campaign using the same technique which I received several weeks after initially posting this article:

From: [email protected]
Subject: Turn your bedroom into paradise of satisfaction
 
https://www.google.com/url?q=http%3A%2F%2F%73lip.h%65al%69DRJn%67%73%65%63%75re%65%73hop.%65%75%2F&sa=D&usg=AFQjCNFeP_XevUiXV-m-DtxAJVi3SMRtVQ

I’ve changed the links slightly so no one gets in trouble by actually following it.

The link is changed each time and so is the sender.

How to report this?
I have been reporting these to Google directly on their page to Report malicious software, https://www.google.com/safebrowsing/report_badware/.

I have reported five to then of these and have never received a response from Google. It seems the best we can hope for is that Google engineers are sufficiently annoyed by my reports that they begin to agree hey there’s a problem here and maybe people will think less of us if we continue to do nothing.

Why this is particularly devastating
Because the malware link uses this combination:

– https (which encrypts everything)
– a very legitimate web site, www.google.com
– malware

It is very tricky to defeat. Many URL filters, e.g., those used on explicit proxies, cannot peer into https traffic and so have to make a single judgment for a whole site, even one as complicated as www.google.com. Either it is all good, or it is all bad. Who would have the courage to categorize Google as a source of malware and hence block all users from it?

So these perpetrators have engaged in what amounts to link laundering. Some of the URI is encoded in hex, I suppose to help avoid detection and create many valid patterns that are hard for Google to stamp out.

This started over a month ago and is stronger than ever today, so we know at press time Google, in spite of all its advanced technology, does not have a handle on it.

If you see something similar I suggest to report it directly to Google. They may need a little more motivation than I can single-handedly provide them.

Conclusion
Link laundering is now an avenue to sneak spam through. It uses links that point to the Google search engine itself. It seems to have eluded them or been under their radar in spite of many reports. Let’s hope the bad guys don’t have the upper hand permanently.

Appendix
If you are interested in how the URL looks decoded I figured there would be decoders available on the Internet and indeed there are. For instance at http://meyerweb.com/eric/tools/dencoder/

So the URL mentioned above decodes as (again just slightly obfuscated to not make good people do bad things by mistake:

https://www.google.com/url?q=http://www.legalizerDRJ.info/&sa=D&usg=AFQjCNG0coaOvXJMkOn0nEMvP-dl11XKnQ

References
enom-originated spam is discussed here.

Categories
Admin Internet Mail Spam

Gmail: not as much a white-glove service as you thought

Intro
I have a pretty high regard for Google and their Gmail email service. They really seem to strive to keep its reputation sterling. But lately a persistent spam has been coming in to me from one of their users and no action is being taken, so I am beginning to wonder.

The details

It’s not that I don’t get spam from Gmail account holders. I do. That’s not shocking as I get hundreds of spam each day since my address is available from whois registrations on hundreds of domains, amongst other readily available sources.

How do I know it’s a real Gmail user and not someone spoofing the sender address? These two headers tell me:

Received-SPF: pass (google.com: domain of [email protected] designates 209.85.160.44 as permitted sender) client-ip=209.85.160.44;
Received: from mail-pb0-f44.google.com ([209.85.160.44]) (using TLSv1) by drjohnstechtalk.com, etc.

In other words one of the received headers provided by a trusted server gives me the IP of the sending server (209.85.160.44), which is in Google’s directly allocated IP space.
This can be confirmed at arin.net.

The email itself looks like this:

From: "Tom Zhu" <[email protected]>
 
Dear Sir/Madam,
 
We are owner of your_domain.mx and Presently we would like to know if you have an
interest in buying it. We are looking to sell this domain for 2,000 Euro.
It has been listed on Sedo.com. You can buy it through the following link:
 
https://sedo.com/search/details.php4?domain=your_domain.mx
...

But instead of your_domain the email mentions a specific copyrighted domain name.

I’ve received it over 10 times from the same sender. The sender is a cybersquatter sending repeated, unsolicited spam. If that doesn’t constitute a violation of their Terms of Use then I don’t know what does. I’ve filed no fewer than five formal abuse complaints to Google over the course of the several months. The Gmail abuse link is in the references. But they keep coming in so I know Google has taken no action whatsoever. And of course I have never heard back from them.

I’ve filed lots of other abuse complaints about other Gmail senders as well, but those others seem to be one-off spams and I don’t get additional emails from them. Yes it takes time to fill out the abuse form, but I do it for the overall good of the Internet. We technical people have a responsibility to make our world better…

Conclusion
I am personally miffed and professionally concerned that Google Gmail may not be operating as clean a shop (white glove) as we all had thought. Here I’ve documented a specific case of documented abuse about which they have done nothing for months on end.

References
Gmail abuse link.
ARIN’s IP lookup service is here.
A detailed look at how enom has handled some spam/domain complaints is written up here.
My successful fight to conquer scads of Chinese language spam is documented here.

Categories
Internet Mail

Exchange Online Protection is currently broken. Resolved.

Intro
6/24th, 3:20 PM
A lot of my outbound emails are currently stuck with this status:

Deferred: 421 4.3.2 The maximum number of concurrent server connections has exceeded a limit, closing transmission channel

For instance, to belcorp.biz.

The MX record:

> dig +short mx belcorp.biz
10 belcorp-biz.mail.eo.outlook.com.

So it ends in outlook.com.

Same for emails to accenture.com. The MX record:

> dig +short mx accenture.com
10 accenture-com.mail.protection.outlook.com.

Also ends in outlook.com.

sheraton.com is another one I see.

So all these domains we can’t currently email to have an MX record ending in outlook.com and so I conclude are using Exchange Online Protection.

The situation has persisted for a couple hours so this doesn’t seem to be a 99.9999% uptime type of service.

Conclusion
Something went seriously wrong with Microsoft’s Exchange Online Protection service today.

6/25/14 update
Apparently this affected lots of Outlook users as well. It was finally fixed last night.

Categories
Internet Mail Spam

enom is the source of recent spam campaigns

Intro
I’m still watching over spam. The latest trend are spam campaigns which have a few characteristics in common perhaps the most interesting of which is that the domains have all been registered at enom.com.

The details
Some other things in common. These recent campaigns fell into two main categories. One set uses domains which are semi-pronounceable. The other are domains which incorporate sensible english words. Both categories have these other features in common.

– brevity (no HTML, for instance)
– valid SPF records (!)
– domains were used for spam almost immediately after having been registered (new domains)

Today’s example

From:        Patriot Survival Plan <[email protected]> 
To:        <[email protected]> 
Date:        05/22/2014 04:22 AM 
Subject:        REVEALED: The Coming Collapse 
 
 
 
--------------------------------------------------------------------------------
 
 
 
 
[email protected]
 
Since I exposed this I'm getting a lot of comments. 
 
People are terrified and they are asking me to spread the word even more...
 
So don't miss this because it might be too late for you and your family!
 
Obama's done a lot of stupid things so far, but this one will freeze the blood in your veins!
 
He's been trying hard to keep this from American Patriots... but now his betrayal has finally come to light.
 
And he'll have to pay through the nose for this.
 
But here's a Warning: the effects of Obama's actions will hit you and your family by the end of this year.
 
And they'll hit you like nothing you've ever seen before...
 
So watch this revealing video to know what to expect...
and how to protect against it.
 
-> Watch Blacklisted video now, before it's too late -->                 http://check.best-survival-plan-types.com
 
 
 
 
 
 
 
No_longer_receive_this _Warning :   http://exit.best-survival-plan-types.com
Patriot Survival Plan _405 W. Fairmont Dr. _Tempe, AZ 85282
 
 
 
 
 
First off, there's nothing special 22409526 in the Ironbound. Food in quantity, 22409526not quality. It's amazing how many people 22409526 rate these establishments as excellent. This said, I've always had fun going to these places, 22409526 as long as your dining expectations are gauged accordingly. Therefore, 22409526 my rating reflects those reduced expectations. :)
 
Being a steakhouse, 22409526 one would expect a thorough steak menu such as those at Gallagher's, Luger's, or even Del Frisco's. However, you're not getting true steakhouse fare here; 22409526 it's the Ironbound after all. So, you're getting a less than Prime cut of beef, 22409526 sometimes cooked to your liking.

Whois lookup of best-survival-plan-types.com shows this:

Domain Name: BEST-SURVIVAL-PLAN-TYPES.COM
Registry Domain ID: 1859701370_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.enom.com
Registrar URL: www.enom.com
Updated Date: 2014-05-21 17:26:19Z
Creation Date: 2014-05-22 00:26:00Z
Registrar Registration Expiration Date: 2015-05-22 00:26:00Z
Registrar: ENOM, INC.
Registrar IANA ID: 48
Registrar Abuse Contact Email: [email protected]
Registrar Abuse Contact Phone: +1.4252744500
Reseller: NAMECHEAP.COM
Domain Status: clientTransferProhibited
Registry Registrant ID:
Registrant Name: DONI FOSTER
Registrant Organization: NONE
Registrant Street: 841-4 SPARKLEBERRY LN
Registrant City: COLUMBIA
Registrant State/Province: SC
Registrant Postal Code: 29229
Registrant Country: US
Registrant Phone: +1.8037886966
Registrant Phone Ext:
Registrant Fax: +1.5555555555
Registrant Fax Ext:
Registrant Email: [email protected]
Registry Admin ID:
Admin Name: DONI FOSTER
Admin Organization: NONE
Admin Street: 841-4 SPARKLEBERRY LN
Admin City: COLUMBIA
Admin State/Province: SC
Admin Postal Code: 29229
Admin Country: US
Admin Phone: +1.8037886966
Admin Phone Ext:
Admin Fax: +1.5555555555
Admin Fax Ext:
Admin Email: [email protected]
Registry Tech ID:
Tech Name: DONI FOSTER
Tech Organization: NONE
Tech Street: 841-4 SPARKLEBERRY LN
Tech City: COLUMBIA
Tech State/Province: SC
Tech Postal Code: 29229
Tech Country: US
Tech Phone: +1.8037886966
Tech Phone Ext:
Tech Fax: +1.5555555555
Tech Fax Ext:
Tech Email: [email protected]
Name Server: DNS1.REGISTRAR-SERVERS.COM
Name Server: DNS2.REGISTRAR-SERVERS.COM
Name Server: DNS3.REGISTRAR-SERVERS.COM
Name Server: DNS4.REGISTRAR-SERVERS.COM
Name Server: DNS5.REGISTRAR-SERVERS.COM

See 1) that it was registered yesterday at 17:26:19 Universal Time, and 2) that the registrar is enom?

And the SPF record:

> dig +short txt best-survival-plan-types.com

"v=spf1 a mx ptr ~all"

Actually this domain is a small aberration insofar as it does not have a SPF record with a -all at the end – the others I checked do.

What to do, what to do
Well, I reported the spam to Postini, but I don’t think that has any effect as they are winding down their business.

I am pinning greater hopes on filling out enom’s abuse form. Of course I have no idea what actions, if any, they take. But they claim to take abuse seriously so I am willing to give them their chance to prove that.

enom’s culpability
I don’t feel enom is complicit in this spam. I’m not even sure they can easily stop these rogue operators. But they have to try. Their reputation is at stake. On the Internet there are complaints like this from years ago, that enom domains are spamming.

Every one that comes across my desk I am reporting to them. The time it takes for me to report any individual one isn’t worth the effort compared to the ease of hitting DELETE, but I am hoping to help lead enom to find a pattern in all these goings-on so they can stop these registrations before new ones cause harm – that is why I feel my actions are for the greater good.

Other recently deployed enom domains

Domain

First spam seen

First registered

onlinetncresults.us

8/22

8/21

checkdnconlinesystems.us

8/20

8/20

extremeconcretecoating.com

8/8

8/8

woodsurface.com

8/7

8/7

shorttermloanspecial.com

7/24

7/23

heartattackfighter1.com

6/19

3/2

handle-unsafe-parasites.me

6/10

6/9

best-survivalplan-learn.com

5/28

5/28

survival-plan-days.com

5/27

5/26

only-survival-plan.com

5/20

5/19

local-vehicle-clearance.us

5/19

5/19

ghiused.com

5/14

5/14

pastutmy.com

5/14

5/14

lekabamow.com

5/14

5/14

etc – there are plenty more!

Finally we hear back
Weeks later, on June 14th, I finally received a formal response concerning only-survival-plan.com and local-vehicle-clearance.us.

From: [email protected]
Subject: [~OOQ-128-23745]: FW: eNom - Report Abuse - Reference #ABUSE-11116
 
Hello, 
 
Thank you for your email. While the domain name(s) reported is registered with Namecheap, it is hosted with another company. So we cannot check the logs for the domain(s) and confirm if it is involved in sending unsolicited bulk emails. We can only take an action if a report is confirmed by blacklists of trusted anti-spam organizations like SpamHaus or SURBL.
 
Thus, we have initiated a case regarding the following domain(s) blacklisted by trusted anti-spam organizations:
only-survival-plan.com
In case the listing is not removed, the domain(s) will be suspended.
 
The following domain(s) has already been suspended:
local-vehicle-clearance.us
 
Let us also suggest you addressing the issue to the hosting company which servers were involved in email transmission for help with investigating the incident of spam. You may find their IP address in the headers. To find their contact details, please whois this IP address. You may use any public Whois tool like https://www.domaintools.com/ 
 
Kindly let us know if you have any question.
 
-------------------------------
Regards,
Alexander XXX.
Legal & Abuse Department
Namecheap Group
http://www.namecheapgroup.com

Analysis of their response
Reading between the lines, here’s my analysis. There’s some not-well-documented relationship between enom and namecheap.com. I reported the abuse to enom and got a response from namecheap.com. I kind of agree that suspending a domain is a BIG DEAL and a registrar has to be on firm footing to do so. As I write this one Jun 16th, the domains do not yet appear to be suspended. Are you really going to trust Spamhaus to render your judgement? That’s basically one of those extortionist enterprises purportedly offering a take-it-or-leave-it service. If the author of that email was a lawyer, well, their English isn’t the best. That doesn’t provide a lot of confidence in their handling of the matter. And wasn’t my complaint by itself good enough for them to initiate action? I do have to concede the point that the sending of the spam was probably out of their control and probably did come from another hosting company. But it is glib advice to suppose it is that easy to track them down the way they describe. Since they are part of the problem and have the evidence why don’t they follow up with the hosting provider themselves?? There was no mention of my other eight or so formal complaints. So this still seems to be getting an ad hoc one-by-one case treatment and not the, Whoa, we got a problem on our hands and there’s something systemically wrong with what we’re doing here reaction I had hoped to provoke.

Actually I got two responses but with slightly different wording. So they were crafted by hand from some boilerplate text, and yet the person stitching together the boilerplate was sufficiently mindless of the task as to forget they had already just sent me the first email??

So their response is better than a blackhole, but perhaps could be characterized as close to the bare minimum.

I have gotten several other responses from some of my other complaints as well, all saying pretty much the same thing. In August the responses started to look different however.

August responses
Here’s one I received this morning about woodsurface.com, 19 days after my initial complaint:

Hello,
 
This is to inform you that woodsurface.com domain was suspended. It is now pointed to non-resolving nameservers and will be nullrouted once the propagation is over. The domain is locked for modifications in our system.
 
Thank you for letting us know about the issue. 
 
------------------
Regards,
Alexander T.
Legal & Abuse Department
Namecheap.com

Conclusion
I hope my actions spur enom into some action of their own in figuring out where there domain registration requirements are too lax that spammers are taking wholesale advantage of the situation and sullying their reputation.

June, 2014 Update
The storm of spam from enom has subsided. I’m basically not seeing any. Oops. Spoke too soon! New enom-registered domains popped up and created more spam storms (documented in the table above), but not as severe as in the past. I don’t know if our anti-spam filter got better or enom stepped up to the plate and improved their scrutiny of domain registrants. If another spam storm hits us I’ll report back…

August, 2014
enom-generated spam is back!

References
My most popular spam-fighting article describes how to defeat Chinese-language spam.
A new type of spam that uses Google search results for link laundering is described here.

Categories
Admin Internet Mail Linux

Sendmail: getting mailertable, smarttable and virtusertable to play nice together

Intro
Remember when I posted about some obscure but cool sendmail features? Well in that posting I mentioned having trouble with a catch-all mailertable entry co-existing with a smarttable. I recently got an incentive to work through that and I am sharing my results here. Plus use of virtusertable thrown in for good measure.

The setup
I have a Secure Mail Gateway (SMGW). Some of my users use it, most do not. So for the few who do I wish to divert my outgoing mail, if it sent by one of them, to the SMGW1. Everything else that’s outbound gets forwarded to an Internet-facing relay2. And for inbound mail I also want to divert their email over to it so it can do S/MIME or PGP decryption3. All other inbound emails should go to my native email system4.

So consider the outbound stream, requirement 1. Sender-based routing. That’s the normal thing in sendmail. That’s why you use smarttable, which Andrzej Filip has developed. See my references for more details on that. But my mailertable has a catch-all entry like:

.              relay:emgw.drj.com

I found if I got rid of this entry my smarttable entries began to work, but they stopped working if I put it back. I rolled up my sleeves and tried to understand ruleset 0. I became pretty convinced that mailertable runs before smarttable, and that if mailertable made a successful lookup of the recipie4nt domain that’s it. You’re done. Then I read a very brief comment in Andrzej’s writeup of smarttable. He bascially said mailertable runs first, and I guess form that brief comment you’re supposed to know that this setup I am trying will never work. And yet I have a colleague with a similar setup who says it does work, which got my competitive juices flowing – if the other guy can do it, then I know it can be done and I can do it as well! I knew I had to eliminate the catch-all entry, but still needed a catch-all feature. What to do? Fudge DNS to the point where all TLDs have a fake MX entry pointing at my external mail server?? Sounds too kludgy.

I was reading here and there in the Sendmail book and the cf/README file when SMARTHOST caught my eye. A smarthost can be defined to deliver all email to a specified relay. I always viewed it as an alternative to the mailertable, where you can specify much more specific delivery rules. But maybe they could co-exist? Mailertable for domain-specific delivery instructions, and smarthost for everything else? Yes, you can indeed do that! And with the catchall entry gone from mailertable does smarttable begin to work? Yes! It does! So our outbound stream is in good shape and all requirements are met. Each user of the SMGW is entered in the smarttable:

[email protected]    relay:smgw.drj.com
[email protected]    relay:smgw.drj.com
etc.

The mc configuration for smarthost looks like this:

dnl smarthost: to take care of the everything-else delivery case - DrJ 2/28/14
define(`SMART_HOST',`internet-facing-relay.drj.com')

Inbound
For inbound we can use virtusertable to rewrite the user domain to a fictional domain, then a mailertable entry which describes that this fictional domain should be routed over to the SMGW! Like this.

Virtusertable

and

Mailertable

drj.com   relay:native-mail-system-gw.drj.com
.smgw     relay:smgw.drj.com

Although this sounded good on paper, I found it was not enough by itself. In addition I needed to throw in a virtuser domain file which included the domain drj.com.

The mc configuration for virtusertable and virtuserdomain look like this.

dnl
dnl add virtusertable to do recipient re-writing to accomodate SMGW routing -DrJ 3/3/14
FEATURE(`virtusertable',`hash -o /etc/mail/virtusertable')dnl
dnl virtuser domain file location - DrJ 2/27/14
VIRTUSER_DOMAIN_FILE(`/etc/mail/virtuserdomains')dnl

virtuserdomains is a standard text file which contains drj.com.

Testing
It is helpful to know how to debug this stuff. Smarttable is probably the hardest. Here’s how to see the before and after effect that smarttable has:

$ sendmail -Csendmail-test.cf -bt<<END
> 3,0 [email protected]
> [email protected]
> 3,0 [email protected]
> END

The 3,0 lines show the result of running rulesets 3 and 0, I guess. The .Df…line defines the sender, so the last 3,0 line shows how delivery to the sender is now altered if the defined sender exists in the smarttable table.

Regular addresses can be tested with the same -bt switch if you want to see the gory details, or simply -bv:

$ sendmail -Csendmail-test.cf -bt< 3,0 [email protected]
> END

$ sendmail -Csendmail-test.cf -bv [email protected]

It’s assumed you put your sendmail config in sendmail-test.cf to not interfere with production.

Even if all the tests succeed, what I found is that smarthost did not take effect dynamically. I needed to re-start sendmail.

Conclusion
By digging into the innards of sendmail we learned enough to see how things should work together and found that it is indeed possible for smarttable, virtusertable and mailertable to peacefully co-exist, but only with a helping of smarthost and virtuserdomains!

References
I describe smarttable here.
Andrzej’s smarttable page is here.

Categories
Admin Internet Mail

Analysis of a spam campaign and how we managed to fight back for a few days

Intro
A long-running spam campaign has been bothering me lately. In this post I analyze it from a sendmail perspective and provide a simple script I wrote which helped me fight back.

The details
Let’s have a look see at the July 3rd variant of this spam. Although somewhat different from the previous campaigns in that this did not provide users with a carefully phished email to their inbox, from a sendmail perspective it had a lot of the same features.

So the July 3rd spam was a spoof of Marriott. Look at these from lines. They pretty much shout the pattern out:

Jul  3 14:12:20 drjemgw sm-mta[4707]: r63IA8dJ004707: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-m
ta, relay=eu1sysamx113.postini.com [217.226.243.182]
Jul  3 14:12:22 drjemgw sm-mta[7088]: r63ICDA7007088: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, r
elay=eu1sysamx138.postini.com [217.226.243.227]
Jul  3 14:12:23 drjemgw sm-mta[7220]: r63ICIhL007220: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-m
ta, relay=eu1sysamx103.postini.com [217.226.243.52]
Jul  3 14:12:24 drjemgw sm-mta[7119]: r63ICEp6007119: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm
-mta, relay=eu1sysamx112.postini.com [217.226.243.181]
Jul  3 14:12:33 drjemgw sm-mta[7346]: r63ICO8H007346: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta,
relay=eu1sysamx110.postini.com [217.226.243.59]
Jul  3 14:12:34 drjemgw sm-mta[7425]: r63ICTsI007425: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-
mta, relay=eu1sysamx107.postini.com [217.226.243.56]
Jul  3 14:12:35 drjemgw sm-mta[7387]: r63ICRMP007387: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=s
m-mta, relay=eu1sysamx108.postini.com [217.226.243.57]
Jul  3 14:12:39 drjemgw sm-mta[1757]: r63I7dfa001757: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, re
lay=eu1sysamx138.postini.com [217.226.243.227]
Jul  3 14:12:40 drjemgw sm-mta[6643]: r63IBpYm006643: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, relay=e
u1sysamx120.postini.com [217.226.243.189]
Jul  3 14:12:42 drjemgw sm-mta[4894]: r63IAFug004894: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta,
 relay=eu1sysamx110.postini.com [217.226.243.59]
Jul  3 14:12:43 drjemgw sm-mta[7573]: r63ICZJq007573: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mt
a, relay=eu1sysamx140.postini.com [217.226.243.229]
Jul  3 14:12:45 drjemgw sm-mta[7698]: r63ICfP9007698: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon
=sm-mta, relay=eu1sysamx102.postini.com [217.226.243.51]
Jul  3 14:12:46 drjemgw sm-mta[7610]: r63ICblx007610: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-
mta, relay=eu1sysamx109.postini.com [217.226.243.58]
Jul  3 14:12:50 drjemgw sm-mta[7792]: r63ICl6Y007792: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, rela
y=eu1sysamx112.postini.com [217.226.243.181]
Jul  3 14:12:51 drjemgw sm-mta[6072]: r63IBGCU006072: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, d
aemon=sm-mta, relay=eu1sysamx126.postini.com [217.226.243.195]
Jul  3 14:12:51 drjemgw sm-mta[7549]: r63ICYnm007549: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, rela
y=eu1sysamx115.postini.com [217.226.243.184]
Jul  3 14:12:55 drjemgw sm-mta[7882]: r63ICrUW007882: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, r
elay=eu1sysamx139.postini.com [217.226.243.228]
Jul  3 14:12:57 drjemgw sm-mta[7925]: r63ICtav007925: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, relay=e
u1sysamx110.postini.com [217.226.243.59]
Jul  3 14:12:57 drjemgw sm-mta[7930]: r63ICu5c007930: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-
mta, relay=eu1sysamx125.postini.com [217.226.243.194]
Jul  3 14:12:58 drjemgw sm-mta[7900]: r63ICsOE007900: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, relay=eu1
sysamx131.postini.com [217.226.243.220]
Jul  3 14:13:00 drjemgw sm-mta[7976]: r63ICwmu007976: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, relay=
eu1sysamx127.postini.com [217.226.243.196]

Of 2035 of these that were sent out, 192 were delivered, meaning got into the users inbox past Postini’s anti-spam defenses. So that’s a pretty high success rate as spam goes. And users get concerned.

Now look at sendmail’s access file which I created shortly after becoming aware of similar phishing of linkedin.com more recently on July 11th:

# 7/11/13
linkedinmail.com DISCARD
linkedinmail.net DISCARD
linkedinmail.org DISCARD
linkedinmail.biz DISCARD
linkedin.net DISCARD
linkedin.org DISCARD
linkedin.biz DISCARD
inbound.linkedin.com DISCARD
complains.linkedin.com DISCARD
emalsrv.linkedin.com DISCARD
clients.linkedin.com DISCARD
emlreq.linkedin.com DISCARD
customercare.linkedin.com DISCARD
m.linkedin.com DISCARD
enc.linkedin.com DISCARD
services.linkedin.com DISCARD
amc.linkedin.com DISCARD
news.linkedin.com DISCARD

You get the idea.

What I noticed in these campaigns is a wide variety of subdomains of the domain being phished, with and without “mail” attached to the domain. In particular some rather peculiar-looking subdomains such as complains and emalsrv. So I realized that instead of waiting for me to get the spam, I can constantly comb the log file for these peculiar subdomains. If I come across a new one, voila, it means a new spam campaign has just started! And I can send myself an alert so I can decide – by hand – how best to treat it, knowing it will generally follow the pattern of the recent campaigns.

Now here’s the script I wrote to catch this type of pattern early on:

#!/usr/bin/perl
# DrJ, 7/2013
# I keep my sendmail log file here in /maillog/stat.log and cut it daily
$sl = "/maillog/stat.log";
# 10000 lines occurs in about eight minutes
$DEBUG = 0;
$i = 0;
$lastlines = "-10000";
$access = "/etc/mail/access";
open(ACCESS,$access) || die "Cannot open $access!!\n";
@access = <ACCESS>;
open(SL,"/usr/bin/tail $lastlines $sl|") || die "cannot run tail $lastlines on $sl!!";
print "anti-spam domain: ";
while(<SL>){
  ($domain) = /from=\w{1,25}@(?:emalsrv|complains)\.([^\.]+)\./;
  if ($domain) {
# test if we already have it on our access table
    $seenit = 0;
    foreach $line (@access) {
# lazy, inaccurate match, but good enough...
      $seenit = 1 if $line =~ /$domain/;
      print "seenit, domain, line: $seenit, $domain, $line\n" if $DEBUG;
    }
    if (! $seenit) {
      $i++;
      print "$domain\n" if $i == 1;
    }
  }
}

I call the script spam-check.pl. I invoke spam-check.pl every couple minutes from HP SiteScope. There I have alerts set up which email me a brief message that includes the new domain that is being phished.

No sooner had I implemented this script than it went off and told me about that linkedin phishing spam campaign! That was sweet.

Recent campaigns
Here is a chronology of spam campaigns which follow the pattern documented above. They seem to cook them up one per day.

5/16
wallmart.com - their misspelling, not mine!
5/29
amazon.com
6/20
adp.com
date uncertain
ebay.com
7/9
eftps.com
7/10
visabusinessnews.com
7/11
linkedin.com
7/15
ups.com
7/16
twitter.com
7/17
marriott.com
mmm.com - this one changed up the pattern a bit
7/18
marriott.com - again
ups.com - with somewhat new pattern
7/22
AA.com
7/23-28
a bit more AA.com, a smattering of marriott.com and ebay.com
7/29
tapering off...
7/30 and later
spammer seems to have gone on hiatus, or finally been arrested
10/2, they're back
staples.com

One example spam
Here was my phishing spam from 3M which I got yesterday:

From: "3M" <[email protected]>
To: DrJ
__________________________________________________
This is an automated e-mail.
PLEASE DO NOT RESPOND TO THIS EMAIL ACCOUNT.
This account is not reviewed for responses.
 
--------------------------------------------------------------------------------
 
 
This email is to confirm that on 07/17/2013, 3M's bank (JP Morgan) has debited $15,956.64 from your bank account.
 
If you have any questions, please visit the 3M EIPP Helpline at this link.

The HTML source for that last line looks like this:

If you have any questions, please visit the 3M EIPP=
 
		 		 Helpline <a href=3D"http://vlayaway.com/download/mmm.com.e-marketing.ht=
ml?help">at this link.</div>

When I checked Bluecoat’s K9 webfilter, which I even use at home, the URL in the link, vlayaway… was not rated. I submitted a suggest category, Malicious Sources, and they efficiently assigned it that category within minutes of my submission.

Also, note that the envelope sender of my email differs from the Sender header. The envelope sender was [email protected].

A word about DISCARD vs ERROR
While I’m waiting for more spam of this sort to come in as I write this on July 22nd, I had a brainstorm. Rather than DISCARDind these emails, which doesn’t tip the sender off, it’s probably better to send a 550 error code, which rattles the system a bit more. I think a sending IP with too many of these errors will be temporarily banned by Postini for all their users. So I changed all my DISCARDs. Here is the syntax for one example line:

linkedin-mail.com ERROR:"550 Sender banned. Please use legitimate domain to send email."

I originally wanted to put the message “No such user,” to try to get the spammer to take that specific recipient off their spam list, but it doesn’t really work in the right way: the error is reported in the context of the sender address, not the recipient address.

Here is the protocol which shows what I am talking about:

$ telnet drj.postini.com 25
Trying 217.136.247.13...
Connected to drj.postini.com..
Escape character is '^]'.
220 Postini ESMTP 133 y678_pstn_c6 ready.  CA Business and Professions Code Section 17538.45 forbids use of this system for unsolicited electronic mail advertisements.
helo localhost
250 Postini says hello back
mail from: [email protected]
250 Ok
rcpt to: [email protected]
550 5.0.0 [email protected]... Sender banned. Please use legitimate domain to send email. - on relay of: mail from: [email protected]
quit
221 Catch you later

So that – on relay of: mail from: … is added by Postini so it really doesn’t make sense to say No such user in that context.

Conclusion
My satisfaction may be short-lived. But it is always sweet to be on top, even for a short while.

References
For a lighthearted discussion of HP SiteScope, read the comments from this post.

Sendmail is discussed in various posts of mine. For instance, Analyzing the sendmail log, and Obscure tips for sendmail admins.

Categories
Internet Mail

Analyzing the sendmail log

Intro
If you’ve read any of my posts you’ll see that I believe sendmail is a technical marvel, but that’ snot to say it’s without its flaws.

One of the annoying things is that the From line and To line are recorded spearately, in defiance of common logic. I present a simple program I wrote to join these lines.

The details

Without further ado, here is the program, which I call combine.pl:

#!/usr/bin/perl
# combine lines from stat.log
# Copyright work under the Artistic License, http://www.opensource.org/licenses/Artistic-2.0
# DrJ 6/2013 - make more readable based on this format:
# Date|Time|Size[Bytes]|Direction|Sender|Recipient|Relay-MTA
#
#
# From= usually has address surrounded by <>, but not always
#
# input of
#Jun 20 10:00:21 drjemgw sm-mta[24318]: r5KE0K1U024318: [email protected], size=5331, class=0, nrcpts=1, msgid=<15936355.7941275772268.JavaMail.easypaynet@Z32C1GBSZ4AN2>, proto=SMTP, daemon=sm-mta, relay=amx139.postini.com [20.26.143.12]
#Jun 20 10:00:22 drjemgw sm-mta[24367]: r5KE0K1U024318: to=<[email protected]>, delay=00:00:02, xdelay=00:00:01, mailer=esmtp, pri=125331, relay=drjinternal.com [50.17.188.196], dsn=2.0.0, stat=Sent (r5KE0M6E027784 Message accepted for delivery)
# produces
#20.6.2013|10:00:21|5331|IN|[email protected]|[email protected]|amx139.postini.com
#
use Getopt::Std;
getopts('s:f'); # -s search_string -f for "full" version of output
$DEBUG = 0;
 
print "$relay{$ID}, $lines{$ID}, $sender{$ID}, $size{$ID}\n";
$year = `date +%Y`;
chomp($year);
while(<>) {
  chomp;
  print $_ ."\n" if $DEBUG;
#
# get ID
  ($ID) = /\[\d{2,10}\]:\s+(\w+):\s+/;
#print "ID: $ID\n";
  if ($lines{"$ID"} && / stat=Sent /) {
    if ($opt_f) {
      $lines{"$ID"} .= '**to**line**'.$_;
    } else {
      ($recip,$relay) = /:\sto=<(.+)>,\s.*\srelay=(\S+)\s/;
# there can be multiple recipients listed
      $recip =~ s/[\<\>]//g;
# disposition of email.  This needs customization for your situation, but it only determines IN vs OUT vs INTERNAL so it's not critical...
# In this example coding we get all our inbound email from postini.com, and outbound mail comes from drjinternal
      if ($relay{$ID} =~ /postini\.com/) {
        $disp = "IN";
      } else {
        $disp = $relay =~ /drjinternal/ ? "INTERNAL" : "OUT";
      }
      $lines = "$lines{$ID}|$size{$ID}|$disp|$sender{$ID}|$recip|$relay{$ID}";
      if ( ($lines =~ /$opt_s/ || ! $opt_s) && ($sender{$ID} || $recip) ) {
        $lines .= "|$ID" if $DEBUG;
#        push @lines, $lines; # why bother?  just spit it out immediately
         print "$lines\n";
      }
# save memory, hopefully? - can't do this. sometimes we have multiple To lines
#      undef $relay{$ID}, $lines{$ID}, $sender{$ID}, $size{$ID};
      print "$recip\n" if $DEBUG;
    }
  } else {
    if ($opt_f) {
      $lines{"$ID"} .= '**from**line**'.$_;
    } else {
      ($mon,$date,$time,$sender,$size,$relay) = /^(\w+)\s+(\d+)\s+([\d:]+)\s.+\sfrom=<?([^<][^,]*\w)>?,\ssize=(\d+).*relay=(\S+)/;
# convert month name to month number
      $monno = index('JanFebMarAprMayJunJulAugSepOctNovDec',$mon)/3 + 1;
# the year is faked - it's not actually recorded in the logs so we assume it's the current year...
      $lines{$ID} = "$date.$monno.$year|$time";
      $size{$ID} = $size;
      $sender{$ID} = $sender;
      $relay{$ID} = $relay;
 
      print "$mon,$date,$time,$sender,$size,$relay\n" if $DEBUG;
    }
  }
}
 
# now start matching
if ($opt_f) {
  foreach (@lines) {
    print $_."\n"
  }
}

What it does is combine the From and To lines based on the message ID which is unique to a message.

Usage
I usually use it to suck in an entire day’s log (I call my sendmail log stat.log) and grep the output to look for a particular string. For instance today there was a spam blast where ADP’s identity was phished. The sending domains all contained some variant of adp: adp.net, adp.org, adpmail.com, adp.biz, etc. So I wanted to find the answer to the question who’s received any of these ADP phishing emails today? Here’s how you use the program, to do that:

$ combine.pl<stat.log|grep adp.com|more

The input lines look like this:

Jun 20 10:00:21 drjemgw sm-mta[24318]: r5KE0K1U024318: [email protected], size=5331, class=0, nrcpts=1, msgid=<15936355.7941275772268.JavaMail.easypaynet@Z32C1GBSZ4AN2>, proto=SMTP, daemon=sm-mta, relay=amx139.postini.com [27.16.14.22]
Jun 20 10:00:22 drjemgw sm-mta[24367]: r5KE0K1U024318: to=<[email protected]>, delay=00:00:02, xdelay=00:00:01, mailer=esmtp, pri=125331, relay=drjinternal.com. [50.17.188.196], dsn=2.0.0, stat=Sent (r5KE0M6E027784 Message accepted for delivery)

The output from combine.pl looks like this:

20.6.2013|10:00:21|5331|IN|[email protected]|[email protected]|amx139.postini.com

Yeah, I got that ADP spam by the way…

Conclusion
A useful Perl script has been presented which helps mail admins combine separate output lines into a single entry, preserving the most important meta-data from the mail.

Other interesting sendmail posts are also available here and here.