Intro
I’m trying to measure the speed of a NJ Transit train that really hauls past my station. I’ll compare results using two different methods, but no fancy equipment like radar guns. Just a Smartphone a ruler and some common sense.
To be continued…
But Google thinks we are:
$ curl -i http://www.google.com/
returns:
HTTP/1.1 302 Found Location: http://www.google.com.br/?gws_rd=cr&ei=nrA4UoPZB8fb4AP3loGYAg Cache-Control: private Content-Type: text/html; charset=UTF-8 P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info." Date: Tue, 17 Sep 2013 19:42:22 GMT Server: gws X-XSS-Protection: 1; mode=block X-Frame-Options: SAMEORIGIN Alternate-Protocol: 80:quic Content-Length: 262 Proxy-Connection: Keep-Alive Connection: Keep-Alive Set-Cookie: PREF=ID=9ed4d9e91767f4ce:FF=0:TM=1379446942:LM=1379446942:S=O9jxL9sXRCc0kF-E; expires=Thu, 17-Sep-2015 19:42:22 GMT; path=/; domain=.google.com Set-Cookie: NID=67=BJA1pj-o2tu-IDI9KS5MjQvoKPJoY8x3uREAmeGCItGKxxfovFqdjPhvBaHUHcISFLVcTWSmCXwiTAuVhF4DIYCbPcuubfBBEGYaNy2wgveeyvGj35xTzM4Oo-yCLaDe; expires=Wed, 19-Mar-2014 19:42:22 GMT; path=/; domain=.google.com; HttpOnly <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>302 Moved</TITLE></HEAD><BODY> <H1>302 Moved</H1> The document has moved <A HREF="http://www.google.com.br/?gws_rd=cr&ei=nrA4UoPZB8fb4AP3loGYAg">here</A>. </BODY></HTML> |
and MSN is similar:
$ curl -i http://www.msn.com/
HTTP/1.1 302 Found Cache-Control: no-cache, no-store Pragma: no-cache Content-Type: application/xhtml+xml; charset=utf-8 Location: http://br.msn.com/?rd=1&ucc=BR&dcc=BR&opt=0 P3P: CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND" X-AspNet-Version: 4.0.30319 S: BN2SCH030301048 Edge-control: no-store Date: Tue, 17 Sep 2013 20:02:42 GMT Content-Length: 172 Proxy-Connection: Keep-Alive Connection: Keep-Alive Set-Cookie: MC1=V=3&GUID=7173a7e9bb5444fc88d52c3d302f0cdf; domain=.msn.com; expires=Thu, 17-Sep-2015 20:02:42 GMT; path=/ Set-Cookie: MC1=V=3&GUID=7173a7e9bb5444fc88d52c3d302f0cdf; domain=.msn.com; expires=Thu, 17-Sep-2015 20:02:42 GMT; path=/ Set-Cookie: brdSample=89; domain=.msn.com; expires=Thu, 17-Sep-2015 20:02:42 GMT; path=/ blah, blah <html><head><title>Object moved</title></head><body> <h2>Object moved to <a href="http://br.msn.com/?rd=1&ucc=BR&dcc=BR&opt=0">here</a>.</h2> </body></html> |
Now a request from the IP Next door to it, on the same subnet, produces an entirely different result:
$ curl -i http://www.google.com/
Date: Tue, 17 Sep 2013 19:48:11 GMT Expires: -1 Cache-Control: private, max-age=0 Content-Type: text/html; charset=ISO-8859-1 P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more in fo." Server: gws X-XSS-Protection: 1; mode=block X-Frame-Options: SAMEORIGIN Alternate-Protocol: 80:quic Transfer-Encoding: chunked Proxy-Connection: Keep-Alive Connection: Keep-Alive Set-Cookie: PREF=ID=0e280a6247dbe89f:FF=0:TM=1379447291:LM=1379447291:S=Ws_EoUItgwgQMFLL; expires=Thu, 17-Sep-2015 19:48:11 GMT; path=/; domain=.google.com Set-Cookie: NID=67=MEOnbrr2pgKz8MVKEzIQ2v9f4nkR0o5FXJxbeBRk3GZg6GsfKNRZm9UEHTzcuRF5A6D2kXKa4N6FQnP88fkVLrgDokdOlXvX1Oba2JzC koZ0K0PiACYSiTPCru5eH9C3; expires=Wed, 19-Mar-2014 19:48:11 GMT; path=/; domain=.google.com; HttpOnly <!doctype html><html itemscope="" itemtype="http://schema.org/WebPage"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features ... |
This is a problem when you do searches in which you expect to get localized results, like a search for dinner. Believe me, you see completely different things if Google thinks your IP is in Brazil.
And some services simply shut themselves off. Pandora simply does not work, for instance.
What is going on?
It seems some Geo-location service has incorrectly tagged our IP as Brazil. Not sure how to fix that…
Google does have an IP location change form, but they say it will take a month for them to make the change. That hardly seems like Internet time! https://support.google.com/websearch/contact/ip And what about the others? I don’t see any simple fix for Pandora for instance.
Possible solution?
Someone suggested that the proxy has cached Google’s home page. This is one thing I didn’t test for. However, these proxies are very sophisticated and know to refresh popular pages frequently so I very much doubt that was the cause of the problem.
Workarounds
Google present a link Google.com in the lower right corner of the page which you can click on to pop you to the English-language version of the site, but it’s still not location-aware and treats you more as an English traveler in Brazil when you search for a generic term like restaurant.
You can also go to http://www.google.com/ncr. I personally haven’t tried it, but that’s a tip I just received.
Oct 30 update
Well, finally, after filling out the IP change form twice, and of course keeping all Brazilian users off that proxy, Google has finally deemed fit to declare our proxy really is in the US after all.
It happens again…
Some light usage by a fraction of the user base and I find all proxies are in Brazil yet again. I fill out the form March 19th 2015 and a couple times after that. Now it is May 11th and Google still has not acted. This time I have not kept Brazil users off the proxies, because really this is ridiculous. When users complain I urge them to use duckduckgo.com for its better privacy protections. June 24th update. I’ve been checking every week just about. I’ve filled out Google’s own form about four times in total. So finally after three months, Google set us back to US Enlgish. Without a word to us, of course. Incredible.
Google has gotten big and unresponsive.
Conclusion
Use duckduckgo.com.
References and related
A good web site to learn where your IP is without too much malvertising that you often get with these kinds of things is: https://ipinfo.io/
Intro
Sometimes you want to run a web server that spits back the same page – a maintenance screen – no matter what URI it was accessed by. This is a simple few lines change to accomplish that.
The details
Why might you want to do this? Suppose you had a load balancer for a particular service. And suppose you have to move all the pool members at the same time to a different data center. You’re left with no service at all.
So you can use priority groups and add a lower-priority “maintenance server” to the pool which is not getting moved, and it will answer all queries destined for the service with your desired maintenance page.
I read through the dreadful documentation on Apache (how about this page for a little guilty pleasure reading) and found this way to do it. OK, disclosure time. When you hold a hammer, everything looks like a nail. I had used Mod Rewrite previously so I had some familiarity with it, and I guessed it could be used for this purpose as well. In reality there are probably lots of ways to accomplish this same end goal.
Inside your virtual server:
<VirtualHost *:80> RewriteEngine on RewriteRule ^.* /maintenance.htm [L] ...other usual stuff establishing home dir and permissions ... </VirtualHost> |
My maintenance page
For the record my page looks like this:
<html><head><title>Site Maintenance</title></head> <body> <font face="arial"> <h1>Site Maintenance</h1> <strong> This site is temporarily down.<br> Service will be restored by 2 PM Saturday. </font> </strong> </body> </html> |
Gotchas
The Rewrite rule would need refinement if you wanted to maintain a corporate identity and offer a maintenance page that has images, stylesheets, etc.
And after posting this I ran into another trouble. The actual pages we wanted the message for weren’t getting that message, which was quite mysterious. Instead the message was like this:
Service Temporarily Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later. |
I guess you could do worse, but that’s not my message so where did it comes from and how come I didn’t see it before?
This one is simple enough. I had only tested with random characters, as in
curl -i http://localhost/asdasdd |
I hadn’t tested with one of the actual URIs, many of which end in .jsp. Long story short, I had re-purposed a web server instance that was front-ending a jBoss application server, so it had statements that made it handle JSP pages differently! In particular, there was this:
# JBoss include stuff Include conf/mod-jk.conf |
and this:
JkMountCopy On |
With those lines both commented out it began to throw my maintenance page for all requests, as originally intended. Crisis averted.
Appendix – How to redirect just a specific page
If you want to implement a redirect for just a specific page you can follow this example:
# redirect for test of PAC file - DrJ 4/10/17 RewriteEngine on RewriteRule /proxy.pac http://50.17.188.196/proxy.pac [L] |
Here I am only redirecting requests for the URI proxy.pac and sending it to another server. All other pages remain unaffected.
Conclusion
We have shown how to create a web server that whatever you’ve asked of it, always returns a maintenance page along with HTTP status code of 200. This can be helpful for maintenance or moving situations.
References
My most creative use of URI rewriting is in creating an Apache “redirect factory,” which is described here.
Intro
A couple years ago I heard about conceptual art on public radio. It’s intriguing and begs the question what is art. The example provided was something like this: the artist proscribes via written instructions what the owner should do to realize the physical manifestation of the artwork, such as
– Take a 24″ x 36″ bank canvas
– draw an X near the upper right corner
– draw a horizontal line to the left of the X until the border
That’s it! A proscription like this by a famous artist is actually has value, just like physical art.
OK, so how about conceptual technology?
The details
I wanted a ring tone that was as unobtrusive as possible, and yet still audible because the vibration-only setting on a Samsung just doesn’t deliver enough vibrating energy as compared to my old Blackberry Bold. So in vibrate-only I missed too many calls. And yet i find all ring tones to be offensive in some sense – disruptive to their surrounding environment.
So I finally hit on it. Here is the conceptual technology:
– record the sound of yourself exhaling* for about 3 seconds of sound.
– turn it into a mp3 file
– download it to your phone’s ringtones
– choose it as your ringtone
* Avoid recoding the full breathing cycle of inhaling and exhaling because that’s just going to sound like panting, and that gets weird! And when I say “exhale” I mean more “sigh,” that will be more effective.
That’s it! Is it art or technology?
I’m not 100% sure it will work out long term. For instance, every time I hear someone in the office sigh loudly I check my phone! But I’ve stuck with it so far so it can’t be all bad. Funny thing is, no matter how often I’ve heard it, I’m always caught by surprise when my phone rings with that ringtone! It’s always “What the heck?” before I come to my senses.
For those who want to skip the first step I’ve made my exhaling sounds available for download.
You can right-click on it and do a Save link as… to save it somewhere.
I purposely made it soft as I wanted to avoid drawing too much attention.
Conclusion
Conceptual technology is presented for what might be the most natural ringtone of all – the sound of yourself breathing. When it comes time to turn off your cellphone it may not be necessary. Everyone is allowed to let out a sigh now and then, right?
Appendix
For the curious, I used Audacity to record my exhale sound, which I edited to stretch it to three seconds and made sure it begins with an exhale sound followed by silence. A ringtone seems to be played in a loop by the phone. I found 3 seconds a good length between “rings.”
Some have suggested it sounds mildly obscene or creepy. Oh, well. that wasn’t what I was going for. As I say the jury’s still out…
Intro
Fromm time-to-time I get an unusual ssl error when using curl to check one of my web sites. This post documents the error and how I recovered from it.
The details
I was bringing up a new web site on the F5 BigIP loadbalancer. It was a secure site. I typically use the F5 as an ssl acclerator so it terminates the ssl connection and makes an http connection back to the origin server.
So I tested my new site with curl:
$ curl -i -k https://secure.drj.com/
curl: (52) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104 |
Weird, I thought. I had taken the certificate from an older F5 unit and maybe I had installed it or its private key wrong?
I tested with openssl:
$ openssl s_client -showcerts -connect secure.drj.com:443
...
SSL handshake has read 2831 bytes and written 453 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : RC4-SHA
Session-ID: E95AB5EA2D896D5B3A8BC82F1962FA4A68669EBEF1699DF375EEE95410EF5A0C
Session-ID-ctx:
Master-Key: EC5CA816BBE0955C4BC24EE198FE209BB0702FDAB4308A9DD99C1AF399A69AA19455838B02E78500040FE62A7FC417CD
Key-Arg : None
Start Time: 1374679965
Timeout : 300 (sec)
Verify return code: 20 (unable to get local issuer certificate)
--- |
This all looks pretty normal – the same as what you get from a healthy working site. So the SSL, contrary to what I was seeing from curl, seemed to be working OK.
OK, so, SSL is handled by the F5 itself we were saying. That leaves the origin server. Bingo!
In F5 you have virtual servers and pools. You configure the SSL CERTs and the public-facing IP and the pool on the virtual server. The pool is where you configure your origin server(s).
I had forgotten to associate a default pool with my virtual server! So the F5 had nowhere to go really with the request after handling the initial SSL dialog.
I don’t think the available help for this error is very good so I wanted to offer this specific example.
So I associated a pool with my virtual server and immediately the problem went away.
Case closed.
Conclusion
We solved a very specific case this week and hopefully provided some guidance to others who might be seeing this issue.
References
My favorite openssl commands
Intro
A long-running spam campaign has been bothering me lately. In this post I analyze it from a sendmail perspective and provide a simple script I wrote which helped me fight back.
The details
Let’s have a look see at the July 3rd variant of this spam. Although somewhat different from the previous campaigns in that this did not provide users with a carefully phished email to their inbox, from a sendmail perspective it had a lot of the same features.
So the July 3rd spam was a spoof of Marriott. Look at these from lines. They pretty much shout the pattern out:
Jul 3 14:12:20 drjemgw sm-mta[4707]: r63IA8dJ004707: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-m ta, relay=eu1sysamx113.postini.com [217.226.243.182] Jul 3 14:12:22 drjemgw sm-mta[7088]: r63ICDA7007088: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, r elay=eu1sysamx138.postini.com [217.226.243.227] Jul 3 14:12:23 drjemgw sm-mta[7220]: r63ICIhL007220: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-m ta, relay=eu1sysamx103.postini.com [217.226.243.52] Jul 3 14:12:24 drjemgw sm-mta[7119]: r63ICEp6007119: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm -mta, relay=eu1sysamx112.postini.com [217.226.243.181] Jul 3 14:12:33 drjemgw sm-mta[7346]: r63ICO8H007346: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, relay=eu1sysamx110.postini.com [217.226.243.59] Jul 3 14:12:34 drjemgw sm-mta[7425]: r63ICTsI007425: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm- mta, relay=eu1sysamx107.postini.com [217.226.243.56] Jul 3 14:12:35 drjemgw sm-mta[7387]: r63ICRMP007387: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=s m-mta, relay=eu1sysamx108.postini.com [217.226.243.57] Jul 3 14:12:39 drjemgw sm-mta[1757]: r63I7dfa001757: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, re lay=eu1sysamx138.postini.com [217.226.243.227] Jul 3 14:12:40 drjemgw sm-mta[6643]: r63IBpYm006643: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, relay=e u1sysamx120.postini.com [217.226.243.189] Jul 3 14:12:42 drjemgw sm-mta[4894]: r63IAFug004894: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, relay=eu1sysamx110.postini.com [217.226.243.59] Jul 3 14:12:43 drjemgw sm-mta[7573]: r63ICZJq007573: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mt a, relay=eu1sysamx140.postini.com [217.226.243.229] Jul 3 14:12:45 drjemgw sm-mta[7698]: r63ICfP9007698: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon =sm-mta, relay=eu1sysamx102.postini.com [217.226.243.51] Jul 3 14:12:46 drjemgw sm-mta[7610]: r63ICblx007610: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm- mta, relay=eu1sysamx109.postini.com [217.226.243.58] Jul 3 14:12:50 drjemgw sm-mta[7792]: r63ICl6Y007792: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, rela y=eu1sysamx112.postini.com [217.226.243.181] Jul 3 14:12:51 drjemgw sm-mta[6072]: r63IBGCU006072: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, d aemon=sm-mta, relay=eu1sysamx126.postini.com [217.226.243.195] Jul 3 14:12:51 drjemgw sm-mta[7549]: r63ICYnm007549: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, rela y=eu1sysamx115.postini.com [217.226.243.184] Jul 3 14:12:55 drjemgw sm-mta[7882]: r63ICrUW007882: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, r elay=eu1sysamx139.postini.com [217.226.243.228] Jul 3 14:12:57 drjemgw sm-mta[7925]: r63ICtav007925: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, relay=e u1sysamx110.postini.com [217.226.243.59] Jul 3 14:12:57 drjemgw sm-mta[7930]: r63ICu5c007930: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm- mta, relay=eu1sysamx125.postini.com [217.226.243.194] Jul 3 14:12:58 drjemgw sm-mta[7900]: r63ICsOE007900: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, relay=eu1 sysamx131.postini.com [217.226.243.220] Jul 3 14:13:00 drjemgw sm-mta[7976]: r63ICwmu007976: [email protected], size=0, class=0, nrcpts=1, proto=SMTP, daemon=sm-mta, relay= eu1sysamx127.postini.com [217.226.243.196] |
Of 2035 of these that were sent out, 192 were delivered, meaning got into the users inbox past Postini’s anti-spam defenses. So that’s a pretty high success rate as spam goes. And users get concerned.
Now look at sendmail’s access file which I created shortly after becoming aware of similar phishing of linkedin.com more recently on July 11th:
# 7/11/13 linkedinmail.com DISCARD linkedinmail.net DISCARD linkedinmail.org DISCARD linkedinmail.biz DISCARD linkedin.net DISCARD linkedin.org DISCARD linkedin.biz DISCARD inbound.linkedin.com DISCARD complains.linkedin.com DISCARD emalsrv.linkedin.com DISCARD clients.linkedin.com DISCARD emlreq.linkedin.com DISCARD customercare.linkedin.com DISCARD m.linkedin.com DISCARD enc.linkedin.com DISCARD services.linkedin.com DISCARD amc.linkedin.com DISCARD news.linkedin.com DISCARD |
You get the idea.
What I noticed in these campaigns is a wide variety of subdomains of the domain being phished, with and without “mail” attached to the domain. In particular some rather peculiar-looking subdomains such as complains and emalsrv. So I realized that instead of waiting for me to get the spam, I can constantly comb the log file for these peculiar subdomains. If I come across a new one, voila, it means a new spam campaign has just started! And I can send myself an alert so I can decide – by hand – how best to treat it, knowing it will generally follow the pattern of the recent campaigns.
Now here’s the script I wrote to catch this type of pattern early on:
#!/usr/bin/perl # DrJ, 7/2013 # I keep my sendmail log file here in /maillog/stat.log and cut it daily $sl = "/maillog/stat.log"; # 10000 lines occurs in about eight minutes $DEBUG = 0; $i = 0; $lastlines = "-10000"; $access = "/etc/mail/access"; open(ACCESS,$access) || die "Cannot open $access!!\n"; @access = <ACCESS>; open(SL,"/usr/bin/tail $lastlines $sl|") || die "cannot run tail $lastlines on $sl!!"; print "anti-spam domain: "; while(<SL>){ ($domain) = /from=\w{1,25}@(?:emalsrv|complains)\.([^\.]+)\./; if ($domain) { # test if we already have it on our access table $seenit = 0; foreach $line (@access) { # lazy, inaccurate match, but good enough... $seenit = 1 if $line =~ /$domain/; print "seenit, domain, line: $seenit, $domain, $line\n" if $DEBUG; } if (! $seenit) { $i++; print "$domain\n" if $i == 1; } } } |
I call the script spam-check.pl. I invoke spam-check.pl every couple minutes from HP SiteScope. There I have alerts set up which email me a brief message that includes the new domain that is being phished.
No sooner had I implemented this script than it went off and told me about that linkedin phishing spam campaign! That was sweet.
Recent campaigns
Here is a chronology of spam campaigns which follow the pattern documented above. They seem to cook them up one per day.
5/16 wallmart.com - their misspelling, not mine! 5/29 amazon.com 6/20 adp.com date uncertain ebay.com 7/9 eftps.com 7/10 visabusinessnews.com 7/11 linkedin.com 7/15 ups.com 7/16 twitter.com 7/17 marriott.com mmm.com - this one changed up the pattern a bit 7/18 marriott.com - again ups.com - with somewhat new pattern 7/22 AA.com 7/23-28 a bit more AA.com, a smattering of marriott.com and ebay.com 7/29 tapering off... 7/30 and later spammer seems to have gone on hiatus, or finally been arrested 10/2, they're back staples.com |
One example spam
Here was my phishing spam from 3M which I got yesterday:
From: "3M" <[email protected]> To: DrJ __________________________________________________ This is an automated e-mail. PLEASE DO NOT RESPOND TO THIS EMAIL ACCOUNT. This account is not reviewed for responses. -------------------------------------------------------------------------------- This email is to confirm that on 07/17/2013, 3M's bank (JP Morgan) has debited $15,956.64 from your bank account. If you have any questions, please visit the 3M EIPP Helpline at this link. |
The HTML source for that last line looks like this:
If you have any questions, please visit the 3M EIPP= Helpline <a href=3D"http://vlayaway.com/download/mmm.com.e-marketing.ht= ml?help">at this link.</div> |
When I checked Bluecoat’s K9 webfilter, which I even use at home, the URL in the link, vlayaway… was not rated. I submitted a suggest category, Malicious Sources, and they efficiently assigned it that category within minutes of my submission.
Also, note that the envelope sender of my email differs from the Sender header. The envelope sender was [email protected].
A word about DISCARD vs ERROR
While I’m waiting for more spam of this sort to come in as I write this on July 22nd, I had a brainstorm. Rather than DISCARDind these emails, which doesn’t tip the sender off, it’s probably better to send a 550 error code, which rattles the system a bit more. I think a sending IP with too many of these errors will be temporarily banned by Postini for all their users. So I changed all my DISCARDs. Here is the syntax for one example line:
linkedin-mail.com ERROR:"550 Sender banned. Please use legitimate domain to send email." |
I originally wanted to put the message “No such user,” to try to get the spammer to take that specific recipient off their spam list, but it doesn’t really work in the right way: the error is reported in the context of the sender address, not the recipient address.
Here is the protocol which shows what I am talking about:
$ telnet drj.postini.com 25 Trying 217.136.247.13... Connected to drj.postini.com.. Escape character is '^]'. 220 Postini ESMTP 133 y678_pstn_c6 ready. CA Business and Professions Code Section 17538.45 forbids use of this system for unsolicited electronic mail advertisements. helo localhost 250 Postini says hello back mail from: [email protected] 250 Ok rcpt to: [email protected] 550 5.0.0 [email protected]... Sender banned. Please use legitimate domain to send email. - on relay of: mail from: [email protected] quit 221 Catch you later |
So that – on relay of: mail from: … is added by Postini so it really doesn’t make sense to say No such user in that context.
Conclusion
My satisfaction may be short-lived. But it is always sweet to be on top, even for a short while.
References
For a lighthearted discussion of HP SiteScope, read the comments from this post.
Sendmail is discussed in various posts of mine. For instance, Analyzing the sendmail log, and Obscure tips for sendmail admins.
Intro
I just got a Samsung Galaxy S3 last week. Before long I was wondering how I might use it to access my cloud server if indeed it was at all possible.
The Details
Looking at my other Android devices I decided to install Terminal Emulator. That’s a cute application, providing shell access to the underlying OS of your phone. But it’s fairly useless. You get a shell, but your account id, 10155, has essentially no privileges, and you can’t do much more than ls, cd, ps and top. You don’t have enough privileges to look into interesting directories. So you can’t do anything interesting. There’s also no native ssh so you can’t connect to another host.
To ssh to my Amazon cloud server I got the app ConnectBot. This app shows promise. I was able to connect to my server. I read some of the introductory screens which gave some helpful tips. So I quickly learned how to resize the window. I found 80×39 is a good size in portrait orientation. Yes, the font is tiny, but it is legible. Getting an elusive Esc or Ctrl character isn’t bad at all, just tap the top half of the screen.
So running constantly refreshing screens like top worked out fine.
vi was a problem. It used multi-colors in displaying my code. Comments, in dark blue, are not legible to me. In fact using vi at all on this device with a soft keyboard is quite unnatural. It might be better to use a curses-based editor like pico, though I haven’t yet tried it. But w/ vi, I found I could get rid of the multi-colors by setting the TERM environment variable to vt100. It had been screen screen. Once that was done:
> export TERM=vt100
vi displayed all characters in white, and background in black – quite legible.
Conclusion
It’s a strange world where you can administer a virtual server on a device that fits in the palm of your hand, and achieve fairly powerful effects.
Being a resourceful person, I had alternatives to reach my server. There is a web-based terminal emulator which works surprisingly well. See this post for a description.
connectBot is a native ssh remote terminal app and is actually useable as such on a Samsung Galaxy S3, if your eyes are still good! Pay attention to a just a few usage tips and you’ll be in full control of your server.
References
See this post for the world’s most natural, unobtrusive ringtone.
Intro
Some things we do as IT folk are just embarrassingly stupid in retrospect. This is such a story. I present it in keeping with my overall theme of throwing stuff out there in the hopes that some of it helps someone else facing the same situation.
The details
I love teraterm (from logmett.com). Teraterm plus screen (as in /usr/bin/screen) makes for a great combination when you live and die by the command line.
Actually I have been told I only use a small fraction of teraterm’s capabilities. It is programmable, apparently. I’m a very basic user.
So I had the self-assigned task to switch out a DNS server from an older Solaris OS to Linux. I completed the task months ago and I noticed one small side-effect: certain commands had the effect of washing out the font to just about the same color as the background. For the record my text is black (R,G,B) = (0,0,0) with Attribute Normal and my background is beige (255,255,183). When it’s behaving normally it looks very pleasant and is easy on the eyes.
I noticed when I ran man pages the text was all washed out – just a slightly brighter yellow against the beige background, same story when I ran vi. Comments such as text following # were washed out.
This was the case if I used a docking station. Using the native laptop display, the text was washed out, but not as severely so I could just make it out by straining my eyes.
I played with font color and background color in Teraterm, but didn’t really get anywhere, so I learned to cope. I learned that if I piped the man page to more the text was all-black and I didn’t really lose any functionality. In vi I learned that if I deleted the whitespace before the #, the whole comment became visible, unless it started a line. Kludgy, but it worked and hardly slowed me down – this is after all just one of many, many hosts I was focussed on.
Then it came time to migrate the second and last Solaris DNS server to Linux and I noticed the same thing happening on the new Linux server. What the…?
Previously I wasn’t really even sure when the washed-out problem occurred. This time I had no doubt that it was fine until the OS switch.
That in turn points to some difference in the environment, especially because on my many other Linux sessions I did not have this problem.
> env
shows the environment. By comparing where it was working to where it was not, I zeroed in on this environment variable: TERM.
TERM=vt100 |
where it wasn’t working
and
TERM=screen |
where it was.
I set TERM=screen:
> export TERM=screen
and immediately noticed the display working when running vi. Even multiple colors are displayed.
But actually, hmm, the man pages are still washed out, e.g.,
> man -s1 ls
shows NAME, SYNOPSIS and DESCRIPTION are all yellowed out, as well as all switches! That makes it really difficult to decipher.
Oh, well. This mystery is not completely solved.
My point was going to be that in Solaris the TERM=vt100 made sense – things worked better – and so it was in my .bashrc file. In Linux (SLES) it didn’t make so much sense. No setting for TERM seems to be necessary as the value screen gets auto-defined somehow if you’re using screen.
What I had done was copy my .bashrc file from Solaris to Linux not really thinking about it. That’s what did me in on these two servers.
If I get around to resolving the man pages I’ll revise this post…
2020 update
Still plagued by this issue of washed out colors, I rolled up my sleeves and got it done. Turns out you have to set the Bold font settings separately. I’m trying settings like in this picture.
References
Teraterm used to be available from logmett.com, (2020 update) but is no longer. I’m looking for a link… Here it is: https://osdn.net/projects/ttssh2/releases/
Conclusion
Problems with washed-out colors using teraterm plus screen are resolved. Once again, this was mostly a self-inflicted problem.
Intro
If you’ve read any of my posts you’ll see that I believe sendmail is a technical marvel, but that’ snot to say it’s without its flaws.
One of the annoying things is that the From line and To line are recorded spearately, in defiance of common logic. I present a simple program I wrote to join these lines.
The details
Without further ado, here is the program, which I call combine.pl:
#!/usr/bin/perl # combine lines from stat.log # Copyright work under the Artistic License, http://www.opensource.org/licenses/Artistic-2.0 # DrJ 6/2013 - make more readable based on this format: # Date|Time|Size[Bytes]|Direction|Sender|Recipient|Relay-MTA # # # From= usually has address surrounded by <>, but not always # # input of #Jun 20 10:00:21 drjemgw sm-mta[24318]: r5KE0K1U024318: [email protected], size=5331, class=0, nrcpts=1, msgid=<15936355.7941275772268.JavaMail.easypaynet@Z32C1GBSZ4AN2>, proto=SMTP, daemon=sm-mta, relay=amx139.postini.com [20.26.143.12] #Jun 20 10:00:22 drjemgw sm-mta[24367]: r5KE0K1U024318: to=<[email protected]>, delay=00:00:02, xdelay=00:00:01, mailer=esmtp, pri=125331, relay=drjinternal.com [50.17.188.196], dsn=2.0.0, stat=Sent (r5KE0M6E027784 Message accepted for delivery) # produces #20.6.2013|10:00:21|5331|IN|[email protected]|[email protected]|amx139.postini.com # use Getopt::Std; getopts('s:f'); # -s search_string -f for "full" version of output $DEBUG = 0; print "$relay{$ID}, $lines{$ID}, $sender{$ID}, $size{$ID}\n"; $year = `date +%Y`; chomp($year); while(<>) { chomp; print $_ ."\n" if $DEBUG; # # get ID ($ID) = /\[\d{2,10}\]:\s+(\w+):\s+/; #print "ID: $ID\n"; if ($lines{"$ID"} && / stat=Sent /) { if ($opt_f) { $lines{"$ID"} .= '**to**line**'.$_; } else { ($recip,$relay) = /:\sto=<(.+)>,\s.*\srelay=(\S+)\s/; # there can be multiple recipients listed $recip =~ s/[\<\>]//g; # disposition of email. This needs customization for your situation, but it only determines IN vs OUT vs INTERNAL so it's not critical... # In this example coding we get all our inbound email from postini.com, and outbound mail comes from drjinternal if ($relay{$ID} =~ /postini\.com/) { $disp = "IN"; } else { $disp = $relay =~ /drjinternal/ ? "INTERNAL" : "OUT"; } $lines = "$lines{$ID}|$size{$ID}|$disp|$sender{$ID}|$recip|$relay{$ID}"; if ( ($lines =~ /$opt_s/ || ! $opt_s) && ($sender{$ID} || $recip) ) { $lines .= "|$ID" if $DEBUG; # push @lines, $lines; # why bother? just spit it out immediately print "$lines\n"; } # save memory, hopefully? - can't do this. sometimes we have multiple To lines # undef $relay{$ID}, $lines{$ID}, $sender{$ID}, $size{$ID}; print "$recip\n" if $DEBUG; } } else { if ($opt_f) { $lines{"$ID"} .= '**from**line**'.$_; } else { ($mon,$date,$time,$sender,$size,$relay) = /^(\w+)\s+(\d+)\s+([\d:]+)\s.+\sfrom=<?([^<][^,]*\w)>?,\ssize=(\d+).*relay=(\S+)/; # convert month name to month number $monno = index('JanFebMarAprMayJunJulAugSepOctNovDec',$mon)/3 + 1; # the year is faked - it's not actually recorded in the logs so we assume it's the current year... $lines{$ID} = "$date.$monno.$year|$time"; $size{$ID} = $size; $sender{$ID} = $sender; $relay{$ID} = $relay; print "$mon,$date,$time,$sender,$size,$relay\n" if $DEBUG; } } } # now start matching if ($opt_f) { foreach (@lines) { print $_."\n" } } |
What it does is combine the From and To lines based on the message ID which is unique to a message.
Usage
I usually use it to suck in an entire day’s log (I call my sendmail log stat.log) and grep the output to look for a particular string. For instance today there was a spam blast where ADP’s identity was phished. The sending domains all contained some variant of adp: adp.net, adp.org, adpmail.com, adp.biz, etc. So I wanted to find the answer to the question who’s received any of these ADP phishing emails today? Here’s how you use the program, to do that:
$ combine.pl<stat.log|grep adp.com|more
The input lines look like this:
Jun 20 10:00:21 drjemgw sm-mta[24318]: r5KE0K1U024318: [email protected], size=5331, class=0, nrcpts=1, msgid=<15936355.7941275772268.JavaMail.easypaynet@Z32C1GBSZ4AN2>, proto=SMTP, daemon=sm-mta, relay=amx139.postini.com [27.16.14.22] Jun 20 10:00:22 drjemgw sm-mta[24367]: r5KE0K1U024318: to=<[email protected]>, delay=00:00:02, xdelay=00:00:01, mailer=esmtp, pri=125331, relay=drjinternal.com. [50.17.188.196], dsn=2.0.0, stat=Sent (r5KE0M6E027784 Message accepted for delivery) |
The output from combine.pl looks like this:
20.6.2013|10:00:21|5331|IN|[email protected]|[email protected]|amx139.postini.com |
Yeah, I got that ADP spam by the way…
Conclusion
A useful Perl script has been presented which helps mail admins combine separate output lines into a single entry, preserving the most important meta-data from the mail.
Other interesting sendmail posts are also available here and here.
Intro
The vast majority of time there is no problem doing an FTP to a server behind a firewall protected by Checkpoint’s Firewall-1. But occasionally there is.
The details
The problem I am about to document I think will only occur on a server that has multiple interfaces. I have seen it occur on multiple operating systems, so that doesn’t seem to matter. On the other hand, I have also not seen it on other similar systems, a point which I don’t fully yet understand.
Nevertheless, a work-around is always appreciated, so I provide what I found here, to complete my extensive documentation of problems I’ve encountered and resolved.
Here is a snippet from the FTP session showing the problem:
ftp> cd uploadDirectory 250 CWD command successful. ftp> put smallfile.txt local: smallfile.txt remote: smallfile.txt 229 Entering Extended Passive Mode (|||36945|) 200 EPRT command successful. 421 Service not available, remote server timed out. Connection closed |
And here is the solution:
Enter epsv4 after logon and before any other commands are issued. Problem fixed!
Conclusion
We have shown a way to fix a firewall-related problem that manifests itself during extended passive mode FTPs. Some more research should be done to understand under what circumstances this problem should be expected, but it seems to occur with a Checkpoint Firewall-1 firewall and an FTP server with multiple interfaces.