Month: June 2013

Categories
Admin Linux

The IT Detective Agency: Teraterm’s colors washed out

Intro
Some things we do as IT folk are just embarrassingly stupid in retrospect. This is such a story. I present it in keeping with my overall theme of throwing stuff out there in the hopes that some of it helps someone else facing the same situation.

The details
I love teraterm (from logmett.com). Teraterm plus screen (as in /usr/bin/screen) makes for a great combination when you live and die by the command line.

Actually I have been told I only use a small fraction of teraterm’s capabilities. It is programmable, apparently. I’m a very basic user.

So I had the self-assigned task to switch out a DNS server from an older Solaris OS to Linux. I completed the task months ago and I noticed one small side-effect: certain commands had the effect of washing out the font to just about the same color as the background. For the record my text is black (R,G,B) = (0,0,0) with Attribute Normal and my background is beige (255,255,183). When it’s behaving normally it looks very pleasant and is easy on the eyes.

I noticed when I ran man pages the text was all washed out – just a slightly brighter yellow against the beige background, same story when I ran vi. Comments such as text following # were washed out.

This was the case if I used a docking station. Using the native laptop display, the text was washed out, but not as severely so I could just make it out by straining my eyes.

I played with font color and background color in Teraterm, but didn’t really get anywhere, so I learned to cope. I learned that if I piped the man page to more the text was all-black and I didn’t really lose any functionality. In vi I learned that if I deleted the whitespace before the #, the whole comment became visible, unless it started a line. Kludgy, but it worked and hardly slowed me down – this is after all just one of many, many hosts I was focussed on.

Then it came time to migrate the second and last Solaris DNS server to Linux and I noticed the same thing happening on the new Linux server. What the…?

Previously I wasn’t really even sure when the washed-out problem occurred. This time I had no doubt that it was fine until the OS switch.

That in turn points to some difference in the environment, especially because on my many other Linux sessions I did not have this problem.

> env

shows the environment. By comparing where it was working to where it was not, I zeroed in on this environment variable: TERM.

TERM=vt100

where it wasn’t working

and

TERM=screen

where it was.

I set TERM=screen:

> export TERM=screen

and immediately noticed the display working when running vi. Even multiple colors are displayed.

But actually, hmm, the man pages are still washed out, e.g.,

> man -s1 ls

shows NAME, SYNOPSIS and DESCRIPTION are all yellowed out, as well as all switches! That makes it really difficult to decipher.

Oh, well. This mystery is not completely solved.

My point was going to be that in Solaris the TERM=vt100 made sense – things worked better – and so it was in my .bashrc file. In Linux (SLES) it didn’t make so much sense. No setting for TERM seems to be necessary as the value screen gets auto-defined somehow if you’re using screen.

What I had done was copy my .bashrc file from Solaris to Linux not really thinking about it. That’s what did me in on these two servers.

If I get around to resolving the man pages I’ll revise this post…

2020 update

Still plagued by this issue of washed out colors, I rolled up my sleeves and got it done. Turns out you have to set the Bold font settings separately.  I’m trying settings like in this picture.

References
Teraterm used to be available from logmett.com, (2020 update) but is no longer. I’m looking for a link… Here it is: https://osdn.net/projects/ttssh2/releases/

Conclusion
Problems with washed-out colors using teraterm plus screen are resolved. Once again, this was mostly a self-inflicted problem.

Categories
Internet Mail

Analyzing the sendmail log

Intro
If you’ve read any of my posts you’ll see that I believe sendmail is a technical marvel, but that’ snot to say it’s without its flaws.

One of the annoying things is that the From line and To line are recorded spearately, in defiance of common logic. I present a simple program I wrote to join these lines.

The details
Without further ado, here is the program, which I call combine.pl:

#!/usr/bin/perl
# combine lines from stat.log
# Copyright work under the Artistic License, http://www.opensource.org/licenses/Artistic-2.0
# DrJ 6/2013 - make more readable based on this format:
# Date|Time|Size[Bytes]|Direction|Sender|Recipient|Relay-MTA
#
#
# From= usually has address surrounded by <>, but not always
#
# input of
#Jun 20 10:00:21 drjemgw sm-mta[24318]: r5KE0K1U024318: [email protected], size=5331, class=0, nrcpts=1, msgid=<15936355.7941275772268.JavaMail.easypaynet@Z32C1GBSZ4AN2>, proto=SMTP, daemon=sm-mta, relay=amx139.postini.com [20.26.143.12]
#Jun 20 10:00:22 drjemgw sm-mta[24367]: r5KE0K1U024318: to=<[email protected]>, delay=00:00:02, xdelay=00:00:01, mailer=esmtp, pri=125331, relay=drjinternal.com [50.17.188.196], dsn=2.0.0, stat=Sent (r5KE0M6E027784 Message accepted for delivery)
# produces
#20.6.2013|10:00:21|5331|IN|[email protected]|[email protected]|amx139.postini.com
#
use Getopt::Std;
getopts('s:f'); # -s search_string -f for "full" version of output
$DEBUG = 0;
 
print "$relay{$ID}, $lines{$ID}, $sender{$ID}, $size{$ID}\n";
$year = `date +%Y`;
chomp($year);
while(<>) {
  chomp;
  print $_ ."\n" if $DEBUG;
#
# get ID
  ($ID) = /\[\d{2,10}\]:\s+(\w+):\s+/;
#print "ID: $ID\n";
  if ($lines{"$ID"} && / stat=Sent /) {
    if ($opt_f) {
      $lines{"$ID"} .= '**to**line**'.$_;
    } else {
      ($recip,$relay) = /:\sto=<(.+)>,\s.*\srelay=(\S+)\s/;
# there can be multiple recipients listed
      $recip =~ s/[\<\>]//g;
# disposition of email.  This needs customization for your situation, but it only determines IN vs OUT vs INTERNAL so it's not critical...
# In this example coding we get all our inbound email from postini.com, and outbound mail comes from drjinternal
      if ($relay{$ID} =~ /postini\.com/) {
        $disp = "IN";
      } else {
        $disp = $relay =~ /drjinternal/ ? "INTERNAL" : "OUT";
      }
      $lines = "$lines{$ID}|$size{$ID}|$disp|$sender{$ID}|$recip|$relay{$ID}";
      if ( ($lines =~ /$opt_s/ || ! $opt_s) && ($sender{$ID} || $recip) ) {
        $lines .= "|$ID" if $DEBUG;
#        push @lines, $lines; # why bother?  just spit it out immediately
         print "$lines\n";
      }
# save memory, hopefully? - can't do this. sometimes we have multiple To lines
#      undef $relay{$ID}, $lines{$ID}, $sender{$ID}, $size{$ID};
      print "$recip\n" if $DEBUG;
    }
  } else {
    if ($opt_f) {
      $lines{"$ID"} .= '**from**line**'.$_;
    } else {
      ($mon,$date,$time,$sender,$size,$relay) = /^(\w+)\s+(\d+)\s+([\d:]+)\s.+\sfrom=<?([^<][^,]*\w)>?,\ssize=(\d+).*relay=(\S+)/;
# convert month name to month number
      $monno = index('JanFebMarAprMayJunJulAugSepOctNovDec',$mon)/3 + 1;
# the year is faked - it's not actually recorded in the logs so we assume it's the current year...
      $lines{$ID} = "$date.$monno.$year|$time";
      $size{$ID} = $size;
      $sender{$ID} = $sender;
      $relay{$ID} = $relay;
 
      print "$mon,$date,$time,$sender,$size,$relay\n" if $DEBUG;
    }
  }
}
 
# now start matching
if ($opt_f) {
  foreach (@lines) {
    print $_."\n"
  }
}

What it does is combine the From and To lines based on the message ID which is unique to a message.

Usage
I usually use it to suck in an entire day’s log (I call my sendmail log stat.log) and grep the output to look for a particular string. For instance today there was a spam blast where ADP’s identity was phished. The sending domains all contained some variant of adp: adp.net, adp.org, adpmail.com, adp.biz, etc. So I wanted to find the answer to the question who’s received any of these ADP phishing emails today? Here’s how you use the program, to do that:

$ combine.pl<stat.log|grep adp.com|more

The input lines look like this:

Jun 20 10:00:21 drjemgw sm-mta[24318]: r5KE0K1U024318: [email protected], size=5331, class=0, nrcpts=1, msgid=<15936355.7941275772268.JavaMail.easypaynet@Z32C1GBSZ4AN2>, proto=SMTP, daemon=sm-mta, relay=amx139.postini.com [27.16.14.22]
Jun 20 10:00:22 drjemgw sm-mta[24367]: r5KE0K1U024318: to=<[email protected]>, delay=00:00:02, xdelay=00:00:01, mailer=esmtp, pri=125331, relay=drjinternal.com. [50.17.188.196], dsn=2.0.0, stat=Sent (r5KE0M6E027784 Message accepted for delivery)

The output from combine.pl looks like this:

20.6.2013|10:00:21|5331|IN|[email protected]|[email protected]|amx139.postini.com

Yeah, I got that ADP spam by the way…

Conclusion
A useful Perl script has been presented which helps mail admins combine separate output lines into a single entry, preserving the most important meta-data from the mail.

Other interesting sendmail posts are also available here and here.

Categories
Admin Network Technologies

Extended Passive Mode FTP through Checkpoint Firewall

Intro
The vast majority of time there is no problem doing an FTP to a server behind a firewall protected by Checkpoint’s Firewall-1. But occasionally there is.

The details
The problem I am about to document I think will only occur on a server that has multiple interfaces. I have seen it occur on multiple operating systems, so that doesn’t seem to matter. On the other hand, I have also not seen it on other similar systems, a point which I don’t fully yet understand.

Nevertheless, a work-around is always appreciated, so I provide what I found here, to complete my extensive documentation of problems I’ve encountered and resolved.

Here is a snippet from the FTP session showing the problem:

ftp> cd uploadDirectory
250 CWD command successful.
ftp> put smallfile.txt
local: smallfile.txt remote: smallfile.txt
229 Entering Extended Passive Mode (|||36945|)
200 EPRT command successful.
421 Service not available, remote server timed out. Connection closed

And here is the solution:

Enter epsv4 after logon and before any other commands are issued. Problem fixed!

Conclusion
We have shown a way to fix a firewall-related problem that manifests itself during extended passive mode FTPs. Some more research should be done to understand under what circumstances this problem should be expected, but it seems to occur with a Checkpoint Firewall-1 firewall and an FTP server with multiple interfaces.

Categories
Admin Network Technologies

The IT Detective Agency: trouble with wireless at home

Intro
I don’t usually have the luxury of writing about a mystery I’ve solved right out of my own home, but there finally is one that I got got to the bottom of recently – poor WiFi performance.

The details
Considering that I deal with this stuff for a living, I have a thread-bare setup at home. After my company-issued router’s WiFi began to work unreliably, I resuscitated an old Linksys wireless router, WRK54G V2. Superficially it seemed to work. But we weren’t very demanding of it.

It eventually seemed to be the case, as visitors mentioned, that streaming videos does not work through wireless. This was hard for me to check, with my broken-down, aging equipment. I have a desktop which always freezes and crashes is you play any Youtube video. And a Netbook which kind of worked better, but its peculiarity is that its ethernet interface doesn’t work. Wirelessly, its version of Flash was too old and insecure for Firefox, and attempts to update Flash using WiFi in turn were unsuccessful.

In general the Linksys router, as I eventually realized, seemed to initially serve up large downloads ok, but then at some point during the download, things begin to crawl and you are left with a download that proceeds at 10 kbit/s or something ridiculously slow like that.

Providing mixed evidence is a Sony BlueRay player. using WiFi it could sort of manage to show a HuluPlus TV episode. You might have to be patient at times while it’s loading, but we did get through a full episode of Grey’s Anatomy recently.

After more complaints I decided enough is enough. It seemed as though my WiFi was the most likely suspect, sifting through the mixed evidence. I perhaps waited so long because who’d think they’d be dealing with two bad WiFi routers from two totally different vendors?

So hedging my bets, I didn’t go all out with a new Gbit router. I reached back in time a little and got a refurbished Cisco 1200E wireless-N router. It was only $28 from Amazon. But before buying it, I read the comments and got one idea about routers: sometimes they need to be rebooted!

This is pretty funny, really, because it is probably apparent to any homeowner, and here I am, a specialist, missing this point. You see with Cisco enterprise-class gear you almost never have to reboot to fix a problem. These things run uninterrupted for not only weeks and months at a time, years at a time is also not at all uncommon. Same for some Unix servers. So from my perspective rebooting is something for consumer devices running Microsoft OSes!

So, before rebooting the Linksys to see if that would cure it, I ran a Ping to Google’s DNS server (very easy to remember its IP) from a CDM window:

> ping -t 8.8.8.8

I didn’t preserve the output, but it wasn’t pretty. It was something like this:

Pinging 8.8.8.8 with 32 bytes of data:
Reply from 8.8.8.8: bytes=32 time=51ms TTL=56
Reply from 8.8.8.8: bytes=32 time=369ms TTL=56
Reply from 8.8.8.8: bytes=32 time=51ms TTL=56
Reply from 8.8.8.8: bytes=32 time=1204ms TTL=56
Reply from 8.8.8.8: bytes=32 time=284ms TTL=56
...

51 msec – fine. But round-trip times much greater than that? That’s not right.

So I hopefully reboot the Linksys router and re-run the test on the Netbook:

Pinging 8.8.8.8 with 32 bytes of data:
Reply from 8.8.8.8: bytes=32 time=51ms TTL=56
Reply from 8.8.8.8: bytes=32 time=51ms TTL=56
Reply from 8.8.8.8: bytes=32 time=51ms TTL=56
Reply from 8.8.8.8: bytes=32 time=50ms TTL=56
Reply from 8.8.8.8: bytes=32 time=51ms TTL=56
...

Much more consistent.

Try a Youtube video from Firefox. Nope, need to update Flash. Update Flash. Nope – download times out and kicks me out.

So I’ve accomplished nothing in rebooting in terms of results that matter.

That’s when I decided to check out of Amazon with that refurbished router.

Aside about Wireless-N
Given my ancient equipment, I was concerned that Wireles-N routers might not be compatible with my wireless radios, which would only support G. Is it backwards compatible? Yes. Some quick research showed that and my own experience confirmed it.

Conclusion
The setup of the router was pretty straightforward although it froze at some point just after I set the wireless password. It helps to have done this a zillion times before. At that point I observed what my default gateway was and hit it as a web site URL. Guessed the admin password incorrectly a zillion times, until I tried the wireless password as the admin password, and, wham – I was in and happily configuring away…

More importantly, I went to that Netbook, updated Flash. No problems. Ran a Youtube video. No problems. Ran a speedtest.net test (which wouldn’t even run before this). Numbers look as good as my wired connection: 6 mbit download, 0.6 mbit upload.

Last test is to see where the speed maxes out within my home network. I plan to hit my Raspberry Pi web server to test this and will provide results as soon as they are available.

Conclusion to the conclusion
So I really was cursed with two bad wireless routers. Sometimes using 10-year-old equipment is really not worth the $30 saved in deferred spending. Read product reviews on Amazon to get hints about real issues others have faced.
To be continued…