Author: john

Categories
Admin Internet Mail SLES

The IT Detecive Agency: emails began piling up this week, no obvious cause

Intro
Today I had my choice of problems I could highlight, but I like this one the best. Our mail server delivers email to a wide variety of recipients. All was going well and it ran pretty much unattended until this week when it didn’t go so well. Most emails were getting delivered, but more and more were starting to pile up in the queues. This is the story of how we unraveled the mystery.

The details
It’s best to work from examples I think. I noticed emails to me.com were being refused delivery as well as emails to rnbdesign.com. The latter is a smaller company so we heard from them the usual story that we’re the only ones who can’t send to them.

So I forced delivery with verbose logging. I’m running sendmail, so that looks like this:

> sendmail -qRrnbdesign.com -Cconfig_file -v

That didn’t work out, producing a no route to host type of error. I did a DNS lookup by hand. That showed one set of results, while sendmail was connecting to an entirely different IP address. How could that be??

I was at a loss so I do what I do when I’m desperate: strace. That looks like this:

> strace -f sendmail -qRrnbdesign.com -Cconfig_file -v > /tmp/strace 2>&1

That produced 12,000 lines of output. All the system calls that the process and any of its forked processes invoke. Is that too much to comb through by hand? No, not at all, not when you begin to see the patterns.

I pored over the trace, not knowing what most of it meant, but looking for especially any activity regarding networking and DNS. Around line 6,000 I found it. There was mention of nscd.

For the unaware the use of nscd (nameserver caching daemon) might seem innocent enough, or even good-intentioned. What could be wrong with caching frequently used DNS results? The only issue is that it doesn’t work right! nscd derives from UC Berkeley Unix code and has never been supported. I didn’t even like it when I was running SunOS. It caches the DNS queries but ignores TTLs. This is fatal for mail servers or just about anything you can think of, especially on servers that are infrequently booted as mine are.

I stopped nscd right away:

> service nscd stop

and re-ran the sendmail queue runner (same command as above). The rnbdesign.com emails flowed out instantly! Soon hundreds of stuck emails were flushed out.

Of course for good measure nscd had to be removed from the startup sequence:

> chkconfig nscd off

An IT pro always keeps unsolved mysteries in his mind. This time I knew I also had in hand the solution an earlier-documented mystery about email to paladinny.com.

Conclusion
nscd might show up in your SLES or OpenSuse server. I strongly suggest to disable it before you wind up with old DNS values and an extremely hard-to-debug issue.

Case closed!

Categories
Admin Linux Raspberry Pi

Ssh access to your Raspberry Pi from anywhere

Editor’s 2017 note: Lots of great alternatives are discussed in the Comments section.

Intro
I’ve done a couple things with my Raspberry Pi. There’s this post on setting it up without a monitor, keyboard or mouse, and this post on using it to monitor power and Internet connection at my home.

I eventually realized that the Pi could be accessed from anywhere, with one big assumption: that you have your own hosted server somewhere on the Internet that you can ssh to from anywhere. This is the same assumption I used in describing the power monitor application.

The details
I can’t really take any credit for originality here. I just copied what I saw in another post. My only contribution is in realizing that the Pi makes a good platform to do this sort of thing with if you are running it as a server like I am.

What you can do is to create a reverse ssh tunnel. I find this easier and probably more secure than opening up ssh (inbound) on your home router and mapping that to the Pi. So I’m not going to talk about that method.

First ssh log in to your Pi.

From that session ssh to your hosted server using syntax like this:

> ssh −f −N −R 10000:localhost:22 username@ip_address_of_your_hosted_sever

You can even log out of your Pi now – this reverse tunnel will stay*.

Now to access your Pi from “anywhere,” log into your server like usual, then from that session, login to your Pi thusly:

> ssh −p 10000 pi@localhost

That’s it! You should be logged on after supplying the password to the pi account.

*Except that in my experience the reverse tunnel does not stay! It’s staying up less than two hours.

But I think the approach is sound.

Feb 15th Update
This is a case of RTFM. That same web page I cited above has the necessary settings. I needed to have them on the Pi. It didn’t help when I put them on my Amazon server. Here they are repeated:

TCPKeepAlive yes
ClientAliveInterval 30
#ClientAliveCountMax 30
ClientAliveCountMax 99999
GatewayPorts yes
AllowTcpForwarding yes

This goes into the /etc/ssh/sshd_config file. Make sure you don’t have these mentioned a second time in that file.

With these settings my reverse tunnel has been up all day. It’s a real permanent tunnel now!

Security note
Make sure you modify the default passwords to your Pi before attempting this. You’re potentially exposing your whole home network in creating a reverse tunnel like this so you really have to be careful.

Conclusion
You can use your Raspberry Pi to create a reverse tunnel tht allows you to access it from anywhere, assuming you have a cooperating hosted server on the Internet as a mutual meeting point for the ssh sessions. Exercise caution, though, as you are opening up your Home network as well.

Currently the tunnel doesn’t stay up for very long – perhaps an hour or so. If I find a way to extend that I’ll revise this post.

References
Having trouble ssh’ing to your Ras Pi under any conditions? This article explains how to get past one common cause of this problem.

Categories
Internet Mail Linux Perl Raspberry Pi

Raspberry Pi phone home

Intro
In this article I described setting up my Raspberry Pi without ever connecting a monitor keyboard and mouse to it and how I got really good performance using an UHS SD card.

This article represents my first real DIY project on my Pi – one of my own design. My faithful subscribers will recall my post after Hurricane Sandy in which I reacted to an intense desire to know when the power was back on by creating a monitor for that situation. It relied on extremely unlikely pieces of infrastructure. I hinted that it may be possible to use the Raspberry Pi to accomplish the same thing.

I’ve given it a lot of thought and assembled all the pieces. Now I have a home power/Internet service monitor based on my Pi!

This still requires a somewhat unlikely but not impossible combination of infrastructure:
– your own hosted server in the cloud
– ability to send emails out from your cloud server
– access log files on your cloud server are rolled over regularly
– your Pi and your cloud server are in the same time zone
– Raspberry Pi which is acting as a server (meaning you are running it 24×7 and not rebooting it and fooling with it too much)
– a smart phone to receive alert emails or TXT messages

I used my old-school knowledge of Perl to whip something up quickly. One of this years I have to bite the bullet and learn Python decently, but it’s hard when you are so comfortable in another language.

The details
Here’s the concept. From your Pi you make regular “phone home” calls to your cloud server. This could use any protocol your server is listening on, but since most cloud servers run web servers, including mine, I phone home using HTTP. Then on your cloud server you look for the phone home messages. If you don’t see one after a certain time, you send an alert to an email account. Then, once service – be it power or Internet connectivity – is restored to your house, your Pi resumes phoning home and your cloud monitor detects this and sends a Good message.

I have tried to write minimalist code that yet will work and even handle common error conditions, so I think it is fairly robust.

Set up your Pi
On your Pi you are “phoning home” to your server. So you need a line something like this in your crontab file:

# This gets a file and leaves a timestamp behind in the access log
* * * * * /usr/bin/curl --connect-timeout 30 http://yourcloudserver.com/raspberrypiPhoneHome?`perl -e 'print time()'` > /dev/null 2>&1

Don’t know what I’m talking about when I say edit your crontab file?

> export EDITOR=vi
> crontab -e

That first line is only required for fans of the vi editor.

That part was easy, right? That will have your server “phone home” every minute, 24×7. But we need an aside to talk about time on the Pi.

Getting the right time on the Raspberry Pi
This monitoring solution assumes Ras Pi and home server are in the same time zone (because we kept it simple). I’ve seen at least a couple of my Raspberry Pi’s where the time zone was messed up so I need to document the fix.

Run the date command
$ date

Sat Apr 29 17:10:13 EDT 2017

Now it shows it is set for EDT so the timezone is correct. At first it showed something like UTC.

Make sure you are running ntp:
$ ntpq ‐p

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+time.tritn.com  63.145.169.3     2 u  689 1024  377   78.380    2.301   0.853
-pacific.latt.ne 68.110.9.223     2 u  312 1024  377  116.254   11.565   5.864
+choppa.chieftek 172.16.182.1     3 u  909 1024  377   65.430    4.185   0.686
*nero.grnet.gr   .GPS.            1 u  106 1024  377  162.089  -10.357   0.459

You should get results similar to those above. In particular the jitter numbers should be small, or at least less than 10 (units are msec for the curious).

If you’re missing the ntpq command then do a

$ sudo apt-get install ntp

Set the correct timezone with a

$ sudo dpkg-reconfigure tzdata

and choose Americas, then new York, or whatever is appropriate for your geography. The Internet has a lot of silly advice on this point so I hope this clarifies the point.

Note that you need to do both things. In my experience time on Raspberry Pis tends to drift so you’ll be off by seconds, which is a bad thing. ntp addresses that. And having it in the wrong timezone is just annoying in general as all your logs and file times etc will be off compared to how you expect to see them.

On your server
Here is the Perl script I cooked up. Some modifications are needed for others to use, such as email addresses, access log location and perhaps the name and switches for the mail client.

So without further ado. here is the monitor script:

#!/usr/bin/perl
# send out alerts related to Raspberry Pi phone home
# this is designed to be called periodically from cron
# DrJ - 2/2013
#
# to test good to error transition,
# call with a very small maxDiff, such as 0!
use Getopt::Std;
getopts('m:d'); # maximum allowed time difference
$maxDiff = $opt_m;
$DEBUG = 1 if $opt_d;
unless (defined($maxDiff)) {
  usage();
  exit(1);
}
# use values appropriate for your situation here...
$mailsender = '[email protected]';
$recipient = '[email protected]';
$monitorName = 'Raspberry Pi phone home';
# access line looks like:
# 96.15.212.173 - - [02/Feb/2013:22:00:02 -0500] "GET /raspberrypiPhoneHome?136456789 HTTP/1.1" 200 455 "-" "curl/7.26.0"
$magicString = "raspberrypiPhoneHome";
# modify as needed for your situation...
$accessLog = "/var/log/drjohns/access.log";
#
# pick up timestamp in access file
$piTime = `grep $magicString $accessLog|tail -1|cut -d\? -f2|cut -d' ' -f1`;
$curTime = time();
chomp($time);
$date = `date`;
chomp($date);
# your PID file is somewhere else. It tells us when Apache was started.
# you could comment out these next lines just to get started with the program
$PID = "/var/run/apache2.pid";
($atime,$mtime,$ctime) = (stat($PID))[8,9,10];
$diff = $curTime - $piTime;
print "magicString, accessLog, piTime, curTime, diff: $magicString, $accessLog, $piTime, $curTime, $diff\n" if $DEBUG;
print "accessLog stat. atime, mtime, ctime: $atime,$mtime,$ctime\n" if $DEBUG;
if ($curTime - $ctime < $maxDiff) {
  print "Apache hasn't been running long enough yet to look for something in the log file. Maybe next time\n";
  exit(0);
}
#
$goodFile = "/tmp/piGood";
$errorFile = "/tmp/piError";
#
# Think of it as state machine. There are just a few states and a few transitions to consider
#
if (-e $goodFile) {
  print "state: good\n" if $DEBUG;
  if ($diff < $maxDiff) {
    print "Remain in good state\n" if $DEBUG;
  } else {
# transition to error state
    print "Transition from good to error state at $date, diff is $diff\n";
    sendMail("Good","Error","Last call was $diff seconds ago");
# set state to Error
    system("rm $goodFile; touch $errorFile");
  }
} elsif (-e $errorFile) {
  print "state: error\n" if $DEBUG;
  if ($diff > $maxDiff) {
    print "Remain in error state\n" if $DEBUG;
  } else {
# transition to good state
    print "Transition from error to good state at $date, diff is $diff\n";
    sendMail("Error","Good","Service restored. Last call was $diff seconds ago");
# set state to Good
    system("rm $errorFile; touch $goodFile");
  }
} else {
  print "no state\n" if $DEBUG;
  if ($diff < $maxDiff) {
    system("touch $goodFile");
    sendMail("no state","Good","NA") if $DEBUG;
    print "Transition from no state to Good at $date\n";
# don't send alert
  } else {
    print "Remain in no state\n" if $DEBUG;
  }
}
####################
sub sendMail {
($oldState,$state,$additional) = @_;
print "oldState,state,additional: $oldState,$state,$additional\n" if $DEBUG;
$subject = "$state : $monitorName";
open(MAILX,"|mailx -r \"$mailsender\" -s \"$subject\" $recipient") || die "Cannot run mailx $mailsender $subject!!\n";
print MAILX qq(
$monitorName is now in state: $state
Time: $date
Former state was $oldState
Additional info: $additional
 
- sent from pialert program
);
close(MAILX);
 
}
###############################
sub usage {
  print "usage: $0 -m <maxDiff (seconds)> [-d (debug)]\n";
}

This is called from my server’s crontab. I set it like this:

 Call monitor that sends an alert if my Raspberry Pi fails to phone home - DrJ 2/13
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /home/drj/pialert.pl -m 300 >> /tmp/pialert.log

My /tmp/pialert.log file looks like this so far:

Transition from no state to Good at Wed Feb  6 12:10:02 EST 2013
Apache hasn't been running long enough yet to look for something in the log file. Maybe next time
Apache hasn't been running long enough yet to look for something in the log file. Maybe next time
Transition from good to error state at Fri Feb  8 10:55:01 EST 2013, diff is 420
Transition from error to good state at Fri Feb  8 11:05:02 EST 2013, diff is 1

The last two lines result from a test I ran – i commented out the crontab entry on my Pi to be absolutely sure it was working.

The error message I got in my email looked like this:

Subject: Error : Raspberry Pi phone home
 
Raspberry Pi phone home is now in state: Error 
Time: Fri Feb  8 10:55:01 EST 2013
Former state was Good
Additional info: Last call was 420 seconds ago
 
- sent from pialert program

Why not use Nagios?
Some will realize that I replicated functions that are provided in Nagios, why not just hang my stuff off that well-established monitoring software? I considered it, but I wanted to stay light. I think my approach, while more demanding of me as a programmer, keeps my server unburdened by running yet another piece of software that has to be understood, debugged, maintained and patched. If you already have it, fine, by all means use it for the alerting part. I’m sure it gives you more options. For an approach to installing nagios that makes it somewhat manageable see the references.

A few words about sending mail
I send mail directly from my cloud server, I have no idea what others do. With Amazon, my elastic IP was initially included in blacklists (RBLs), etc, so I really couldn’t send mail without it being rejected. they have procedures you can follow to remove your IP from those lists, and it really worked. Crucially, it allowed me to send as a TXT message. Just another reason why you can’t really beat Amazon hosting (there was no charge for this feature).

And sending TXT messages
I think most wireless providers have an email gateway that allows you to send a TXT message (SMS) to one of their users via email (SMTP) if you know their cell number. For instance with Verizon the formula is

Verizon

[email protected]

AT&T
cell-number@txt.att.net.

T-Mobile
cell-number@tmomail.net

Conclusion
We have assembled a working power/Internet service monitor as a DIY project for a Raspberry Pi. If you want to use your Pi for a lot of other things I suggest to leave this one for your power monitor and buy another – they’re cheap (and fun)!

I will now know whenever I lose power – could be any minute now thanks to Nemo – and when it is restored, even if I am not home (thanks to my SmartPhone). See in my case my ISP, CenturyLink, is pretty good and rarely drops my service. JCP&L, not so much.

Admittedly, most people, unlike me, do not have their own cloud-hosted server, but maybe it’s time to get one?

References
Open Monitoring Distribution (OMD) makes installing and configuring nagios a lot easier, or so I am told. It is described here.
I’ve gotten my mileage out of the monitor perl script in this post: I’ve recently re-used it with modifications for a similar situation except that the script is being called by HP SiteScope, and, again, a Raspberry Pi is phoning home. Described here.

Categories
Admin DNS IT Operational Excellence

The IT Detective Agency: since when can a powered off PC do dynamic DNS updates?

Intro
The IT Detectives are back after a short lull during which no great mysteries needed expert resolution – you knew that situation couldn’t last too long. The following tale was relayed to me, I unfortunately cannot claim to have been any help whatsoever. The details have been somewhat obscured in this retelling.

The details
One of our DNS servers at drjohns was busy fielding lots and lots of DDNS updates. Good, right? No, not so. Because our employee PCs are all configured to not do this very thing. In Windows 7 drilling down into the advanced DNS settings you have a Checkbox for Register this connection’s addresses in DNS. And that is unchecked. So although we use DHCP, the PCs shouldn’t be sending their DDNS updates. Yet they were. In fact at one point a considerable amount of bandwidth was being eaten up with these unwanted updates, so we had to investigate and act. But where to begin?

Word finally got around to one of our PC experts who I guess probably had his suspicions. He suggested the following test:

turn the PC off and look for DDNS updates on the DNS server

Amazingly, that’s exactly what we found to be the case – DDNS updates coming from a powered off PC. The DDNS updates did not always go to the same DNS server. The chosen DNS server seemed randomly chosen, but they all were drjohns DNS servers.

A Wireshark examination of a trace (taken by a network engineer) showed lots of Dynamic Update SOA drj.com. I looked at the trace and found that that was just a title given by Wireshark for what was happening, and not a very accurate one. If you expand the packet you saw inside of it that (mostly) it was a workstation trying to register its A record on the DNS server (a DDNS update). It wasn’t literally trying to change the SOA record for the zone though that might have been the logical result of updating its A record.

What the power-off test showed to our subject-area expert is that Intel vPro was responsible for these DDNS updates. Wait, you ask, what the heck is vPro? We didn’t know either. As I understand it, it’s an additional Intel chip that some business-class laptops (e.g., DELL Latitude) might include that permits more and better remote management, allowing perhaps even some hardware diagnostics to occur.

So let’s go back to that test. Note that I said PC powered off, I did not say disconnected from the network! Powered-off-but-network-connected produces the DDNS update, powered-off-and-disconnected – no update, of course (Hey, it’s not magic going on here!).

So the solution, obvisouly, is to turn off DDNS in vPro. We thought it was off, but maybe not. We expect and hope this to the solution, but a few more days will be needed before this all plays out and we know for sure.

Conclusion
I better hold off on any conclusion until our premise is confirmed! But one feeling I have is that sometimes you have to ingratiate yourself to the right people because no one person has all the answers!

Categories
Admin First Robotics

Interactive Frisbee Trajectory

Intro
The FIRST FRC challenge for 2013, Ultimate Ascent, involves shooting heavy flying discs (sturdy Frisbees) into goals. The physics of the equations of motion have been studied and published. I’ve created an interactive web page which allows you to vary some of the initial conditions to see how the trajectory is affected.

The details
Go here for the web page.

For last year’s challenge, foam balls were to be shot into a basketball hoop, so similar equations of motion applied. Here is that page.

Categories
Admin DNS Internet Mail SLES

Strange problem with email to paladinny.com

Intro
This is probably the most obscure of all postings I will ever do – it’s really just opening up my private journal to the Internet, which helps me when I need to recall how I fixed something.

So the story is that I’m having trouble sending email to anyone in the domain paladinny.com, and I just couldn’t figure out why.

The details
With my sendmail config I finally rolled up my sleeves, and did some debugging, even though I am pressed for time. Start up our sendmail debugging session:

> sendmail -Cconfig_file.cf -bt -d35.9

This produces a lot of blah, blah, configuration settings, blah, blah, and finally a sort of sendmail debugging shell. So let’s test a good “normal” domain:

> 3,0 [email protected]

canonify           input: test @ gmail . com
Canonify2          input: test < @ gmail . com >
Canonify2        returns: test < @ gmail . com . >
canonify         returns: test < @ gmail . com . >
parse              input: test < @ gmail . com . >
Parse0             input: test < @ gmail . com . >
Parse0           returns: test < @ gmail . com . >
ParseLocal         input: test < @ gmail . com . >
ParseLocal       returns: test < @ gmail . com . >
Parse1             input: test < @ gmail . com . >
Mailertable        input: < gmail . com > test < @ gmail . com . >
Mailertable        input: gmail . < com > test < @ gmail . com . >
Mailertable      returns: test < @ gmail . com . >
Mailertable      returns: test < @ gmail . com . >
SmartTable         input: test < @ gmail . com . >
SmartTable       returns: test < @ gmail . com . >
MailerToTriple     input: < > test < @ gmail . com . >
MailerToTriple   returns: test < @ gmail . com . >
Parse1           returns: $# esmtp $@ gmail . com . $: test < @ gmail . com . >
parse            returns: $# esmtp $@ gmail . com . $: test < @ gmail . com . >

and then this problem domain:

> 3,0 [email protected]

canonify           input: test @ paladinny . com
Canonify2          input: test < @ paladinny . com >
Canonify2        returns: test < @ paladinny . no-ip . biz . >
canonify         returns: test < @ paladinny . no-ip . biz . >
parse              input: test < @ paladinny . no-ip . biz . >
Parse0             input: test < @ paladinny . no-ip . biz . >
Parse0           returns: test < @ paladinny . no-ip . biz . >
ParseLocal         input: test < @ paladinny . no-ip . biz . >
ParseLocal       returns: test < @ paladinny . no-ip . biz . >
Parse1             input: test < @ paladinny . no-ip . biz . >
Mailertable        input: < paladinny . no-ip . biz > test < @ paladinny . no-ip . biz . >
Mailertable        input: paladinny . < no-ip . biz > test < @ paladinny . no-ip . biz . >
Mailertable        input: paladinny . no-ip . < biz > test < @ paladinny . no-ip . biz . >
Mailertable      returns: test < @ paladinny . no-ip . biz . >
Mailertable      returns: test < @ paladinny . no-ip . biz . >
Mailertable      returns: test < @ paladinny . no-ip . biz . >
SmartTable         input: test < @ paladinny . no-ip . biz . >
SmartTable       returns: test < @ paladinny . no-ip . biz . >
MailerToTriple     input: < > test < @ paladinny . no-ip . biz . >
MailerToTriple   returns: test < @ paladinny . no-ip . biz . >
Parse1           returns: $# esmtp $@ paladinny . no-ip . biz . $: test < @ paladinny . no-ip . biz . >
parse            returns: $# esmtp $@ paladinny . no-ip . biz . $: test < @ paladinny . no-ip . biz . >

I have to look more into what Canonify2 does. But this gives me an idea: force the mailertable to handle paladinny . no-ip . biz the way I want it to, namely:

paladinny.no-ip.biz relay:barracuda.cblconsulting.com

because in DNS my DNS server returns this funny result:

> dig mx paladinny.com

; <<>> DiG 9.6-ESV-R7-P3 <<>> mx paladinny.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17559
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0
 
;; QUESTION SECTION:
;paladinny.com.                 IN      MX
 
;; ANSWER SECTION:
paladinny.com.          351     IN      CNAME   paladinny.no-ip.biz.
 
;; AUTHORITY SECTION:
no-ip.biz.              60      IN      SOA     nf1.no-ip.com. hostmaster.no-ip.com. 2052775595 600 300 604800 600
 
;; Query time: 30 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Jan 18 08:53:49 2013
;; MSG SIZE  rcvd: 121

whereas Google’s public DNS says this, which looks like the intended result:

> dig mx paladinny.com @8.8.8.8

; <<>> DiG 9.6-ESV-R7-P3 <<>> mx paladinny.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3749
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
 
;; QUESTION SECTION:
;paladinny.com.                 IN      MX
 
;; ANSWER SECTION:
paladinny.com.          1800    IN      MX      10 barracuda.cblconsulting.com.
 
;; Query time: 236 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Jan 18 08:55:42 2013
;; MSG SIZE  rcvd: 71

So at least we know where that odd paladinny.no-ip.biz comes from, sort of. It comes from my nameserver, but where it got that answer from I have no idea. It doesn’t come from the authoritative nameservers:

> dig mx paladinny.com @dns1.name-services.com.

; <<>> DiG 9.6-ESV-R7-P3 <<>> mx paladinny.com @dns1.name-services.com.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45704
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
 
;; QUESTION SECTION:
;paladinny.com.                 IN      MX
 
;; ANSWER SECTION:
paladinny.com.          1800    IN      MX      10 barracuda.cblconsulting.com.
 
;; Query time: 82 msec
;; SERVER: 98.124.192.1#53(98.124.192.1)
;; WHEN: Fri Jan 18 08:59:50 2013
;; MSG SIZE  rcvd: 71

A CNAME is not an MX record, so why my nameserver is returning an answer (ANSWER: 1)when queried for the MX record when all it thinks it has is a CNAME seems to be an out-and-out error.

And putting the resolved name in the mailertable is also not normal. Normally you put the domain itself, as in:

paladinny.com relay:barracuda.cblconsulting.com

and of course that’s the first thing I tried, but it has no effect whatsoever.

February Update and Conclusion
The mystery was solved when a whole bunch of email deliveries started failing on my system and I was forced to do some serious debugging. Long story short my SLES system was regrettably running nscd, the nameserver caching daemon. I didn’t even bother to check paladinny.com. So many other things cleared up when I killed it I’m sure it was the cause of the paladinny.com issue as well. This is all described in this post.

Categories
Apache Linux Network Technologies Perl Raspberry Pi

Getting started on my Raspberry Pi

Intro
The Raspberry Pi computer is an awesome idea. Its performance is surprisingly good as well, as I will show below. Available packages? Not so impressive. I share some old X-windows tricks which will allow you to bring up the GUI without ever using the HDMI port.

The details
My Methodology
I was too lazy to set up an HDMI console plus keyboard and mouse. I’m more a server guy anyways so I’m more interested in what I can accomplish from a command prompt. And this also makes getting started that much easier. I had burned the Raspbian Wheezy image to a super-fast SD card (more on that below) the day that my Pi came in the mail. I attached power and ethernet, booted it up, guessed the IP it acquired by running some PINGs, did an ssh using the pi/raspberry user and Bingo! I was in. It couldn’t be easier. How I tested GUI applications without a console is explained further down below.

First Impressions
It feels fast.

Packages
Not much seems to be there by default – no apache, not many X utilities. There is a lame X browser called x-www-browser. I thought this is Debian, right? So we can just start downloading Debian packages, like Firefox. Wrong! It doesn’t work that way. There’s no Firefox, Safari, Chrome or Opera! It does come pre-loaded with curl, however, ha, ha.

No, the Raspbian FAQ explains why this is. It’s rather complicated. I guess the compiler works though I haven’t tested it yet. So I suppose you could compile packages from their source code.

The x-terminal-emulator is pretty decent, however.

If it comes with a web server, I didn’t notice. So I quickly checked for the availability of apache. It’s available. Then installed it:

> sudo apt-get install apache2

That worked out well. It installed it and the packages it depended on and even launched it, and it all felt fairly peppy. See the suggested fix further down if this gives you errors. The default HTML DOCROOT is /var/www. I accessed it locally:

> curl localhost

And a welcome message displayed. A good start.

Where’s the rest of my 16 GB SD card gone to?

Original disk layout:

pi@raspberrypi:~$ df -k
Filesystem     1K-blocks    Used Available Use% Mounted on
rootfs           1804128 1492908    219572  88% /
/dev/root        1804128 1492908    219572  88% /
devtmpfs          224436       0    224436   0% /dev
tmpfs              44900     204     44696   1% /run
tmpfs               5120       0      5120   0% /run/lock
tmpfs              89780       0     89780   0% /run/shm
/dev/mmcblk0p1     57288   16872     40416  30% /boot

Layout after raspi-config:

pi@raspberrypi:~$ df -k
Filesystem     1K-blocks    Used Available Use% Mounted on
rootfs          15251960 1494852  12982544  11% /
/dev/root       15251960 1494852  12982544  11% /
devtmpfs          224436       0    224436   0% /dev
tmpfs              44900     196     44704   1% /run
tmpfs               5120       0      5120   0% /run/lock
tmpfs              89780       0     89780   0% /run/shm
/dev/mmcblk0p1     57288   16872     40416  30% /boot

Whew! That was easy. All 16 GB accounted for and actively used.

Was it worth it to buy that UHS SD card?
I didn’t want a sluggish server, so I paid a couple bucks more and bought a 16 GB SD UHS (ultra high speed) card for my “disk,” not knowing whether or not the Pi had the muscle to put it to work.

A quick aside about SD cards
I did a quickie self-education on this topic. Most SD cards are rated by class, so a class 4 SD card can do 4 MB/sec I/O, and a class 10 card can do at least 10 MB/sec. Faster still are the UHS SD cards. My Sandisk, which only cost about $19, is rated for 45 MB/sec I/O. A great write-up on this topic specifically for Raspberry Pi is: Raspberry Pi SD Card Speed Test – Raspberry Pi

diskSpec.pl benchmark (higher numbers are better)
1333 file creation/destruction operations per second – Raspberry Pi with UHS SD card
6666 file creation/destruction operations per second – EBS volume on small image running CentOS in Amazon cloud
26000 file creation/destruction operations per second – high-end HP server (G7 DL380) running SLES 11

I think I provided the source for this simple Perl program I wrote, diskSpec.pl. It creates a file, writes a random number into it, then deletes it – that all counts as one operation. Here it is:

#!/usr/bin/perl
# DrJ, 1/2000
# Test disk I/O
$DIR = $ARGV[0];
chdir($DIR);
$t0 = time();
while(1) {
  $ran = rand();
  open(FILE,"&gt;$ran") || die "Cannot open file $ran in directory $DIR!!\n";
  print FILE $ran;
  close(FILE);
  unlink($ran);
  $cnt++;
  if ($cnt % 20000 == 0) {
    $rate = $cnt / (time() - $t0) ;
    print "File creation/desctruction rate: $rate\n";
  }
}

DrJ 2017 Note: The notes below are historical and does not seem to work at all for the Raspberry Pi 3 loaded with NOOBS. In NOOBS you select your OS to install. You can’t ssh to it. I know. I just tried! Even after you install Raspbian Wheezy, you still can’t access it via ssh until you enable the ssh daemon with raspi-condfig.

How to get the GUI working without a console
I have this feeling that many people trying out the Pi won’t have the faintest idea how X windows works, unlike us Unix old-timers. It’s fun to put 20-year-old lessons to work on something new. Like I said I’m lazy and didn’t feel the need to set up an actual console to the thing. I used some old X features to allow me to launch specific X-windows applications that are pre-loaded on the device, and display them on my PC. How?

On a Windows PC you install Cygwin. Then launch the XWin Server. You ssh to your pi. How do you know its IP the first time? Guess! It picks it up via DHCP, so start PINGing around the range where your other devices are numbered. My PC is 192.168.5.12/24, my pi was 192.168.5.16. Maybe you have a bunch of devices responding to PING and are unsure which is which? Your MAC table is your friend. Here’s mine:

C:\Documents and Settings&gt;arp -a
 
Interface: 192.168.5.12 --- 0x2
  Internet Address      Physical Address      Type
  192.168.5.1           00-14-f6-e0-c0-4c     dynamic
  192.168.5.16          b8-27-eb-dd-21-02     dynamic
  192.168.5.99          00-90-a9-bb-3d-76     dynamic

arp displays the MAC table with the IP-to-physical (MAC) address correspondence. So most Pi’s will have a MAC address whose beginning is similar to b8-27-eb. A quick aside. Does the MAC address follow the board (SOC) or the SD Card? The board – I tested this with a friend’s SD Card.

You login with the pi/raspberry.

Then set your DISPLAY environment variable:

> export DISPLAY=192.168.5.12:0

Most of your X applications begin with the letter “x,” so enter

> x<tab><tab>

to see a display of available programs like this:

xapian-config        xdg-screensaver      xkbevd               xpdf.real            xxd
xarchiver            xdg-settings         xkbprint             xprop                xz
xargs                xdpyinfo             xkbvleds             x-session-manager    xzcat
xauth                xdriinfo             xkbwatch             xsubpp               xzcmp
xdg-desktop-icon     xev                  xkill                xtables-multi        xzdiff
xdg-desktop-menu     xfd                  xlsatoms             x-terminal-emulator  xzegrep
xdg-email            xfontsel             xlsclients           xvinfo               xzfgrep
xdg-icon-resource    xinit                xlsfonts             x-window-manager     xzgrep
xdg-mime             xkbbell              xmessage             xwininfo             xzless
xdg-open             xkbcomp              xpdf                 x-www-browser        xzmore

Actually I don’t know how many of these are X. But at least a few are.

Start an xterm in Cygwin. In the xterm window, give permission to the Pi to use it as its Xserver:

> xhost +

Now in the Pi shell (ha, ha), type:

> x-terminal-emulator

and you should see the colorful terminal emulator on your PC in a few seconds. this is a true GUI application. You similarly launch the x-www-browser. Don’t forget to background your X-windows in the Pi shell:

<Ctrl-Z>
> bg

so you can use the one window to launch multiple X windows.

Another example the book Programming the Raspberry Pi has is the Python interactive development environment. I reasoned from the screen shots that idles3 would also be an X application – hey, they don’t have to start with the letter x – and indeed it is!

Want the whole ball of wax, a complete console? I just figured this one out by taking an educated guess:

> x-session-manager

and you will see the complete GUI on your PC! Cool, huh?

Want to get rid of the last thing you backgrounded, like, say, that x-session-mnager which has taken over your PC?! Type

> fg
<Ctrl-C>

and it will be killed.

How to get the GUI working without a console, Method 2
The above steps look a little daunting? Even I don’t want to install cygwin on my new PC. There is an alternative which can suffice for light usage.

On the Pi install a vnc server:

$ sudo apt-get install tightvncserver

Launch it:

$ vncserver

The first time only it will ask you to set up a password. Might as well make it raspberry like everything else we do on the Pi.

Then install a VNC client on your PC (Or Macbook). I use RealVNC.

Launch your VNC client and connect to your Pi’s IP address (which you need to know) + the display number, like this:

192.168.0.100:1

For a Pi at IP 192.168.0.100 in which the vncserver started display 1. Normally it will be display 1, but I guess it might be display 0.

Don’t launch vncserver more than once! You don’t want a bunch of those running and dragging on performance.

Anyways, that’s it! You should see the Pi’s GUI on your PC, but it might seem a wee bit small.

Setting a static IP
If you’re going to use the Pi more as a server like I am, I think it’s a good idea to give it a static IP. What I did is to edit /etc/network/interfaces. Mine now looks like this:

Nagios can be installed! That's pretty cool - it's a sophisticated network monitoring utility.

Get a decent browser
The web browsers that come with the Pi are horrible. Midori? Seriously? I found you can get Firefox, but the downside is that it’s sloooww. But at least it works. The secret is that it’s not called Firefox. Instead:

$ sudo apt-get install iceweasel

Yes, it’s iceweasel, not Firefox, in Debian Linux. Go figure.

My cool transparent case
I recommend to get a case. I got the one with the best reviews. It’s kind of expensive, about $20, but worth it. It’s practically a work-of-art. Clear, the PC board fits snugly. I put it in my pocket and showed it around to my friends, feeling it was well protected, and yet also a sight to behold the first time. I even has a thoughtful light guide so the LEDs look beautiful as their light follows the rectangular opening to open air. I never had this much fun in show-and-tell! I just pulled the Pi wrapped in its case from my shirt pocket and amazed those around me. So go ahead and splurge. Anyways some of the cheaper cases look just that. Here is what I bought:

Helping a friend out with his Pi
So I dutifully take my friend’s Pi home and offer to install a web server. What did I do wrong? Well, duh, I could have just taken his SD Card home and plugged it into my Pi case! That concept takes some getting used to! We all have the same hardware. Our SD cards – our disk – are what make one Pi different from another.

So I followed my own blog post to recall some things. This Pi also had a MAC address beginning with the same six characters.

The apache2 installation did not work out, however. What to do? Well, I eventually read the darn output from running it. It suggests to try this:

> sudo apt-get update

So I ran that, figuring it could do no harm. Then I re-ran

> sudo apt-get install apache2

and this time the install actually worked!

Reading a flash drive
I was curious to see if you could stick a flash drive in the thing and just read it. I didn’t think so since I thought it would be formatted for NTFS. But if you have the GUI running and bring up a file manager, I’ll be darned if it doesn’t just work. I noticed the drive is mounted as /media/Cruzer (my flash drive has the brand name Cruzer).

If you don’t launch the file manager, I think you can still work with it as follows:

$ sudo mkdir /media/Cruzer; sudo mount /dev/sda1 /media/Cruzer

Then when you’re all done and before you remove it:

$ sudo umount /media/Cruzer

So that’s pretty cool. You can create tar archives on the flash drive, plug it into someone else’s Pi and untar it, etc, just like on Windows.

Conclusion
Raspberry Pi is respectable as a computer. It will be a lot of fun to explore for the hobbyist.

References

Raspberry Pi SD Card Speed Test – Raspberry Pi – a great discussion of the various speeds of Micro SD cards and how to measure yours
Go here for my next project – using your Raspberry Pi to monitor your home’s power or Internet connection.
Interested in networking? A lot of useful tips can be found in this posting describing how to turn your Pi into a router.
Realvnc.com distributes realVNC viewers for various platforms.
How about a Raspberry Pi-driven digital photo frame? I describe an approach in this article.
Brief Nagios for Raspberry Pi writeup.

Categories
First Robotics

The best Labview shortcuts and tips

Intro
Labview is used for real-time instrument controls and in First Robotics. It is a purely visual programming environment, which is very strange for me. I am a complete Labview amateur. If you’ve read some of my other posts you might see a pattern emerging! But nevertheless I know a few things. Then I use the program so infrequently I forget them. And Labview is the kind of program where you can spend a lot of time that could have been saved if you just knew a better way. So here is the better way…

The most useful Labview shortcuts
– – undo the last thing you did. I think you can undo more than just the last by repeating
– – remove broken wires
– – to toggle between Front panel and Block diagram
– – context-sensitive help
– – run the VI
– and hold-click, then drag mouse and release – do this between two nodes to push them (and everything else) apart. Works horizontally and vertically!
– Select (a node), -click, move mouse nearby and release – to create a duplicate of a node
– Right-click a node, then choose similar items from this function palette – to quickly find a related function

Tips
– Rt-click on wire, then Clean wire – to make it more tidy
– Click while dragging wire, then move the mouse at a right angle – to create a right-angle bend in the wire just where you want it

Labview mentor training
https://decibel.ni.com/content/docs/DOC-21180 This is a really helpful link for mentors as well as students.


The best resource I have found
Team 358 has put together the best resource for FRC teams I have found – every basic- and intermediate-level difficulty thing you’ll need is explained on this one page. Well worth a bookmark. Thank you team 358! http://team358.org/files/programming/ControlSystem2009-/LabVIEW/. (2/2016 update) But see the references for a working link to this page…

That’s about all I know!

References and related
Team 358’s web site is often not working. I have found a working copy on the Wayback machine of their great Labview tutorial: Team 358 Labview tutorial on archive.org

Categories
Admin

Setting up your SSL-VPN Portal on Juniper SA – a Guide for the rest of us

Intro
I was consulting for a company which got a shiny brand new Juniper SA2500. I’ve heard about such things. In fact I heard about it in the days when Netscreen themselves presented it to us. I was impressed at the time and still am. But how to get the thing working when the documentation is pretty formidable? This company was getting off an old Nortel Alteon platform which I was more familiar with. It, too, was very difficult to master at first. you gotta get their paradigm, I suppose. Once you have it, it’s a walk in the park.

The problem we are addressing
Provide secure access to selected external users to selected Intranet sites. Can’t be that hard, right? That’s the bread and butter for these boxes after all. Ha! 1164 pages of documentation and the basic concepts are almost completely lost in the forest.

The details
So I assembled all the important information here in one short, sweet example. I can’t speak for anyone else, but I myself will use this as a reference from now on!

Open source zealots will say that this SA is an overpriced waste of money since everything we’re doing can be done by apache or Squid or tools like that. We’re just setting up reverse proxy with some fancy authentication after all. But I do genuinely think this network appliance is an appropriate expense for a mid- to large-sized company with It staff too busy to investigate “free” alternatives.

To mix things up a bit – everyone has something unique about their use-case, right? – the same SA2500 is to be used for both Network Connect VPN-style connections as well as this SSL-VPN poprtal application. We will associate the SSL-VPN portal with a VIP on the external interface and give it the name ssl-vpn-portal-vip1.drj.com.

Assumptions
I assume there is RSA authentication, though that doesn’t matter much, and that LDAP group-based authorization will be used.

Other assumptions: the desired users, let’s say the company sales force, are in an LDAP group SalesExtranet_access; and that they need access to Intranet sites https://sales-intranet.drj.com/ and http://quality-asurance.johnthedoctor.net/QA/home.htm; my LDAP authentication server has been set up and called LDAP-extranet.

So here goes.

Authentication
Auth. Servers|LDAP-extranet
click Server Catalog
click Search
in filter field: cn=SalesExtranet_access, then Search
click on group name, then OK. It should now be available for roles.

Role Definition
User Roles|New User Role…
Name: SSL-VPN-SalesExtranet-role
Access: Web.
Save Changes

Resource Definition
Resource Profiles|Web|New profile
Name: SSL-VPN-SalesExtranet-profile
Description: Profile for the SalesExtranet_access group. – DrJ 12/3/12
base URL: https://sales-intranet.drj.com/
Save and Continue
add SSL-VPN-SalesExtranet-role

Add 2nd web site w/ different domain to this role
Resource Profile|Web. Pick SSL-VPN-SalesExtranet-profile, then Duplicate
Rename this profile to SSL-VPN-SalesExtranet-profile-QA or similar
Enter correct URL for this new profile: http://quality-asurance.johnthedoctor.net/QA/home.htm

Role to Group Mapping
For me this is the key step that ties all the pieces together, and is a little non-intuitive
User Realms|SSL-VPN-portal-RSA-realm|Role Mapping
New Rule
Rule based on Group membership.
Update
Name: SSL-VPN-SalesExtranet-roleMapping
If User is a member of SalesExtranet_Access. Add
…then assign these roles: SSL-VPN-SalesExtranet-role
Save Changes

Nota Bene
SSL-VPN-portal-page sign-in page has already been set up. This is a one-time setup. This is the place where you can present a custom message to your users.
Signing-in|Sign-in Policies is where we establish the URL for this portal, e.g.,
Put ssl-vpn-portal-vip1.drj.com first in the list of user URLs in the Signing in screen

Certificate Loading
It’s fairly obvious that to stuff a certificate into the SA you go to
Configuration|Certificates|Device Certificates
because where else would it be?
But it’s less obvious that you need to click on the certificate name to associate it with your vlan and possibly external interface. You can even associate the certificate with both an internal virtual interface as well as an external virtual interface. This kind of flexibility is what drew me to the Juniper solution.

Appendix A
Finding the LDAP Group in Novell eDirectory
This threw me for a loop at first. the single most important thing in this case is that group membership is indicated by the attribute member when you’re setting up the LDAP authentication server, even though when you conduct an ldap query you’d think it would be groupMembership. But I think this could vary according to your particular installation.

Categories
Admin Network Technologies

The IT Detective Agency: Internet Explorer cannot display https web page, part II

Intro
They say when it rains it pours. As a harassed It support specialist its tempting to lump all similar-looking problems that come across your desk at the same time in the same bucket. This case shows the shallowness of that way of thinking, for that’s exactly what happened in this case. It occurred exactly when this case was occurring in which a user had an issue displaying a secure web site. That other case was described here.

The details
I was doing some work for a company when they came to me with a problem one user at HQ was having accessing an https web site. Everything else worked fine. The same web page worked fine from other sites.

Since I had helped them set up their proxy services I was keen to prove that the proxy wasn’t at fault. The user removed the proxy settings – still the error occurred. The desktop support got involved. I suggested a whole battery of tests because this was a weird one.
– what if you access via proxy?
– what if you take that laptop and access it from a VPN connection?
– what if you access another site which uses the same certificate-issuer?
– what if you access the host, but using http?
– can you PING the server?
– can you telnet to that server on port 443?
– what if you access the webpage by IP address?
– what if accessed via Firefox?
– etc.
The error, by the way, was

Internet Explorer cannot display the webpage

What you can try:
Diagnose Connection Problems

The answers came back like this:
– what if you access via proxy? It works!
– what if you take that laptop and access it from a VPN connection? It works.
– what if you access another site which uses the same certificate-issuer? It works.
– what if you access the host, but using http? It works.
– can you PING the server? Yes.
– can you telnet to that server on port 443? Yes
– what if you access the webpage by IP address? It does not work.
– Firefox? I don’t remember the answer to this.

Most of that thinking behind those tests is pretty obvious. Why so many tests? The networking group is kind of crotchety and understaffed, so they really wanted to eliminate all other possibilities first. And at one point I thought it could have been a desktop issue. Why would the network allow some of these packets through but not others when it doesn’t run a firewall? The desktop did have A-V and local firewall, after all.

So I didn’t have proof or even a good idea. Time to take out the big guns. We ran a trace on the server with tcpdump while the error occurred. The server was busy so we had to be pretty specific:

> tcpdump -s 1580 -w /tmp/drjcap.cap -i any host 10.19.79.216 and port 443

What do we see? We see the initial handshake go through just fine. Then a client Hello, then a server Hello, followed by a Certificate sent from the server to the client. The Certificate packet kept getting re-sent because there was no ACK to it from the client. So it’s beginning to smell like a dropped packet somewhere. I was hot on the trail but I decided I needed even stronger proof. We arranged to do simultaneous traces on both PC and server to compare the two results.

On the PC we had to install Wireshark. Actually I think Microsoft also has a utility to do traces but I’ve never used it. What did we find? Proof that there was indeed a dropped packet. That server certificate? Never received by the PC (which is acting as the SSL client). But why?

I had noticed one other funny thing about the trace comparisons. The server hello packet left as a packet of length 1518 bytes, but arrived, apparently, as two packets, one of length 778 bytes, the other of 770 bytes, i.e., it was fragmented. That should be OK. I don’t think the don’t fragment flag was set (have to check this). But it got me to wondering. Because the server certificate packet was also on the large side – 1460 bytes.

Regardless, I had enough ammo to go to the networking group with, which I did. It was like, “Oh yeah, our Telecom provider (let’s call it “CU”) implemented GETVPN around that time.” And further discussion revealed suspected other problems related to this change with MTU, etc. In fact they dredged up this nice description of the pitfalls that await GET VPN implementations from the Cisco deployment guide:

3.8 Designing Around MTU Issues Because of additional IPsec overhead added to each packet, MTU related issues are very common in IPsec deployments, and MTU size becomes a very important design consideration. If MTU value is not carefully selected by either predefining the MTU value on the end hosts or by dynamically setting it using PMTU discovery, the network performance will be impacted because of fragmentation and reassembly. In the worst case, the user applications will not work because network devices might not be able to handle the large packets and are unable to fragment them because of the df-bit setting. Some of the scenarios which can adversely affect traffic in a GET VPN environment and applicable mitigation techniques are discussed below. LAN MTU of 1500 – WAN MTU 44xx (MPLS) In this scenario, even after adding the 50-60 byte overhead, MTU size is much less than the MTU of the WAN. The MTU does not affect GET VPN traffic in any shape or form. LAN MTU of 1500 – WAN MTU 1500 In this scenario, when IPsec overhead is added to the maximum packet size the LAN can handle (i.e. 1500 bytes) the resulting packet size becomes greater than the MTU of the WAN. The following techniques could help reduce the MTU size to a value that the WAN infrastructure can actually handle. Manually setting a lower MTU on the hosts By manually setting the host MTU to 1400 bytes, IP packets coming in on the LAN segment will always have 100 extra bytes for encryption overhead. This is the easiest solution to the MTU issues but is harder to deploy because the MTU needs to be tweaked on all the hosts. TCP Traffic Configure ip tcp adjst-mss 1360 on GM LAN interface. This command will ensure that resulting IP packet on the LAN segment is less than 1400 bytes thereby providing 100 bytes for any overhead. If the maximum MTU is lowered by other links in the core (e.g. some other type of tunneling such as GRE is used in the core), the adjust-mss value can be lowered further. This value only affects TCP traffic and has no bearing on the UDP traffic. 3.8.1.1 Host compliant with PMTU discovery For non-TCP traffic, for a 1500 packet with DF bit set, the GM drops the packet and send ICMP message back to sender notifying it to adjust the MTU. If sender and the application is PMTU compliant, this will result in a packet size which can successfully be handled by WAN. For example, if a GM receives a 1500 byte IP packet with the df-bit set and encryption overhead is 60 bytes, GM will notify the sender to reduce the MTU size to 1440 bytes. Sender will comply with the request and the resulting WAN packets will be exactly 1500 bytes

I don’t claim to understand all those scenarios, but it shouts pretty loudly. Watch out for problems with packets of size 1400 bytes or larger!

Why would they want GET VPN? To encrypt the HQ communication over the WAN. So the idea is laudable, but the execution lacking.

Case: understood but not yet closed! The fix is not yet in…

To be continued…