Category: Raspberry Pi

Categories
Admin Network Technologies Raspberry Pi

Use Raspberry Pi to explore mDNS

Intro
I am confounded by the Bonjour field on my d-Link DCS-931L IP webcam. I should be able to use it to see my desired hostname, but it doesn’t take. Why?

The details
Having a Raspberry Pi on the same network I realized I could at least learn definitively whether or not my new name was being taken, or what the old name was.

You install avahi-discover to do that:

$ sudo apt-get install avahi-discover

Those who follow my blog will realize I am big on Linux command-line, not so much on GUIs. I mention it because unfortunately avahi-discover only works in the GUI. Not having a console I actually had to fire up vncserver and use my vncviewer on my PC! Then I could launch avahi-discover from a terminal window running on the GUI.

The extra fuss was just a few steps anyway, and well worth it.

avahi-discover broke down my home network and all the discovered devices in a very orderly fashion, e.g., the webcam appeared under web servers.

And what did I learn? Indeed, my name had not “taken” for some reason. So the system-supplied name was there instead. For the record that is

dcs931le1a6.local

And testing it:

$ ping dcs931le1a6.local

did indeed show me that it was resolvable by that name form my local network. My PC could reach it by that name as well. I tied to name it DCS-931L-BALL, and I know someone else who did this successfully, and I had even done it in the past, but it was just not taking it this time.

References and related
mDNS is multicast DNS. It’s designed for home networks. It’s pretty common from wjhat I see, yet largely unknown since It people do not encounter it in enterprise environments. As usual Wikipedia has a good article on it.
Superimposing crosshairs on a webcam image.

Categories
CentOS DNS Linux Network Technologies Raspberry Pi Security Web Site Technologies

Roll your own dynamic DNS update service

Intro
I know my old Cisco router only has built-in support for two dynamic DNS services, dyndns.org and TZO.com. Nowadays you have to pay for those, if even they work (the web site domain names seem to have changed, but perhaps they still support the old domain names. Or perhaps not!). Maybe this could be fixed by firmware upgrades (to hopefully get more choices and hopefully a free one, or a newer router, or running DD-WRT. I didn’t do any of those things. Being a network person at heart, I wrote my own. I found the samples out there on the Internet needed some updating, so I am sharing my recipe. I didn’t think it was too hard to pull off.

What I used
– GoDaddy DNS hosting (basically any will do)
– my Amazon AWS virtual server running CentOS, where I have sudo access
– my home Raspberry Pi
– a tiny bit of php programming
– my networking skills for debugging

As I have prior experience with all these items this project was right up my alley.

Delegating our DDNS domain from GoDaddy
Just create a nameserver record from the domain, say drj.com, called, say, raspi, which you delegate to your AWS server. Following the example, the subdomain would be raspi.drj.com whose nameserver is drj.com.


DNS Setup on an Amazon AWS server
/etc/named.conf

//
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//
 
options {
//      listen-on port 53 { 127.0.0.1; };
//      listen-on port 53;
        listen-on-v6 port 53 { ::1; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        allow-query     { any; };
        recursion no;
 
        dnssec-enable yes;
        dnssec-validation yes;
        dnssec-lookaside auto;
 
        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";
 
        managed-keys-directory "/var/named/dynamic";
};
 
logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};
 
zone "." IN {
        type hint;
        file "named.ca";
};
 
include "/etc/named.rfc1912.zones";
include "/var/named/dynamic.conf";
include "/etc/named.root.key";

/var/named/dynamic.conf

zone "raspi.drj.com" {
  type master;
  file "/var/named/db.raspi.drj.com";
// designed to work with nsupdate -l used on same system - DrJ 10/2016
// /var/run/named/session.key
  update-policy local;
};

/var/named/db.raspi.drj.com

$ORIGIN .
$TTL 1800       ; 30 minutes
raspi.drj.com      IN SOA  drj.com. postmaster.drj.com. (
                                2016092812 ; serial
                                1700       ; refresh (28 minutes 20 seconds)
                                1700       ; retry (28 minutes 20 seconds)
                                1209600    ; expire (2 weeks)
                                600        ; minimum (10 minutes)
                                )
                        NS      drj.com.
$TTL 3600       ; 1 hour
                        A       125.125.73.145

Named re-starting program
Want to make sure your named restarts if it happens to die? nanny.pl is a good, simple monitor to do that. Here is the version I use on my server. Note the customized variables towards the top.

#!/usr/bin/perl
#
# Copyright (C) 2004, 2007, 2012  Internet Systems Consortium, Inc. ("ISC")
# Copyright (C) 2000, 2001  Internet Software Consortium.
#
# Permission to use, copy, modify, and/or distribute this software for any
# purpose with or without fee is hereby granted, provided that the above
# copyright notice and this permission notice appear in all copies.
#
# THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH
# REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
# AND FITNESS.  IN NO EVENT SHALL ISC BE LIABLE FOR ANY SPECIAL, DIRECT,
# INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
# LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
# OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
# PERFORMANCE OF THIS SOFTWARE.
 
# $Id: nanny.pl,v 1.11 2007/06/19 23:47:07 tbox Exp $
 
# A simple nanny to make sure named stays running.
 
$pid_file_location = '/var/run/named/named.pid';
$nameserver_location = 'localhost';
$dig_program = 'dig';
$named_program =  '/usr/sbin/named -u named';
 
fork() && exit();
 
for (;;) {
        $pid = 0;
        open(FILE, $pid_file_location) || goto restart;
        $pid = <FILE>;
        close(FILE);
        chomp($pid);
 
        $res = kill 0, $pid;
 
        goto restart if ($res == 0);
 
        $dig_command =
               "$dig_program +short . \@$nameserver_location > /dev/null";
        $return = system($dig_command);
        goto restart if ($return == 9);
 
        sleep 30;
        next;
 
 restart:
        if ($pid != 0) {
                kill 15, $pid;
                sleep 30;
        }
        system ($named_program);
        sleep 120;
}

The PHP updating program myip-update.php

<?php
# DrJ: lifted from http://pablohoffman.com/dynamic-dns-updates-with-a-simple-php-script
# but with some security improvements
# 10/2016
# PHP script for very simple dynamic DNS updates
#
# this script was published in http://pablohoffman.com/articles and
# released to the public domain by Pablo Hoffman on 27 Aug 2006
 
# CONFIGURATION BEGINS -------------------------------------------------------
# define password here
$mysecret = 'myBigFatsEcreT';
# CONFIGURATION ENDS ---------------------------------------------------------
 
 
$ip = $_SERVER['REMOTE_ADDR'];
$host = $_GET['host'];
$secret = $_POST['secret'];
$zone = $_GET['zone'];
$tmpfile = trim(`mktemp /tmp/nsupdate.XXXXXX`);
 
if ((!$host) or (!$zone) or (!($mysecret == $secret))) {
    echo "FAILED";
    unlink($tmpfile);
    exit;
}
 
$oldip = trim(`dig +short $host.$zone @localhost`);
if ($ip == $oldip) {
    echo "UNCHANGED. ip: $ip\n";
    unlink($tmpfile);
    exit;
}
 
echo "$ip - $oldip";
 
$nsucmd = "update delete $host.$zone A
update add $host.$zone 3600 A $ip
send
";
 
$fp = fopen($tmpfile, 'w');
fwrite($fp, $nsucmd);
fclose($fp);
`sudo nsupdate -l $tmpfile`;
unlink($tmpfile);
echo "OK ";
echo `date`;
?>

In the above file I added the “sudo” after awhile. See explanation further down below.

Raspberry Pi requirements
I’ve assumed you can run your Pi 24 x 7 and constantly and consistently on your network.

Crontab entry on the Raspberry Pi
Edit the crontab file for periodically checking your IP on the Pi and updating external DNS if it has changed by doing this:

$ crontab ‐e
and adding the line below:

# my own method of dynamic update - DrJ 10/2016
0,10,20,30,40,50 * * * * /usr/bin/curl -s -k -d 'secret=myBigFatsEcreT' 'https://drj.com/myip-update.php?host=raspi&zone=drj.com' >> /tmp/ddns 2>&1

A few highlights
Note that I’ve switched to use of nsupdate -l on the local server. This will be more secure than the previous solution which suggested to have updates from localhost. As far as I can tell localhost updates can be spoofed and so should be considered insecure in a modern infrastructure. I learned a lot by running nsupdate -D -l on my AWS server and observing what happens.
And note that I changed the locations of the secret. The old solution had the secret embedded in the URL in a GET statement, which means it would also be embedded in every single request in the web server’s access file. That’s not a good idea. I switched it to a POSTed variable so that it doesn’t show up in the web server’s access file. This is done with the -d switch of curl.

Contents of temporary file
Here are example contents. This is useful when you’re trying to run nsupdate from the command line.

update delete raspi.drj.com A
update add raspi.drj.com 3600 A 51.32.108.37
send


Permissions problems
If you see something like this on your DNS server:

$ ll /var/run/named

total 8
-rw-r--r-- 1 named www-data   6 Nov  6 03:15 named.pid
-rw------- 1 named www-data 102 Oct 24 09:42 session.key

your attempt to run nsupdate by your web server will be foiled and produce something like this:

$ /usr/bin/nsupdate ‐l /tmp/nsupdate.LInUmo

06-Nov-2016 17:14:14.780 none:0: open: /var/run/named/session.key: permission denied
can't read key from /var/run/named/session.key: permission denied

The solution may be to permit group read permission:

$ cd /var/run/named; sudo chmod g+r session.key

and make the group owner of the file your webserver user ID (which I’ve already done here). I’m still working this part out…

That approach doesn’t seem to “stick,” so I came up with this other approach. Put your web server user in sudoers to allow it to run nsupdate (my web server user is www-data for these examples):

Cmnd_Alias     NSUPDATE = /usr/bin/nsupdate
# allow web server to run nsupdate
www-data ALL=(root) NOPASSWD: NSUPDATE

But you may get the dreaded

sudo: sorry, you must have a tty to run sudo

if you manage to figure out how to turn on debugging.

So if your sudoers has a line like this:

Defaults    requiretty

you will need lines like this:

# turn of tty requirements only for www-data user
Defaults:www-data !requiretty

Debugging
Of course for debugging I commented out the unlink line in the PHP update file and ran the
nsupdate -l /tmp/nsupdate.xxxxx
by hand as user www-data.

During some of the errors I worked through that wasn’t verbose enough so I added debugging arguments:

$ nsupdate ‐D ‐d ‐l /tmp/nsupdate.xxxxx

When that began to work, yet when called via the webserver it wasn’t working, I ran the above command from within PHP, recording the output to a file:

...
`sudo nsupdate -d -D -l $tmpfile > /tmp/nsupdate-debug 2>&1`

That turned out to be extremely informative.

Conclusion
We have shown how to combine a bunch of simple networking tools to create your own DDNS service. The key elements are a Raspberry Pi and your own virtual server at Amazon AWS. We have built upon previous published solutions to this problem and made them more secure in light of the growing sophistication of the bad guys. Let me know if there is interest in an inexpensive commercial service.

References and related
nanny.pl write-up: https://www.safaribooksonline.com/library/view/dns-bind/0596004109/ch05s09.html

Categories
DNS Linux Perl Raspberry Pi Web Site Technologies

Roll your own domain drop catching service using GoDaddy

Intro
I’m after a particular domain and have been for years. But as a matter of pride I don’t want to overpay for it, so I don’t want to go through an auction. There are services that can help grab a DNS domain immediately after it expires, but they all want $$. That may make sense for high-demand domains. Mine is pretty obscure. I want to grab it quickly – perhaps within a few seconds after it becomes available, but I don’t expect any competition for it. That is a description of domain drop catching.

Since I am already using GoDaddy as my registrar I thought I’d see if they have a domain catching service. They don’t which is strange because they have other specialized domain services such as domain broker. They have a service which is designed for much the same purpose, however, called backorder. That creates an auction bid for the domain before it has expired. The cost isn’t too bad, but since I started down a different path I will roll my own. Perhaps they have an API which can be used to create my own domain catcher? It turns out they do!

It involves understanding how to read a JSON data file, which is new to me, but otherwise it’s not too bad.

The domain lifecycle
This graphic from ICANN illustrates it perfectly for your typical global top-level domain such as .com, .net, etc:
gtld-lifecycle

To put it into words, there is the
initial registration,
optional renewals,
expiration date,
auto-renew grace period of 0 – 45 days,
redemption grace period of 30 days,
pending delete of 5 days, and then
it’s released and available.

So in domain drop catching we are keenly interested in being fully prepared for the pending delete five day window. From an old discussion I’ve read that the precise time .com domains are released is usually between 2 -3 PM EST.

A word about the GoDaddy developer site
It’s developer.godaddy.com. It looks like one day it will be a great site, but for now it is wanting in some areas. Most of the menu items are duds and are placeholders. Really there are only three (mostly) working sections: get started, documentation and demo. Get started is only a few words and one slender snippet of Ajax code, and the demo itself also extremely limited, so the only real resource they provide is Documentation. Documentation is designed as an active documentation that you can try out functions with your data. You run it and it shows you all the needed request headers and data as well as the received response. The thing is that it’s very finicky. It’s supposed to show all the available functions but I couldn’t get it to work under Firefox. And with Internet Explorer/Edge it only worked about half the time. It seems to help to access it with a newly launched browser. The documentation, as good as it is, leaves some things unsaid. I have found:

https://api.ote-godaddy.com/ – use for TEST. Maybe ote stands for optional test environment?
https://api.godaddy.com/ – for production (what I am calling PROD)

The TEST environment does not require authentication for some things that PROD does. This shell script for checking available domains, which I call available-test.sh, works in TEST but not in PROD:

#!/bin/sh
# pass domain as argument
# apparently no AUTH is rquired for this one
curl -k 'https://api.ote-godaddy.com/v1/domains/available?domain='$1'&amp;checkType=FAST&amp;forTransfer=false'

In PROD I had to insert the authorization information – the key and secret they showed me on the screen. I call this script available.sh.

#!/bin/sh
# pass domain as argument
curl -s -k -H 'Authorization: sso-key *******8m_PwFAffjiNmiCUrKe******:**FF73L********' 'https://api.godaddy.com/v1/domains/available?domain='$1'&amp;checkType=FULL&amp;forTransfer=false'

I found that my expiring domain produced different results about five days after expiring if I used checkType of FAST versus checkType of FULL – and FAST was wrong. So I learned you have to use FULL to get an answer you can trust!

Example usage of an available domain

$ ./available.sh dr-johnstechtalk.com

{"available":true,"domain":"dr-johnstechtalk.com","definitive":false,"price":11990000,"currency":"USD","period":1}

2nd example – a non-available domain
$ ./available.sh drjohnstechtalk.com

{"available":false,"domain":"drjohnstechtalk.com","definitive":true,"price":11990000,"currency":"USD","period":1}

Example JSON file
I had to do a lot of search and replace to preserve my anonymity, but I feel this post wouldn’t be complete without showing the real contents of my JSON file I am using for both validate, and, hopefully, as the basis for my API-driven domain purchase:

{
  "domain": "dr-johnstechtalk.com",
  "renewAuto": true,
  "privacy": false,
  "nameServers": [
  ],
  "consent": {
    "agreementKeys": ["DNRA"],
    "agreedBy": "50.17.188.196",
    "agreedAt": "2016-09-29T16:00:00Z"
  },
  "period": 1,
  "contactAdmin": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    },
    "phone": "+1.5555551212"
  },
  "contactBilling": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    },
    "phone": "+1.5555551212"
  },
  "contactRegistrant": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "phone": "+1.5555551212",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    }
  },
  "contactTech": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "phone": "+1.5555551212",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    }
  }
}

Note the agreementkeys value: DNRA. GoDaddy doesn’t document this very well, but that is what you need to put there! Note also that the nameservers are left empty. I asked GoDaddy and that is what they advised to do. The other values are pretty much what you’d expect. I used my own server’s IP address for agreedBy – use your own IP. I don’t know how important it is to get the agreedAt date close to the current time. I’m going to assume it should be within 24 hours of the current time.

How do we test this JSON input file? I wrote a validate script for that I call validate.sh.

#!/bin/sh
# DrJ 9/2016
# godaddy-json-register was built using GoDaddy's documentation at https://developer.godaddy.com/doc#!/_v1_domains/validate
jsondata=`tr -d '\n' &lt; godaddy-json-register`
 
curl -i -k -H 'Authorization: sso-key *******8m_PwFAffjiNmiCUrKe******:**FF73L********' -H 'Content-Type: application/json' -H 'Accept: application/json' -d "$jsondata" https://api.godaddy.com/v1/domains/purchase/validate

Run the validate script
$ ./validate.sh

HTTP/1.1 100 Continue
 
HTTP/1.1 100 Continue
Via: 1.1 api.godaddy.com
 
HTTP/1.1 200 OK
Date: Thu, 29 Sep 2016 20:11:33 GMT
X-Powered-By: Express
Vary: Origin,Accept-Encoding
Access-Control-Allow-Credentials: true
Content-Type: application/json; charset=utf-8
ETag: W/"2-mZFLkyvTelC5g8XnyQrpOw"
Via: 1.1 api.godaddy.com
Transfer-Encoding: chunked

Revised versions of the above scripts
So we can pass the domain name as argument I revised all the scripts. Also, I provide an agreeddAt date which is current.

The data file: godaddy-json-register

{
  "domain": "DOMAIN",
  "renewAuto": true,
  "privacy": false,
  "nameServers": [
  ],
  "consent": {
    "agreementKeys": ["DNRA"],
    "agreedBy": "50.17.188.196",
    "agreedAt": "DATE"
  },
  "period": 1,
  "contactAdmin": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    },
    "phone": "+1.5555551212"
  },
  "contactBilling": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    },
    "phone": "+1.5555551212"
  },
  "contactRegistrant": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "phone": "+1.5555551212",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    }
  },
  "contactTech": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "phone": "+1.5555551212",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    }
  }
}

validate.sh

#!/bin/sh
# DrJ 10/2016
# godaddy-json-register was built using GoDaddy's documentation at https://developer.godaddy.com/doc#!/_v1_domains/validate
# pass domain as argument
# get date into accepted format
domain=$1
date=`date -u --rfc-3339=seconds|sed 's/ /T/'|sed 's/+.*/Z/'`
jsondata=`tr -d '\n' &lt; godaddy-json-register`
jsondata=`echo $jsondata|sed 's/DATE/'$date'/'`
jsondata=`echo $jsondata|sed 's/DOMAIN/'$domain'/'`
#echo date is $date
#echo jsondata is $jsondata
curl -i -k -H 'Authorization: sso-key *******8m_PwFAffjiNmiCUrKe******:**FF73L********' -H 'Content-Type: application/json' -H 'Accept: application/json' -d "$jsondata" https://api.godaddy.com/v1/domains/purchase/validate

available.sh
No change. See listing above.

purchase.sh
Exact same as validate.sh, with just a slightly different URL. I need to test that it really works, but based on my reading I think it will.

#!/bin/sh
# DrJ 10/2016
# godaddy-json-register was built using GoDaddy's documentation at https://developer.godaddy.com/doc#!/_v1_domains/purchase
# pass domain as argument
# get date into accepted format
domain=$1
date=`date -u --rfc-3339=seconds|sed 's/ /T/'|sed 's/+.*/Z/'`
jsondata=`tr -d '\n' &lt; godaddy-json-register`
jsondata=`echo $jsondata|sed 's/DATE/'$date'/'`
jsondata=`echo $jsondata|sed 's/DOMAIN/'$domain'/'`
#echo date is $date
#echo jsondata is $jsondata
curl -s -i -k -H 'Authorization: sso-key *******8m_PwFAffjiNmiCUrKe******:**FF73L********' -H 'Content-Type: application/json' -H 'Accept: application/json' -d "$jsondata" https://api.godaddy.com/v1/domains/purchase

Putting it all together
Here’s a looping script I call loop.pl. I switched to perl because it’s easier to do certain control operations.

#!/usr/bin/perl
#DrJ 10/2016
$DEBUG = 0;
$status = 0;
open STDOUT, '&gt;', "loop.log" or die "Can't redirect STDOUT: $!";
                   open STDERR, "&gt;&amp;STDOUT"     or die "Can't dup STDOUT: $!";
 
                   select STDERR; $| = 1;      # make unbuffered
                   select STDOUT; $| = 1;      # make unbuffered
# edit this and change to your about-to-expire domain
$domain = "dr-johnstechtalk.com";
while ($status != 200) {
# show that we're alive and working...
  print "Now it's ".`date` if $i++ % 10 == 0;
  $hr = `date +%H`;
  chomp($hr);
# run loop more aggressively during times of day we think Network Solutions releases domains back to the pool, esp. around 2 - 3 PM EST
  $sleep = $hr &gt; 11 &amp;&amp; $hr &lt; 16 ? 1 : 15;
  print "Hr,sleep: $hr,$sleep\n" if $DEBUG;
  $availRes = `./available.sh $domain`;
# {"available":true,"domain":"dr-johnstechtalk.com","definitive":false,"price":11990000,"currency":"USD","period":1}
  print "$availRes\n" if $DEBUG;
  ($available) = $availRes =~ /^\{"available":([^,]+),/;
  print "$available\n" if $DEBUG;
  if ($available eq "false") {
    print "test comparison OP for false result\n" if $DEBUG;
  } elsif ($available eq "true") {
# available value of true is extremely unreliable with many false positives. Confirm availability by making a 2nd call
    print "available.sh results: $availRes\n";
    $availRes = `./available.sh $domain`;
    print "available.sh re-test results: $availRes\n";
    ($available2) = $availRes =~ /^\{"available":([^,]+),/;
    next if $available2 eq "false";
# We got two available eq true results in a row so let's try to buy it!
    print "$domain is very likely available. Trying to buy it at ".`date`;
    open(BUY,"./purchase.sh $domain|") || die "Cannot run ./purchase.pl $domain!!\n";
    while() {
# print out each line so we can analyze what happened
      print ;
# we got it if we got back
# HTTP/1.1 200 OK
      if (/1.1 200 OK/) {
        print "We just bought $domain at ".`date`;
        $status = 200;
     }
    } # end of loop over results of purchase
    close(BUY);
    print "\n";
    exit if $status == 200;
  } else {
    print "available is neither false nor true: $available\n";
  }
  sleep($sleep);
}

Running the loop script
$ nohup ./loop.pl > loop.log 2>&1 &
Stopping the loop script
$ kill ‐9 %1

Description of loop.pl
I gotta say this loop script started out as a much simpler script. I fortunately started on it many days before my desired domain actually became available so I got to see and work out all the bugs. Contributing to the problem is that GoDaddy’s API results are quite unreliable. I was seeing a lot of false positives – almost 20%. So I decided to require two consecutive calls to available.sh to return true. I could have required available true and definitive true, but I’m afraid that will make me late to the party. The API is not documented to that level of detail so there’s no help there. But so far what I have seen is that when available incorrectly returns true, simultaneously definitive becomes false, whereas all other times definitive is true.

Results of running an earlier and simpler version of loop.pl
This shows all manner of false positives. But at least it never allowed me to buy the domain when it wasn’t available.

Now it's Wed Oct  5 15:20:01 EDT 2016
Now it's Wed Oct  5 15:20:19 EDT 2016
Now it's Wed Oct  5 15:20:38 EDT 2016
available.sh results: {"available":true,"domain":"dr-johnstechtalk.com","definitive":false,"price":11990000,"currency":"USD","period":1}
dr-johnstechtalk.com is available. Trying to buy it at Wed Oct  5 15:20:46 EDT 2016
HTTP/1.1 100 Continue
 
HTTP/1.1 100 Continue
Via: 1.1 api.godaddy.com
 
HTTP/1.1 422 Unprocessable Entity
Date: Wed, 05 Oct 2016 19:20:47 GMT
X-Powered-By: Express
Vary: Origin,Accept-Encoding
Access-Control-Allow-Credentials: true
Content-Type: application/json; charset=utf-8
ETag: W/"7d-O5Dw3WvJGo8h30TqR7j8zg"
Via: 1.1 api.godaddy.com
Transfer-Encoding: chunked
 
{"code":"UNAVAILABLE_DOMAIN","message":"The specified `domain` (dr-johnstechtalk.com) isn't available for purchase","name":"ApiError"}
Now it's Wed Oct  5 15:20:58 EDT 2016
Now it's Wed Oct  5 15:21:16 EDT 2016
Now it's Wed Oct  5 15:21:33 EDT 2016
available.sh results: {"available":true,"domain":"dr-johnstechtalk.com","definitive":false,"price":11990000,"currency":"USD","period":1}
dr-johnstechtalk.com is available. Trying to buy it at Wed Oct  5 15:21:34 EDT 2016
HTTP/1.1 100 Continue
 
HTTP/1.1 100 Continue
Via: 1.1 api.godaddy.com
 
HTTP/1.1 422 Unprocessable Entity
Date: Wed, 05 Oct 2016 19:21:36 GMT
X-Powered-By: Express
Vary: Origin,Accept-Encoding
Access-Control-Allow-Credentials: true
Content-Type: application/json; charset=utf-8
ETag: W/"7d-O5Dw3WvJGo8h30TqR7j8zg"
Via: 1.1 api.godaddy.com
Transfer-Encoding: chunked
 
{"code":"UNAVAILABLE_DOMAIN","message":"The specified `domain` (dr-johnstechtalk.com) isn't available for purchase","name":"ApiError"}
Now it's Wed Oct  5 15:21:55 EDT 2016
Now it's Wed Oct  5 15:22:12 EDT 2016
Now it's Wed Oct  5 15:22:30 EDT 2016
available.sh results: {"available":true,"domain":"dr-johnstechtalk.com","definitive":false,"price":11990000,"currency":"USD","period":1}
dr-johnstechtalk.com is available. Trying to buy it at Wed Oct  5 15:22:30 EDT 2016
...

These results show why i had to further refine the script to reduce the frequent false positives.

Review
What have we done? Our looping script, loop.pl, loops more aggressively during the time of day we think Network Solutions releases expired .com domains (around 2 PM EST). But just in case we’re wrong about that we’ll run it anyway at all hours of the day, but just not as quickly. So during the aggressive period we sleep just one second between calls to available.sh. When the domain finally does become available we call purchase.sh on it and exit and we write some timestamps and the domain we’ve just registered to our log file.

Performance
Miserable. This API seems tuned for relative ease-of-use, not speed. The validate call often takes, oh, say 40 seconds to return! I’m sure the purchase call will be no different. For a domainer that’s a lifetime. So any strategy that relies on speed had better turn to a registrar that’s tuned for it. GoDaddy I think is more aiming at resellers of their services.

Don’t have a linux environment handy?
Of course I’m using my own Amazon AWS server for this, but that needn’t be a barrier to entry. I could have used one of my Raspberry Pi’s. Probably even Cygwin on a Windows PC could be made to work.

Appendix A
How to remove all newline characters from your JSON file

Let’s say you have a nice JSON file which was created for you from the Documentation exercises called godaddy-json-register. It will contain lots of newline (“\n”) characters, assuming you’re using a Linux server. Remove them and put the output into a file called compact-json:

$ tr ‐d ‘\n'<godaddy‐json‐register>compact‐json

I like this because then I can still use curl rather than wget to make my API calls.

Appendix B
What an expiring domain looks like in whois
Run this from a linux server
$ whois <expiring‐domain.com>

Domain Name: expiring-domain.com
...
Creation Date: 2010-09-28T15:55:56Z
Registrar Registration Expiration Date: 2016-09-27T21:00:00Z
...
Domain Status: clientDeleteHold
Domain Status: clientDeleteProhibited
Domain Status: clientTransferProhibited
...

You see that Domain Status: clientDeleteHold? You don’t get that for regular domains whose registration is still good. They’ll usually have the two lines I show below that, but not that one. This is shown for my desired domain just a few days after its official expiration date.

2020 update

Four years later, still hunting for that domain – I am very patient! So I dusted off the program described here. Suprisingly, it all still works. Except maybe the JSON file. The onlything wrong with that was the lack of nameservers. I added some random GoDaddy nameservers and it seemed all good.

Conclusion
We show that GoDaddy’s API works and we provide simple scripts which can automate what is known as domain dropcatching. This approach should attempt to register a domain within a cople seconds of its being released – if we’ve done everything right. The GoDaddy API results are a little unstable however.

References and related
If you don’t mind paying, Fabulous.com has a domain drop catching service.
ICANN’s web site with the domain lifecycle infographic.
GoDaddy’s API documentation: http://developer.godaddy.com/
More about Raspberry Pi: http://raspberrypi.org/
I really wouldn’t bother with Cygwin – just get your hands on real Linux environment.
Curious about some of the curl options I used? Just run curl ‐‐help. I left out the description of the switches I use because it didn’t fit into the narrative.
Something about my Amazon AWS experience from some years ago.
All the perl tricks I used would take another blog post to explain. It’s just stuff I learned over the years so it didn’t take much time at all.
People who buy and sell domains for a living are called domainers. They are professionals and my guide will not make you competitive with them.

Categories
Linux Raspberry Pi

Solution to this week’s NPR puzzle using simple Linux commands

Intro
Every now and then the weekend puzzle is particularly amenable to partial solution by use of simple Linux commands. I suspected such was the case for this week’s, and I was right.

The challenge for this week
Take a seven letter word of one syllable, add the consecutive letters “it” somewhere in the middle to create a nine letter word of four syllables.

The Linux command-line method of solution
On a CentOS system there is a file with words, lots of words. It’s /usr/share/dict/linux.words:

$ cd /usr/share/dict; wc linux.words

479829  479829 4953699 linux.words

So, 479829 words! A lot of them are junk words however, but it has the real ones in there too. This file comes from the RPM package words-3.0-17.el6.noarch.

So here’s a sort of stream-of-consciousness of a Unix person solving the puzzle without doing too much work or too much thinking:

How many seven-letter words are there? First what’s an expression that can answer that? I think this is it but let’s check:

$ egrep '^[a‐z]{7}$' linux.words|more

aaronic
aarrghh
abacate
abacaxi
abacist
abactor
abaculi
abaddon
abadejo
abaised
abaiser
abaisse
abalone
abandon
abandum
abasers
abashed
abashes
abasias
abasing
abatage
abaters
abating
abators
abattis
abattue
abature
abaxial
...

OK, that egrep expresison is right. So the seven-letter word count is then:

$ egrep '^[a‐z]{7}$' linux.words|wc ‐l

40230

That’s a lot – too many to eyeball. OK, so how many nine-letter words are there?

$ egrep '^[a‐z]{9}$' linux.words|wc ‐l

50748

Wow, even more.

OK, we have an idea, based not on what may be the best approach, but based on which Linux commands we know inside and out. The idea is to start from the nine-letter words which contain “it”, remove the “it” and then match the resulting seven-letter character strings against our dictionary to see which are actually words. We know how to do that. The hope is the resulting list will be small enough we can review by hand.

How many nine-letter words contain the consecutive characters “it”?

$ egrep '^[a‐z]{9}$' linux.words|grep it|wc ‐l

3245

They look like this:

abilities
abnormity
aboiteaus
aboiteaux
abolition
abrazitic
absurdity
academite
acanthite
accipiter
acclivity
accredits
accubitum
accubitus
acetosity
acidities
acinacity
aconitine
aconitums
acquisita
...

so it would take forever to go through. If we had a dictionary with the syllable count we coul really narrow it down. I think I’ve seen that, but I’d have to dig that up. We introduce the sed operator to remove the “it” from these words:

$ egrep '^[a‐z]{9}$' linux.words|grep it|sed 's/it//'|more

abilies
abnormy
aboeaus
aboeaux
abolion
abrazic
absurdy
academe
acanthe
acciper
acclivy
accreds
accubum
accubus
acetosy
acidies
acinacy
aconine
aconums
acquisa
...

There are more efficient ways to loop through these results using xargs, but I’m old school and have memorized this older construct which I use:

$ egrep '^[a‐z]{9}$' linux.words|grep it|sed 's/it//'|while read line; do
> grep $line linux.words >> /tmp/lw
> done

We look at the resulting file and found we made a little goof – we didn’t limit the resulting matches to seven characters:

$ more /tmp/lw

academe
academes
Pleuracanthea
vacanthearted
vacantheartedness
aconine
japaconine
pseudaconine
adderspit
affidavit
affidavits
affidavy
preaffidavit
divagating
extravagating
indagating
propagating
self-propagating
...

But that’s easily corrected:

$ cd /tmp; egrep '^[a‐z]{7}$' lw > lw2
$ wc -l lw2

376 lw2

Now that’s a number we can review by hand. Very few of these have only one syllable:

$ more lw2

academe
aconine
alumine
alveole
ammonic
barbary
barbone
basting
bauxite
berline
bethank
boraces
bullion
capella
carbone
carmele
carmine
cascade
catline
cavated
celeste
ceruses
chloric
chondre
chromes
cations
claries
cockney
coenobe
compose
...

I quickly reviewed the list and the answer popped out, somewhere towards the end – you can’t miss it.

Friday update – the solution
The 7-letter word that pops out at you? Reigned, which you immediately see becomes reignited – nine letters and four syllables!

Want to do this on your Raspberry Pi?
The dictionary file there is /usr/share/dictd/wn.index, but you probably don’t have it by default so you’ll need to install a few packages to get it which is simple enough. This post about Words with Friends explains the packages I used to provide that dictionary. Aside from the location of the dictionary, and that it contains fewer(?) words, everything else should be the same.

Conclusion
We have solved this week’s NPR puzzle without any complex programming just by using some simple Linux commands.

References and related
This link is nice because it has a transcription of the puzzle so you don’t have to waste time listening to the whole six-minute segment.
Another NPR puzzle we solved in a similar way.

Categories
Raspberry Pi

The IT detective agency: Raspberry Pi kept dying

Intro
My Raspberry Pi Model B worked fine in one room in my house. I was very excited to trade it in for a newer and faster Raspberry Pi model 2. I just re-used the existing power adapter because it was hard to access. But my new Raspberry Pi, unlike the old unit, was not fine. It crashed from time to time. Just freeze up and need a power cycle to get restarted.

The solution
I guess the Raspberry Pi 2 consumed just a little more power than the old model. But not during bootup and normal usage. I eventually and only recently figured out that it was every time I did a sudo apt-get update that it crashed. I guess this gets the CPU literally fired up and consuming more power and putting the thing over the edge, power-wise.

I switched to a kind of adapter used for charging smartphones – i.e., with a 2 amp capacity. Since then it has been fine. This same adapter in fact powers both the Ras Pi 2 and the Raspberry Pi touch display, so it’s much better than the cheapo old adapter.

It took me so long to figure this out because I had a bias that Raspberry Pis are somewhat prone to crash anyway, and sometimes they don’t boot properly, etc. So I figured it was all part of the fragility of the system.

References and related
A first look at the Raspberry Pi 7″ touch display.
Using Raspberry Pis to drive a four-monitor display.

Categories
Linux Raspberry Pi

What I’m working on now: Raspberry Pi Touch Display

Intro
I’ve been waiting for a decent, inexpensive display for the Raspberry Pi and now they make one. http://www.mcmelectronics.com/product/83-16872 I’m one of the lucky ones – I got one before it was put on backorder! Anyway it’s just $60 plus shipping. that link is long since dead. But I see Amazon carries it: Amazon.com: Raspberry Pi 7″ Touch Screen Display: Electronics

Power
The jumper wires can be used for power. I initially connected it that way, but the supplied wires have nowhere really safe to rest so it seems extremely likely to bend a pin. Too bad they didn’t supply right-angle jumper wire connectors, if such a thing exists. (They do not.) But you can also do USB out from the adapter board to micro-USB in on the Pi. That’s little better. I bought a 6″ micro-USB cable from Amazon and now I use that. But boy that’s a really, really tight fit. You’re as likely to break the display and the Pi due to the tension as you are to succeed in getting them connected. I pre-bent the connectors to make it work. Another inch to work with would have been ideal.

Virtual keyboard
Apparently
$ sudo apt-get install matchbox-keyboard
will install a virtual keyboard. I realized the thing wasn’t as portable as I had hoped as long as I had to have a USB keyboard attached! But the little virtual keyboard is really hard to use.

Touchscreen
I find the touchscreen really hard to work with to control window sizing. And although their specs say it’s 800×480, it doesn’t come up that way, so you really don’t have much display area in fact. It comes up as 752×448 and the launched windows seemed cropped at the bottom. If you change a window to full screen then it’s OK however, but even at full size it just doesn’t feel that large. See below to get the full 800×400 display operational.

The edges are really hard to tap with even slender fingers. Someone suggested a capacitive pen – that would probably work a lot better but I don’t have one to test with.

In general accurate tapping is kind of hard. For instance an X-window has three little icons in the upper right corner – minimize, maximize and close. It’s hard to tap the one you want – I get it wrong most of the time.

Display is smaller than the monitor
This has bugged me for over a year and I finally found it. This happens due to overscan being on. I doubt overscan is needed for most modern displays. So just go into raspi-config’s advanced options and disable overscan! Then your display will be 800 x 400 from now on.

Application idea – visual alarm
I’m thinking of creating a “visual alarm.” This could gently wake me up at night for problems at work, by turning on the screen and displaying an alternating light/dark pattern. This will be another post if I ever implement it.

Conclusion
My initial enthusiasm at the idea of a portable display designed specifically for the Pi was dampened by my actual experience with one. I was hoping for a 10″ display – better for educational purposes. The user experience with this display might be a lot better with a capacitive pen.

References and related
Useful Raspberry Pi touch display setup guide.
Four Pi display.
Have no display whatsoever for your Raspberry Pi? I show a way to work with it under even those circumstances. In fact that is what I mostly do.

The Pi Display at amazon costs about $64: Amazon.com: Raspberry Pi 7″ Touch Screen Display: Electronics

There is a an alternative which I haven’t tried that you may want to check out, the RasPad: https://www.raspad.com/

Categories
Network Technologies Raspberry Pi

Dig for Windows or Raspberry Pi

Windows

Intro
Dig is a really useful networking tool. I use it several times a day. But always on Linux where it’s usually built-in. On Raspberry Pi’s raspbian you can install it with a simple apt-get install dnsutils. Then I learned it wasn’t hard at all to install on Windows, especially as a fairly minimalist installation that just puts files on your PC and makes no changes to the Registry, which is all you really need for light use.

The details
Go to http://www.isc.org/downloads/. Expand BIND.
ISC-BIND-download-screen
Click download button for the current stable release.
Pick the win-64-bit link (because chances are you’re running Windows 64 bit these days) and wait for download to complete.
Open up zip file.
Unzip or extract all files to (this is my suggestion) c:\apps\bind.

To run it
Open a command window. Probably easiest way is hold down Windows key + r and type in cmd. In CMD window simply type \apps\bind\dig to run dig like you do on Linux.

Example commands
Example 1, Resolve address for google.com

C:\> \apps\bind\dns google.com

; &lt;&lt;&gt;&gt; DiG 9.9.8-P2 &lt;&lt;&gt;&gt; google.com
;; global options: +cmd
;; Got answer:
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 24929
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;google.com.                    IN      A
 
;; ANSWER SECTION:
google.com.             88      IN      A       173.194.207.113
google.com.             88      IN      A       173.194.207.139
google.com.             88      IN      A       173.194.207.138
google.com.             88      IN      A       173.194.207.101
google.com.             88      IN      A       173.194.207.102
google.com.             88      IN      A       173.194.207.100
 
;; Query time: 41 msec
;; SERVER: 192.168.2.1#53(192.168.2.1)
;; WHEN: Mon Jan 11 12:16:17 Eastern Standard Time 2016
;; MSG SIZE  rcvd: 135

This gives all kinds of useful information – what your default DNS server is (at the bottom – mine is 192.168.2.1), how long the query took *this one: 41 msec), whether the answer is authoritative or not (no AA flag here, so this is not an authoritative answer), as well as the answer to the question posed.

Example 2, Resolve nameserver records for the domain amazon.com using Google’s DNS server 8.8.8.8 over TCP from our local IP address of 192.168.2.3

We started out slow, but this example throws the kitchen sink at you to show the power of dig!

C:\> \apps\bind\dig +tcp -b 192.168.2.3 ns amazon.com @8.8.8.8

; &lt;&lt;&gt;&gt; DiG 9.9.8-P2 &lt;&lt;&gt;&gt; +tcp -b 192.168.2.3 ns amazon.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 64444
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;amazon.com.                    IN      NS
 
;; ANSWER SECTION:
amazon.com.             3599    IN      NS      ns3.p31.dynect.net.
amazon.com.             3599    IN      NS      ns4.p31.dynect.net.
amazon.com.             3599    IN      NS      ns1.p31.dynect.net.
amazon.com.             3599    IN      NS      pdns1.ultradns.net.
amazon.com.             3599    IN      NS      pdns6.ultradns.co.uk.
amazon.com.             3599    IN      NS      ns2.p31.dynect.net.
 
;; Query time: 50 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jan 11 12:27:26 Eastern Standard Time 2016
;; MSG SIZE  rcvd: 188

The only problem is that I don’t think the TCP option actually worked – I gotta run wireshark to verify. On Linux it definitely works! Not sure what’s wrong with windows. But the other options are working as designed.

OK, wireshark install is failing, but I ran tcpdump on a DNS server I run and confirmed that indeed the +tcp option is working forcing dig to use TCP communication for those queries.

Raspberry Pi

I believe you do

$ sudo apt-get install bind9-dnsutils

At least on a generic Debian system that works. I have to confirm on RPi still.

Conclusion
We’ve demonstrated a low-impact way to install dig for Windows and shown some examples of using it.

References and related

Current BIND link from ISC: https://downloads.isc.org/isc/bind9/9.16.8/BIND9.16.8.x64.zip

Or…you get get dig through a Cygwin installation. I’ve written about Cygwin here: Cygwin. Or just go to cygwin.com.

Categories
Admin Python Raspberry Pi

Building a Four Monitor Media Show using Raspberry Pis

Intro
This is the paper a student wrote under my guidance.

Building a Four Monitor Media Show using Raspberry Pis

The first page
4-monitor-media-display

Link to full article

References
My write-up concerning our novel use of the Pi Presents program, which has a different emphasis and no pictures.

Categories
Python Raspberry Pi

Raspberry Pi visual alerting with Pibrella

I just came across this article: http://www.itworld.com/article/2919325/personal-technology/website-monitoring-with-a-raspberry-pi-for-nighttime-alerts.html?phint=newt%3Ditworld_linux&phint=idg_eid%3Db4ed050ca62ad3f6d5c69e8448d945d7#tk.ITWNLE_nlt_linux_2015-05-19

I wish I had thought of it first! I’m in a similar situation – I constantly get emails and TXT messages overnight so I have to put my phone in airplane mode. A visual alerting might just help give me an early warning for come critical failures.

The Pibrella module that lights up and buzzes has its own web site, pibrella.com.

If I do anything with it I’ll be sure to post it here.

Categories
Admin Raspberry Pi

Can’t ssh to Raspberry Pi

Intro
I did not experience this problem but it happened to a friend of mine – more than once. I’m not sure what he’s doing during installation of the OS to cause this but I know the solution.

Also there is some really misguided information out there on the Internet. Pure ignorance talking. There’s some accurate discussion as well. By re-posting correct information I hope to increase the average correct treatment. I have nothing original to say per se.

The details
The problem is that you can”t log in to your Raspberry Pi via ssh. See if the ssh daemon is running:

$ sudo service ssh status

sshd is running.

Probably it is and you see the response above.

Try to log in locally:

$ ssh localhost

If you get something like this:

Read from socket failed: Connection reset by peer

then you have the problem which can be fixed by this procedure below.

The fix

$ sudo dpkg-reconfigure openssh-server

Or more generally:

$ sudo rm /etc/ssh/ssh_host_* && sudo dpkg-reconfigure openssh-server

There must be some way to screw up the Raspbian installation that produces this particular problem.

And all this is assuming you already had the presence of mind to enable ssh in raspi-config. That of course is a prerequisite.

Conclusion
You should be able to ssh (remote shell access) to your Raspberry Pi. If you cannot the recipe above is probably the needed fix. This discussion is generally valid for all Debian installations, except for the comment about raspi-config!

References
The correct discussion on this topic in the official Raspberry Pi forum is here.
I’ve published quite a few articles now on Raspberry Pi.
A digital photo frame is described in this article.
Using a Raspberry Pi as a router is described here.