Category: Admin

Categories
Admin Apache Hosting Service Web Site Technologies

Server Name Indication and what it means for those with only a single IP address

Intro
Sometimes everything is there in place, ready to be used, but you just have to either mistakenly try it, or learn it works by reading about it, because it may be counter-intuitive. Such is the case with Server Name Indication. I thought I knew enough about https to “know” that you can only have one key/certificate for a single IP address. That CERT can be a SAN (subject alternative name) CERT covering multiple names, but you only get one shot at getting your certificate right. Or so I thought. Turns out I was dead wrong.

Some details
Well, SNI guess is a protocol extension to https. You know I always wondered why in proxy server logs it was able to log the domain name? How would it know that if the http protocol conversation is all encrypted? Maybe it’s SNI at work.

Who supports it?
Since this is an extension it has to be supported by both server and browser. It is. Apache24 supports it. IE, Firefox and Chrome support it. Even my venerable curl supports it! What does not support it, right out of the box, is openssl. The openssl s_client command fetches a site’s certificate, but as I found the hard way, you need to add the -servername switch to tell it which certificate you want to examine, i.e., to force it to use SNI.

This is mainly used by big hosting companies so they can easily and flexibly cram lots of web sites onto a single IP, but us small-time self-hosted sites benefit as well. I host a few sites for friends after all.

Testing methodology
This is pretty simple. I have a couple different virtual servers. I set each up with a completely different certificate in my apache virtual server setups. Then I accessed them by name like usual. Each showed me their own, proper, certificate. That’s it! So this is more than theoretical for me. I’ve already begun to use it.

Enterprise usage
F5 BigIP supports this protocol as well, of course. This article describes how to set it up. But it looks limited to only one server name per certificate, which will be inadequate if there are SAN certificates.

Conclusion
https using Server Name Indication allows to run multiple virtual servers, each with its own unique certificate, on a single IP address.

References and related
I get my certificates for free using the acme.sh interface to Let’s Encrypt
I’ve written some about apache 2.4 in this post
I don’t think Server Name Indication is explained very well anywhere that I’ve seen. The best dewscription I’ve found is that F5 Devcentral article: https://devcentral.f5.com/articles/ssl-profiles-part-7-server-name-indication
RFC 4366 is the spec describing Server Name Indication.
My favorite openssl commands are listed in this blog post.
SNI is considered insecure because the hostname is sent in plaintext. encrypted SNI is the proposal to address that. Here’s a good write-up about that: https://nakedsecurity.sophos.com/2018/09/26/finally-a-fix-for-the-encrypted-webs-achilles-heel/?utm_source=Naked+Security+-+Sophos+List&utm_campaign=27caad8932-Naked+Security+daily+news+email&utm_medium=email&utm_term=0_31623bb782-27caad8932-418487137

Categories
Admin Linux

Relative time with the linux date command

The situation
A server in Europe needs to transfer a log file which is written every hour from a server in the US. The filename format is
20171013-1039.log.gz
And we want the transfer to be done every hour.

How we did it
I learned something about the date command. I wanted to do date arithmetic, like calculate the previous hour. I’ve only ever done this in Perl. Then I saw how someone did it within a bash script.

First the timezone

export TZ=America/New_York

sets the timezone to that of the server which is writing the log files. This is important.

Then get the previous hour
$ onehourago=`date ‐‐date='1 hours ago' '+%Y%m%d‐%H'`

That’s it!

Then the ftp command looks like
$ get $onehourago

If we needed the log from two hours ago we would have had

$ twohourago=`date ‐‐date='2 hours ago' '+%Y%m%d‐%H'`

If one day ago

$ onedayago=`date ‐‐date='1 days ago' '+%Y%m%d‐%H'`

One hour from now

$ onedayago=`date ‐‐date='+1 hours' '+%Y%m%d‐%H'`

etc.

Why the timezone setting?
Initially I skipped the timezone setting and I simply put 7 hours ago, given that Europe and New York are six hours apart, and that’ll work 95% of the time. But because Daylight Savings time starts and ends at different times in the two continents, that will produce bad results for a few weeks a year. So it’s cleaner to simply switch the timezone before doing the date arithmetic.

Conclusion
The linux date command has more features than I thought. We’ve shown how to create some relative dates.

References and related
On a linux system
$ info date
will give you more information and lots of examples to work from.

Categories
Admin Linux Security

Fail2ban fails to work, I built my own

Intro
I’ve sung the praises of fail2ban as a modern way to shutdown those annoying probes of your cloud server. I recently got to work with a Redhat v 7.4 system, so much newer than my old CentOS 6 server. And fail2ban failed even to work! Instead of the usual extensive debugging I just wrote my own. I’m sharing it here.

The details
I have a bare-bones RHEL 7.4 system. A yum search fail2ban does not find that package. Supposedly you simply need to add the EPEL repository to make that package available but the recipe on how to do that is not obvious. So I got the source for fail2ban and built it. Although it runs, you gotta build a local jail to block ssh attempts and that’s where it fails. So instead of going down that rabbit hole – I was already too deep, I decided to heck with it and I’m building my own.

All I really wanted was to ban IPs which are hitting my sshd server endlessly, often once per second or more. I take it personally.

RHEL 7 has a new firewall concept, firewalld. It’s all new to me and I don’t want to go down that rabbit hole either, at least not right down. So I rely on that old standard of mine: cut off an attacker by making an invalid route to his IP address, along the lines of

$ route add ‐host gw 127.0.0.1

And voila, they can no longer establish a TCP connection. It’s not quite as good as a firewall rule because their source UDP packets could still get through, but come on, we don’t need to be purists. And furthermore, in practice it produces the desired behaviour: stops the ssh dictionary attacks cold.

I knocked tghis out in one night, avoiding the rabbit hole of “fixing” fail2ban. So I had to use the old stuff I know so well, perl and stupid little tricks. I call drjfail2ban.

#!/bin/perl
# suppress IPs with failed logins
# DrJ - 2017/10/07
$DEBUG = 0;
$sleep = 30;
$cutoff = 3;
$headlines = 60;
@goodusers =("drjohn1","user57");
%blockedips = ();
while(1) {
#  $time = `date +%Y%m%d%H%M%S`;
  main();
  sleep($sleep);
}
 
sub main() {
if ($DEBUG) {
  for $ips (keys %blockedips) {
    print "blocked ip: $ips "
  }
}
# man last shows what this means: -i forces IP to be displayed, etc.
open(LINES,"last -$headlines -i -f /var/log/btmp|") || die "Problem with running last -f btmp!!\n";
# output:
#ubnt     ssh:notty    185.165.29.197   Sat Oct  7 19:30    gone - no logout
while(<LINES>) {
  ($user,$ip) = /^(\S+)\s+\S+\s+(\S+)/;
  print "user,ip: $user,$ip\n" if $DEBUG;
  next if $blockedips{$ip};
#we can't handle hostnames right now
  next if $ip =~ /[a-z]/i;
  $candidateips{$ip} += 1;
  $bannedusers{$ip} = $user;
}
for (keys %candidateips) {
  $ip = $_;
# allow my usual source IPs without blocking...
  next if $ip =~ /^(50\.17\.188\.196|51\.29\.208\.176)/;
  next if $blockedips{$ip};
  $usr = $bannedusers{$ip};
  $ipct = $candidateips{$ip};
  print "ip, usr, ipct: $ip, $usr, $ipct\n" if $DEBUG;
# block
  $block = 1;
  for $gu (@goodusers) {
    print "gu: $gu\n" if $DEBUG;
    $block = 0 if $usr eq $gu;
  }
  if ($block) {
# more tests: persistence of attempt
    $hitcnt = $candidateips{$ip};
    if ($hitcnt < $cutoff) {
# do not block and reset counter for next go-around
      print "Not blocking ip $ip and resetting counter\n" if $DEBUG;
      $candidateips{$ip} = 0;
    } else {
      $blockedips{$ip} = 1;
      print "Blocking ip $ip with hit count $hitcnt at " . `date`;
# prevent further communication...
      system("route add -host $ip gw 127.0.0.1");
   }
  }
  #print "route add -host $ip gw 127.0.0.1\n";
}
close(LINES);
} # end main function

Highlights from the program
The comments are pretty self-explanatory. Just a note about the philosophy. I fear making a goof and locking myself out! So I was conservative and try to not do any blocking if the source IP matches one of my favored source IPs, or if the user matches one of my usual usernames like drjohn1. I use obscure userids and the hackers try the stupid stuff like root, admin, etc. So they may be dictionary attacking the password, but they certainly aren’t dictionary attacking the username!

I don’t mind wiping the slate clean of all created routes after sever reboot so I only plan to run this from the command line. To make it persistent until the next reboot you just run it from the root account like so (let’s say we put it in /usr/local/sbin):

$ nohup /usr/local/sbin/drjfail2ban > /var/log/drjfail2ban &

And it just sits there and runs, even after you log out.

Results
Since it hasn’t been running for long I can provide a partial log file as of this publication.

Blocking ip 103.80.117.74 with hit count 6 at Sun Oct  8 17:34:43 CEST 2017
SIOCADDRT: File exists
Blocking ip 89.176.96.45 with hit count 5 at Sun Oct  8 17:34:43 CEST 2017
SIOCADDRT: File exists
Blocking ip 31.162.51.206 with hit count 3 at Sun Oct  8 17:34:43 CEST 2017
SIOCADDRT: File exists
Blocking ip 218.95.142.218 with hit count 6 at Sun Oct  8 17:34:43 CEST 2017
SIOCADDRT: File exists
Blocking ip 202.168.8.54 with hit count 5 at Sun Oct  8 17:34:43 CEST 2017
SIOCADDRT: File exists
Blocking ip 13.94.29.182 with hit count 4 at Sun Oct  8 17:34:43 CEST 2017
SIOCADDRT: File exists
Blocking ip 40.71.185.73 with hit count 4 at Sun Oct  8 17:34:43 CEST 2017
SIOCADDRT: File exists
Blocking ip 77.72.85.100 with hit count 13 at Sun Oct  8 17:34:43 CEST 2017
SIOCADDRT: File exists
Blocking ip 201.180.104.63 with hit count 7 at Sun Oct  8 17:34:43 CEST 2017
SIOCADDRT: File exists
Blocking ip 121.14.27.58 with hit count 4 at Sun Oct  8 17:40:43 CEST 2017
Blocking ip 36.108.234.99 with hit count 6 at Sun Oct  8 17:47:13 CEST 2017
Blocking ip 185.165.29.69 with hit count 6 at Sun Oct  8 18:02:43 CEST 2017
Blocking ip 190.175.40.195 with hit count 6 at Sun Oct  8 19:05:43 CEST 2017
Blocking ip 139.199.167.21 with hit count 4 at Sun Oct  8 19:29:13 CEST 2017
Blocking ip 186.60.67.51 with hit count 5 at Sun Oct  8 20:49:14 CEST 2017

And what my route table looks like currently:

$ netstat ‐rn|grep 127.0.0.1

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
2.177.217.155   127.0.0.1       255.255.255.255 UGH       0 0          0 lo
13.94.29.182    127.0.0.1       255.255.255.255 UGH       0 0          0 lo
31.162.51.206   127.0.0.1       255.255.255.255 UGH       0 0          0 lo
36.108.234.99   127.0.0.1       255.255.255.255 UGH       0 0          0 lo
37.204.23.84    127.0.0.1       255.255.255.255 UGH       0 0          0 lo
40.71.185.73    127.0.0.1       255.255.255.255 UGH       0 0          0 lo
42.7.26.15      127.0.0.1       255.255.255.255 UGH       0 0          0 lo
46.6.60.240     127.0.0.1       255.255.255.255 UGH       0 0          0 lo
59.16.74.234    127.0.0.1       255.255.255.255 UGH       0 0          0 lo
77.72.85.100    127.0.0.1       255.255.255.255 UGH       0 0          0 lo
89.176.96.45    127.0.0.1       255.255.255.255 UGH       0 0          0 lo
103.80.117.74   127.0.0.1       255.255.255.255 UGH       0 0          0 lo
109.205.136.10  127.0.0.1       255.255.255.255 UGH       0 0          0 lo
113.195.145.13  127.0.0.1       255.255.255.255 UGH       0 0          0 lo
118.32.27.85    127.0.0.1       255.255.255.255 UGH       0 0          0 lo
121.14.27.58    127.0.0.1       255.255.255.255 UGH       0 0          0 lo
139.199.167.21  127.0.0.1       255.255.255.255 UGH       0 0          0 lo
162.213.39.235  127.0.0.1       255.255.255.255 UGH       0 0          0 lo
176.50.95.41    127.0.0.1       255.255.255.255 UGH       0 0          0 lo
176.209.89.99   127.0.0.1       255.255.255.255 UGH       0 0          0 lo
181.113.82.213  127.0.0.1       255.255.255.255 UGH       0 0          0 lo
185.165.29.69   127.0.0.1       255.255.255.255 UGH       0 0          0 lo
185.165.29.197  127.0.0.1       255.255.255.255 UGH       0 0          0 lo
185.165.29.198  127.0.0.1       255.255.255.255 UGH       0 0          0 lo
185.190.58.181  127.0.0.1       255.255.255.255 UGH       0 0          0 lo
186.57.12.131   127.0.0.1       255.255.255.255 UGH       0 0          0 lo
186.60.67.51    127.0.0.1       255.255.255.255 UGH       0 0          0 lo
190.42.185.25   127.0.0.1       255.255.255.255 UGH       0 0          0 lo
190.175.40.195  127.0.0.1       255.255.255.255 UGH       0 0          0 lo
193.201.224.232 127.0.0.1       255.255.255.255 UGH       0 0          0 lo
201.180.104.63  127.0.0.1       255.255.255.255 UGH       0 0          0 lo
201.255.71.14   127.0.0.1       255.255.255.255 UGH       0 0          0 lo
202.100.182.250 127.0.0.1       255.255.255.255 UGH       0 0          0 lo
202.168.8.54    127.0.0.1       255.255.255.255 UGH       0 0          0 lo
203.190.163.125 127.0.0.1       255.255.255.255 UGH       0 0          0 lo
213.186.50.82   127.0.0.1       255.255.255.255 UGH       0 0          0 lo
218.95.142.218  127.0.0.1       255.255.255.255 UGH       0 0          0 lo
221.192.142.24  127.0.0.1       255.255.255.255 UGH       0 0          0 lo

Here’s a partial listing of the many failed logins, just to keep it real:

...
root     ssh:notty    190.175.40.195   Sun Oct  8 19:05 - 19:28  (00:23)
root     ssh:notty    190.175.40.195   Sun Oct  8 19:05 - 19:05  (00:00)
root     ssh:notty    190.175.40.195   Sun Oct  8 19:05 - 19:05  (00:00)
root     ssh:notty    190.175.40.195   Sun Oct  8 19:05 - 19:05  (00:00)
root     ssh:notty    190.175.40.195   Sun Oct  8 19:05 - 19:05  (00:00)
root     ssh:notty    190.175.40.195   Sun Oct  8 19:05 - 19:05  (00:00)
admin    ssh:notty    185.165.29.69    Sun Oct  8 18:02 - 19:05  (01:02)
admin    ssh:notty    185.165.29.69    Sun Oct  8 18:02 - 18:02  (00:00)
admin    ssh:notty    185.165.29.69    Sun Oct  8 18:02 - 18:02  (00:00)
admin    ssh:notty    185.165.29.69    Sun Oct  8 18:02 - 18:02  (00:00)
root     ssh:notty    185.165.29.69    Sun Oct  8 18:02 - 18:02  (00:00)
root     ssh:notty    185.165.29.69    Sun Oct  8 18:02 - 18:02  (00:00)
root     ssh:notty    185.165.29.69    Sun Oct  8 18:02 - 18:02  (00:00)
root     ssh:notty    36.108.234.99    Sun Oct  8 17:47 - 18:02  (00:15)
root     ssh:notty    36.108.234.99    Sun Oct  8 17:47 - 17:47  (00:00)
root     ssh:notty    36.108.234.99    Sun Oct  8 17:47 - 17:47  (00:00)
root     ssh:notty    36.108.234.99    Sun Oct  8 17:47 - 17:47  (00:00)
root     ssh:notty    36.108.234.99    Sun Oct  8 17:47 - 17:47  (00:00)
root     ssh:notty    36.108.234.99    Sun Oct  8 17:46 - 17:47  (00:00)
ubuntu   ssh:notty    121.14.27.58     Sun Oct  8 17:40 - 17:46  (00:06)
ubuntu   ssh:notty    121.14.27.58     Sun Oct  8 17:40 - 17:40  (00:00)
aaaaaaaa ssh:notty    121.14.27.58     Sun Oct  8 17:40 - 17:40  (00:00)
aaaaaaaa ssh:notty    121.14.27.58     Sun Oct  8 17:40 - 17:40  (00:00)
root     ssh:notty    206.71.63.4      Sun Oct  8 17:34 - 17:40  (00:06)
root     ssh:notty    206.71.63.4      Sun Oct  8 17:34 - 17:34  (00:00)
root     ssh:notty    89.176.96.45     Sun Oct  8 16:15 - 17:34  (01:19)
root     ssh:notty    89.176.96.45     Sun Oct  8 16:15 - 16:15  (00:00)
root     ssh:notty    89.176.96.45     Sun Oct  8 16:15 - 16:15  (00:00)
root     ssh:notty    89.176.96.45     Sun Oct  8 16:15 - 16:15  (00:00)
...

Before running drjfail2ban it was much more obnoxious, with the same IP hitting my server every second or so.

Conclusion
I found it easier to roll my own than battle someone else’s errors. It’s kind of fun for me to create these little scripts. I don’t care if anyone else uses them. I will refer to this post myself and probably re-use it elsewhere!

References and related
In an earlier time, I was singing the praises of fail2ban on CentOS.

Categories
Admin Linux

Reverse upper and lower case letters

Intro
I am a look-at-the-keyboard typist! I can’t count the number of times I’ve begun an email with CAPs lock on, and found a nice correspondence looking like this:

hI aNDRES,
 
i RE-CREATED THE SCRIPT.
...

Tired of the re-typing I researched how to quickly repair this, at least for those with a linux command prompt at their disposal.

The details
I added this to my .aliases file:

# reverse upper/lower case
reverse () { tr 'A-Za-z' 'a-zA-Z'; }

Now when I make this mistake I put the text into my clipboard, even if it is multi-line, and fix it like this:

$ echo 'hI aNDRES,

i RE-CREATED THE SCRIPT.'|reverse

Hi Andres,
 
I re-created the script.

After the single quote I pasted in my clipboard text and closed that out with another single quote.

Alternative for those uncertain about Linux aliases
Alternatively, right on the command line you would just run

$ echo 'hI aNDRES,

i RE-CREATED THE SCRIPT.'|tr 'A-Za-z' 'a-zA-Z'

Conclusion
We developed an alias expression to make exchanging upper and lower case in a character string fast and easy, if you just remember a few things and have access to a Linux command prompt.

Categories
Admin CentOS Digital Currency Linux

Adding a swap file in Amazon AWS for CentOS

Intro
I was running a new daemon on my server, factomd, to experiment with digital currency. It’s an old m1.small instance with only 1.7 GB of memory. The first few times I ran it it would 70000 or so blocks, I would let it run overnight, and then it would run out of memory and crash. My admin skills are a little rusty and dated but I eventually realized that adding swap space to my server could help.

The details
I’ve been running this server for five years and never bothered to create a swap area, as it turns out. My CentOS version is, I think, version 6.0, but it’s hard to tell at this point. Anyway, this command shows the lack of an active swap space:

$ sudo swapon ‐s

Filename                                Type            Size    Used    Priority

What to do?
Amazon has introduced SSD storage and that is recommended for high I/O demands. That makes sense to me to use for swap, which is basically an extension of your memory. It’s also inexpensive in small volumes. I decided to create a 2 GB swap file – roughly the same size as the machine’s physical memory. So I bought a gp2 – general purpose – SSD volume of 2 GB. It’s only $0.20/month!

Where did it go?
After attaching it to my instance, I got what is apparently a one-time message saying what device it would appear as on my instance – /dev/sdg. I was a little nervous – justifiably as it turns out – that I would not see it from CentOS. I tried to mount it – no go. Then I did Internet research and found these two informative commands:

$ sudo lsblk ‐‐output NAME,TYPE,SIZE,FSTYPE,MOUNTPOINT,LABEL

NAME    TYPE  SIZE FSTYPE MOUNTPOINT LABEL
xvdj    disk  100G ext4   /mnt/vol
xvde    disk    6G
`-xvde1 part    6G ext4   /
xvde3   disk  896M swap
xvdk    disk    2G

and

$ sudo fdisk ‐l

Disk /dev/xvdj: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
 
 
Disk /dev/xvde: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xaae7682d
 
    Device Boot      Start         End      Blocks   Id  System
/dev/xvde1   *           1         783     6289416   83  Linux
 
Disk /dev/xvde3: 939 MB, 939524096 bytes
255 heads, 63 sectors/track, 114 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
 
 
Disk /dev/xvdk: 2147 MB, 2147483648 bytes
22 heads, 16 sectors/track, 11915 cylinders
Units = cylinders of 352 * 512 = 180224 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x83d4c8ed

Turns out I had a swap file all along but had never activated it! Further, both these commands show that the new volume is appearing as xvdk, not xvdg. Go figure. I guess I had an xvdj volume and it took the next available letter. The mount command also showed me which of the above volumes was in use so I could see which had been added.

Then I used fdisk to create a swap space on it:

$ fdisk /dev/xvdk

Command (m for help): c
DOS Compatibility flag is not set
 
Command (m for help): u
Changing display/entry units to sectors
 
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First sector (2048-4194303, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-4194303, default 4194303):
Using default value 4194303
 
Command (m for help): w
The partition table has been altered!
 
Calling ioctl() to re-read partition table.
Syncing disks.

$ ls /dev/xvdk*

/dev/xvdk  /dev/xvdk1

$ sudo mkswap /dev/xvdk1

Setting up swapspace version 1, size = 2096124 KiB
no label, UUID=0d782596-03e6-48fd-a0fa-2d0e3174f727

$ sudo swapon /dev/xvdk1
The previous command activated our new swap file. To show that we run this command:
$ sudo swapon ‐s

Filename                                Type            Size    Used    Priority
/dev/xvdk1                              partition       2096120 0       -1

Finally to make this swap partition persist after a reboot I added this line to /etc/fstab:

/dev/xvdk1      swap            swap    defaults        0 0

Did it help?
Why yes it did! Now I am using over 900 Mb of swap space, so it was needed pretty badly in fact:

$ sudo swapon ‐s

Filename                                Type            Size    Used    Priority
/dev/xvdk1                              partition       2096120 945552  -1

. And my original motivation – keeping factomd from crashing – was achieved as well. Perhaps it wasn’t so important to use an SSD volume. Mostly the i/o per second was well below 100. But I did have the satisfaction of seeing this burst to 1000, a figure I never could have hit with a traditional drive.

Appendix
Monitoring i/o
These blockchain verifiers can be killers in terms of resource consumption on little servers like mine. The best tool for analyzing what is going on is iostat:

$ iostat ‐xz 10

Linux 2.6.32-131.17.1.el6.x86_64 (ip-10-185-21-116)     05/01/17        _x86_64_        (1 CPU)
 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.92    0.00    0.17    0.24    0.85   97.83
 
Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdj              0.00     0.45    0.22    0.35     6.90     6.41    23.60     0.01   11.87    8.33   14.05   1.43   0.08
xvde              0.00     0.02    0.02    0.57     0.55     4.70     8.93     0.01   15.32    6.62   15.64   2.84   0.17
xvdep3            0.00     0.00    0.00    0.00     0.00     0.00     8.73     0.00    1.95    1.95    0.00   1.94   0.00
xvdk              0.00     0.01    0.02    0.01     0.19     0.16    11.35     0.00    3.23    0.92   10.75   0.19   0.00
 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.65    0.00    6.44   83.93    1.42    4.56
 
Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdj              0.00     1.71  232.42    2.11  3440.68    30.54    14.80     0.43    1.84    1.80    6.95   1.72  40.38
xvde              0.00     0.00   74.59    3.65  3773.45    29.17    48.61     0.31    3.99    3.36   16.91   0.99   7.77
xvdk              5.47   414.93  606.78  230.37  4898.01  5162.39    12.02     1.89    2.26    0.88    5.89   0.18  14.89
 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.63    0.00    4.19   89.55    1.23    2.40
 
Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdj              0.00     0.00  374.08    0.50  5435.98     4.02    14.52     0.84    2.25    2.25    4.33   1.32  49.32
xvde              0.00     0.00    3.52    0.28   185.03     2.23    49.29     0.01    1.66    1.41    4.80   0.72   0.27
xvdk              1.79    99.72  521.96  108.88  4189.94  1668.83     9.29     0.76    1.21    0.72    3.53   0.14   8.95
 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.05    0.00    7.10   72.87    8.46    3.52
 
Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdj              0.00     0.00  338.02    8.25  6812.99    66.04    19.87     0.94    2.72    2.71    3.18   1.44  49.84
xvde              0.00     0.00   52.17    1.76  2317.73    14.07    43.24     0.15    2.72    2.43   11.23   0.67   3.63
xvdk              9.20   381.12 1180.58  256.16  9518.27  5098.24    10.17     1.95    1.36    0.78    4.04   0.14  20.65
...

Always mentally discard the first set of numbers when iostat starts up. It needs to initialize its counters from that reading. But this is chock full of information. The cpu time spent waiting for i/o is too high: 70 – 90 % and a lot of that can be blamed on xvdj (%util column for device xvdj). The way I see it if your i/o were instantaneous this number would drop to 0 and our cpu could be doing other more productive things, hence it shows it is a bottleneck 60% of the time. This also shows my swap, xvdk, being sometimes heavily used and not being too much a bottleneck (20% util).

Then of course there is top, which just confirms that factomd is the resource hog:

$ top

top - 11:45:12 up 1246 days, 14:49,  3 users,  load average: 1.55, 1.73, 1.67
Tasks: 108 total,   1 running, 107 sleeping,   0 stopped,   0 zombie
Cpu(s): 10.6%us,  1.7%sy,  0.0%ni,  4.6%id, 82.3%wa,  0.0%hi,  0.2%si,  0.6%st
Mem:   1695600k total,  1682160k used,    13440k free,     1400k buffers
Swap:  2096120k total,  1003088k used,  1093032k free,    45348k cached
 
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
29702 john      20   0 2956m 1.3g 3984 S 21.4 77.9 490:35.59 factomd
...

Type of cpu
Just for the record here’s the type of cpu you get with an m1 small instance:

$ cat /proc/cpuinfo

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
stepping        : 7
cpu MHz         : 1799.999
cache size      : 20480 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc up rep_good aperfmperf unfair_spinl
ock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm arat epb xsaveopt pln pts
bogomips        : 3599.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

So that’s a single 2 GHz cpu.

Conclusion
We showed how to economically add swap to a CentOS image on Amazon AWS. We showed factomd successfully running on this small instance and we showed linux commands that can be used to monitor resource consumption. Knowing what I know now – that factomd is i/o limited – in addition to creating a swap space I probably would have put its files onto its own SSD drive, which is their recommendation anyway.

References and related
I followed this post for the swap partition creation steps: http://network-howtos.blogspot.com/2015/04/adding-new-swap-partition-to-centos-vm.html

Categories
Admin DNS Network Technologies

Bluecoat ProxySG and DNS using edns seem incompatible

Intro
Imagine your DNS server had this behaviour when queried using dig:

$ dig drjohnstechtalk.com @146.201.145.30

; <<>> DiG 9.9.2-P2 <<>> drjohnstechtalk.com @10.201.145.30
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: FORMERR, id: 48905
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;drjohnstechtalk.com.           IN      A
 
;; Query time: 1 msec
;; SERVER: 10.201.145.30#53(146.201.145.30)
;; WHEN: Fri Feb 24 12:16:42 2017
;; MSG SIZE  rcvd: 48

That would be pretty disturbing, right? The only way to get dig to behave is to turn off edns like this:

$ dig +noedns drjohnstechtalk.com @146.201.145.30

; <<>> DiG 9.9.2-P2 <<>> +noedns drjohnstechtalk.com @10.201.145.30
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31299
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
 
;; QUESTION SECTION:
;drjohnstechtalk.com.           IN      A
 
;; ANSWER SECTION:
drjohnstechtalk.com.    3277    IN      A       50.17.188.196
 
;; Query time: 3 msec
;; SERVER: 10.201.145.30#53(146.201.145.30)
;; WHEN: Fri Feb 24 12:17:00 2017
;; MSG SIZE  rcvd: 53

Nslookup works. But who uses nslookup anyway?

Furthermore, imagine that DNS client and server are on the same subnet, so there is no firewall intermediating their traffic. so we know we can’t blame firewall cutting off large DNS packets, unlike the suggestions made in the references section.

Well, this is exactly the situation in a large company where I consult. The DNS server is unusual: a Bluecoat ProxySG, which can conveniently combine replies from nameservers from two different namespaces.

There does not seem to be an option to handle edns queries correctly on a Bluecoat device.

The client is running SLES version 11. The real question is how will applications behave? Which type of query will they make?

Bluecoat Response
Bluecoat does not support eDNS and gives a response permitted by RFC2671. RFC2671 also encourages clients to account for error responses and drop the use of eDNS in a retry.

References and related
EDNS: What is it all about? is a really good explanation of edns and how it came about, how it’s supposed to work, etc.
This post suggests some scenarios where edns may not work, though it does not address the Bluecoat issue: http://blog.fpweb.net/strange-dns-issues-better-check-out-edns/#.WLBmw3dvDkk
RFC 2671

Categories
Admin Network Technologies Raspberry Pi

Use Raspberry Pi to explore mDNS

Intro
I am confounded by the Bonjour field on my d-Link DCS-931L IP webcam. I should be able to use it to see my desired hostname, but it doesn’t take. Why?

The details
Having a Raspberry Pi on the same network I realized I could at least learn definitively whether or not my new name was being taken, or what the old name was.

You install avahi-discover to do that:

$ sudo apt-get install avahi-discover

Those who follow my blog will realize I am big on Linux command-line, not so much on GUIs. I mention it because unfortunately avahi-discover only works in the GUI. Not having a console I actually had to fire up vncserver and use my vncviewer on my PC! Then I could launch avahi-discover from a terminal window running on the GUI.

The extra fuss was just a few steps anyway, and well worth it.

avahi-discover broke down my home network and all the discovered devices in a very orderly fashion, e.g., the webcam appeared under web servers.

And what did I learn? Indeed, my name had not “taken” for some reason. So the system-supplied name was there instead. For the record that is

dcs931le1a6.local

And testing it:

$ ping dcs931le1a6.local

did indeed show me that it was resolvable by that name form my local network. My PC could reach it by that name as well. I tied to name it DCS-931L-BALL, and I know someone else who did this successfully, and I had even done it in the past, but it was just not taking it this time.

References and related
mDNS is multicast DNS. It’s designed for home networks. It’s pretty common from wjhat I see, yet largely unknown since It people do not encounter it in enterprise environments. As usual Wikipedia has a good article on it.
Superimposing crosshairs on a webcam image.

Categories
Admin Web Site Technologies

Tips for creating a PayPal Donate button for not-not-for-profits

Intro
I am treating my blog as a public service, so I thought it would be appropriate to have a Donation link even though I have not incorporated as a 501c3 non-profit. Is it even possible? Can I keep my anonymity? These were concerns I raised in my head and ones for which there is no good guidance. Short answer: Yes and Yes.

The details
The link to PayPal’s Donate button creation page is pretty well documented. That’s here. Needless to say, you need a PayPal account first!

The PayPal site walks you through a few options. Since Donate is so often associated with not-for-profits I wanted a different word on the button, like Contribute. But customizing the wording was not one of their offered options.

And this is important: you don’t seem to be able to get out of revealing your PayPal email address. It will appear on your Donation page. So what to do?

Well your PayPal account can have up to eight email addresses associated with it. So in my case where the DNS for drjohnstechtalk.com is hosted by GoDaddy (not the web site, mind you, just the DNS), I simply need to set up an arbitrary address like drjohn At drjohnstechtalk DOT com and have it forward to my personal Gmail account. That’s simple enough with GoDaddy’s service, and they don’t even charge extra! Then back to PayPal where you add that address to your account, receive a confirmatory email that they generate, and confirm using their emailed link.

That’s about it. Now you can go back to the button creation page, call it a Donate button, put in your anonymous email address, and let it generate the HTML code. The HTML code is pretty simple and it can be pasted right into a WordPress page if you use or switch to HTML editing.

Here’s a screen shot of my donate page with its PayPal donate button.

OK, I lied. You probably also want a Thank you page in case someone actually does contribute! So build a Thank you page on your blog and record the URL. In WordPress I see you can create this as a hidden page – not appearing in any menu yet still using your theme. Here is my hidden thank you page as an example. Now you really have everything you need to build your Paypal button – one of the requested fields is the URL for your thank you page which we now have.

But is it appropriate?
Well, if like me you only expect the occasional, rare contribution, it’s probably akin to finding yourself in public short of money and asking for a few bucks to take the subway, or whatever… If you’re expecting to get thousands a year you’d better incorporate as a 501c3! As a courtesy I remind people that their contribution is not tax deductible. I guess for low volume stuff it can be treated the way gifts are treated for tax purposes.

Conclusion
drjohnstechtalk.com I hope serves as an example of “paying it forward.” I have benefitted from some good advice on the Internet (and suffered from some terrible advice as well), so I hope to benefit others by providing some clear guidance on topics where I can contribute. But I don’t see that as a barrier to asking for a modest contribution to help defray operational and research costs. In this post I show how I did it. It turns out to be simple, but not well documented on balance.

References and related
PayPal’s Donate Button creation web page with step-by-step instructions.
Care to donate to drjohnstechtalk.com? Here’s my donate page.
My thank you page.

Categories
Admin Web Site Technologies

What I’m working on now: saving $69 a year on my certificate costs: Lets Encrypt

Intro
I recently turned off my GoDaddy web site certificate and implemented one that cost me nothing. This will save me $69 per year.

I wrote up my experience in this article: Idea for free web server certificates: Let’s Encrypt

When I originally wrote that article it was a theoretical concept, but since then I’ve encountered web sites using those certificates so I realized that their CA must be widely accepted now and I decided to try for myself. The ACME (automated certificate management environment) environment was something completely new to me and the terminology a little confusing (as a user I don’t “issue” certificates, so whose perspective does the description take anyway?), but I got it to work in the end with the help of a command-line-based utility called acme.sh. I am actually more comfortable with command-line than with GUI programs. You get greater control and greater understanding.

Example of issuing a certificate using the manual DNS method
If you have full control over DNS but not the web server this approach is a good one.

$ sudo acme.sh ‐‐issue ‐‐dns ‐d www2.drjohnstechtalk.com

2021 update

Now the syntax is this:

acme.sh –issue –dns –yes-I-know-dns-manual-mode-enough-go-ahead-please -d www2.drjohnstechtalk.com

and once you’ve made the DNS entries, this:

acme.sh –issue –dns –yes-I-know-dns-manual-mode-enough-go-ahead-please -d www2.drjohnstechtalk.com –renew

[Thu Feb 23 11:55:52 EST 2017] Single domain='www2.drjohnstechtalk.com'
[Thu Feb 23 11:55:52 EST 2017] Getting domain auth token for each domain
[Thu Feb 23 11:55:52 EST 2017] Getting webroot for domain='www2.drjohnstechtalk.com'
[Thu Feb 23 11:55:52 EST 2017] _w='dns'
[Thu Feb 23 11:55:52 EST 2017] Getting new-authz for domain='www2.drjohnstechtalk.com'                                     [Thu Feb 23 11:55:54 EST 2017] The new-authz request is ok.
[Thu Feb 23 11:55:54 EST 2017] Add the following TXT record:
[Thu Feb 23 11:55:54 EST 2017] Domain: '_acme-challenge.www2.drjohnstechtalk.com'
[Thu Feb 23 11:55:54 EST 2017] TXT value: '7kU6pGgcNRtrPKuN2Wu1TIGS7YZDBhuiumLb9aEJwqc'
[Thu Feb 23 11:55:54 EST 2017] Please be aware that you prepend _acme-challenge. before your domain
[Thu Feb 23 11:55:54 EST 2017] so the resulting subdomain will be: _acme-challenge.www2.drjohnstechtalk.com
[Thu Feb 23 11:55:54 EST 2017] Please add the TXT records to the domains, and retry again.
[Thu Feb 23 11:55:54 EST 2017] Please add '--debug' or '--log' to check more details.
[Thu Feb 23 11:55:54 EST 2017] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.s/drjohnstechtalk.com/drjohnstechtalk.com/g

If you see an error like:

Usage: acme.sh --issue  -d  a.com  -w /path/to/webroot/a.com/

you probably left out the -d argument or something. I do that all the time for some reason. Go back and check your arguments.

Also note the order of arguments is significant! You have to put the ‐d last!

Anyway, after you’ve worked through any syntax errors like that, make the requested DNS entry in the zone file (do not include the quotes around the TXT value). Verify your entry with a command like this:

$ dig txt www2.drjohnstechtalk.com

Then run acme.sh again like this
$ sudo acme.sh ‐‐renew ‐d www2.drjohnstechtalk.com

[Thu Feb 23 12:02:18 EST 2017] Renew: 'www2.drjohnstechtalk.com'
[Thu Feb 23 12:02:18 EST 2017] Single domain='www2.drjohnstechtalk.com'
[Thu Feb 23 12:02:18 EST 2017] Getting domain auth token for each domain
[Thu Feb 23 12:02:19 EST 2017] Verifying:www2.drjohnstechtalk.com
[Thu Feb 23 12:02:22 EST 2017] Success
[Thu Feb 23 12:02:22 EST 2017] Verify finished, start to sign.
[Thu Feb 23 12:02:23 EST 2017] Cert success.
-----BEGIN CERTIFICATE-----
MIIFDjCCA/agAwIBAgISAzFs92uVHvnQCId6qFoaui9gMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xNzAyMjMxNjAyMDBaFw0x
NzA1MjQxNjAyMDBaMCAxHjAcBgNVBAMTFXd3dzIucmVmaW5pc2guYmFzZi51czCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALa9QjcqPSZxXrJmt7kxV9ii
AAI7W0fQdWIk6aDHLfkhsSt55eASOAe05oxnZNqEmcJgXd+2+xUUmm8T5F92l4co
TFiIcMIQ5Jr7Gwg61RnDUe87vI4gctechhhnGcE//4iaJ8Va19exg8HD19dfpdGg
9J2DVnUBqKh4pvItUVe3h8KZOmftQZlLy1QXnEfwEyTj4kGo3phBgOIqh2x15/8Q
tKtM9yr4TKO8clgKLeyXV90ftg4t7kjCdtPx6oRIIq27CO3nwzFq/vFqg6HprE4b
RziUv1yajjw7aHkyhsjWmxdXBlIrokxK3EBrNUo3FUzlLHGGFT2RM1rkbhdaHB0C
AwEAAaOCAhYwggISMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcD
AQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUCvxAhkhpo5Elmge/
3fz5asbprXEwHwYDVR0jBBgwFoAUqEpqYwR93brm0Tm3pkVl7/Oo7KEwcAYIKwYB
BQUHAQEEZDBiMC8GCCsGAQUFBzABhiNodHRwOi8vb2NzcC5pbnQteDMubGV0c2Vu
Y3J5cHQub3JnLzAvBggrBgEFBQcwAoYjaHR0cdrjo2NlcnQuaW50LXgzLmxldHNl
bmNyeXB0Lm9yZy8wIAYDVR0RBBkwF4IVd3d3Mi5yZWZpbmlzaC5iYXNmLnVzMIH+
BgNVHSAEgfYwgfMwCAYGZ4EMAQIBMIHmBgsrBgEEAYLfEwEBATCB1jAmBggrBgEF
BQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwgasGCCsGAQUFBwICMIGe
DIGbVGhpcyBDZXJ0aWZpY2F0ZSBtYXkgb25seSBiZSByZWxpZWQgdXBvbiBieSBS
ZWx5aW5nIFBhcnRpZXMgYW5kIG9ubHkgaW4gYWNjb3JkYW5jZSB3aXRoIHRoZSBD
ZXJ0aWZpY2F0ZSBQb2xpY3kgZm91bmQgYXQgaHR0cHM6Ly9sZXRzZW5jcnlwdC5v
cmcvcmVwb3NpdG9yeS8wDQYJKoZIhvcNAQELBQADggEBADHFgJ3O8v/0zdG48Y1D
ci4wSQqZQSRtalknubVLRC12cr8ct1m0nZ7bxxQnZ/bSFShhcjepFowOfjfdY2ar
GzKP9IRgWavwm259WX5gM5FKUeN/Pjcz++vwUB8ci2+DpZu1tBf0eZcZ6JKsehij
A3lSPb4Ttq/GqDodumeJmvH4/FjCsjIqCJfQQYlz42QdZsLcA2l3oQ8smX0ZK8gh
sIm0SX4RggfiNGuNjw/tkfeC4xfjVNUSjKepUqgsVe63F6gRNjjm+/zsIlPLTdw3
AXGxE71Ww68avev7vRCvp+zPa43DQfVfV98v5M4i0M1/bT0Qh/4hLECma9wBBXb8
jXM=
-----END CERTIFICATE-----
[Thu Feb 23 12:02:23 EST 2017] Your cert is in  /root/.acme.sh/www2.drjohnstechtalk.com/www2.drjohnstechtalk.com.cer
[Thu Feb 23 12:02:23 EST 2017] Your cert key is in  /root/.acme.sh/www2.drjohnstechtalk.com/www2.drjohnstechtalk.com.key
[Thu Feb 23 12:02:23 EST 2017] The intermediate CA cert is in  /root/.acme.sh/www2.drjohnstechtalk.com/ca.cer
[Thu Feb 23 12:02:23 EST 2017] And the full chain certs is there:  /root/.acme.sh/www2.drjohnstechtalk.com/fullchain.cer

More complex example of issuing a SAN certificate using the manual DNS approach

$ ./acme.sh ‐‐issue ‐d johnstechtalk.mobi ‐‐dns ‐d www.johnstechtalk.mobi ‐d drjohnstechtalk.mobi ‐d www.drjohnstechtalk.mobi

[Mon Jan 23 09:21:55 EST 2017] Multi domain='DNS:www.johnstechtalk.mobi,DNS:drjohnstechtalk.mobi,DNS:www.drjohnstechtalk.mobi'
[Mon Jan 23 09:21:55 EST 2017] Getting domain auth token for each domain
[Mon Jan 23 09:21:55 EST 2017] Getting webroot for domain='johnstechtalk.mobi'
[Mon Jan 23 09:21:55 EST 2017] _w='dns'
[Mon Jan 23 09:21:55 EST 2017] Getting new-authz for domain='johnstechtalk.mobi'
[Mon Jan 23 09:21:57 EST 2017] The new-authz request is ok.
[Mon Jan 23 09:21:57 EST 2017] Getting webroot for domain='www.johnstechtalk.mobi'
[Mon Jan 23 09:21:57 EST 2017] _w='dns'
[Mon Jan 23 09:21:57 EST 2017] Getting new-authz for domain='www.johnstechtalk.mobi'
[Mon Jan 23 09:21:57 EST 2017] The new-authz request is ok.
[Mon Jan 23 09:21:57 EST 2017] Getting webroot for domain='drjohnstechtalk.mobi'
[Mon Jan 23 09:21:57 EST 2017] _w='dns'
[Mon Jan 23 09:21:57 EST 2017] Getting new-authz for domain='drjohnstechtalk.mobi'
[Mon Jan 23 09:21:58 EST 2017] The new-authz request is ok.
[Mon Jan 23 09:21:58 EST 2017] Getting webroot for domain='www.drjohnstechtalk.mobi'
[Mon Jan 23 09:21:58 EST 2017] _w='dns'
[Mon Jan 23 09:21:58 EST 2017] Getting new-authz for domain='www.drjohnstechtalk.mobi'
[Mon Jan 23 09:21:58 EST 2017] The new-authz request is ok.
[Mon Jan 23 09:21:58 EST 2017] Add the following TXT record:
[Mon Jan 23 09:21:58 EST 2017] Domain: '_acme-challenge.johnstechtalk.mobi'
[Mon Jan 23 09:21:58 EST 2017] TXT value: 'CDK_dACa_29apV30lc68Vo-mAx3e923ZOh6u-kyhXLo'
[Mon Jan 23 09:21:58 EST 2017] Please be aware that you prepend _acme-challenge. before your domain
[Mon Jan 23 09:21:58 EST 2017] so the resulting subdomain will be: _acme-challenge.johnstechtalk.mobi
[Mon Jan 23 09:21:58 EST 2017] Add the following TXT record:
[Mon Jan 23 09:21:58 EST 2017] Domain: '_acme-challenge.www.johnstechtalk.mobi'
[Mon Jan 23 09:21:58 EST 2017] TXT value: 'UC6JLg1hbXo0oRlYwSyrSRMD5nZgEKgdcIDZfhlqCrU'
[Mon Jan 23 09:21:58 EST 2017] Please be aware that you prepend _acme-challenge. before your domain
[Mon Jan 23 09:21:58 EST 2017] so the resulting subdomain will be: _acme-challenge.www.johnstechtalk.mobi
[Mon Jan 23 09:21:58 EST 2017] Add the following TXT record:
[Mon Jan 23 09:21:58 EST 2017] Domain: '_acme-challenge.drjohnstechtalk.mobi'
[Mon Jan 23 09:21:58 EST 2017] TXT value: 'y8ZCkJ-PXxGbeQFxh7RULCLGKyHH3G7FMFhKpMNF7ow'
[Mon Jan 23 09:21:58 EST 2017] Please be aware that you prepend _acme-challenge. before your domain
[Mon Jan 23 09:21:58 EST 2017] so the resulting subdomain will be: _acme-challenge.drjohnstechtalk.mobi
[Mon Jan 23 09:21:58 EST 2017] Add the following TXT record:
[Mon Jan 23 09:21:58 EST 2017] Domain: '_acme-challenge.www.drjohnstechtalk.mobi'
[Mon Jan 23 09:21:58 EST 2017] TXT value: '8nyb_V7AKaxy0U5pGTKmUejKEXgPv66VKne8yZYZMDg'
[Mon Jan 23 09:21:58 EST 2017] Please be aware that you prepend _acme-challenge. before your domain
[Mon Jan 23 09:21:58 EST 2017] so the resulting subdomain will be: _acme-challenge.www.drjohnstechtalk.mobi
[Mon Jan 23 09:21:59 EST 2017] Please add the TXT records to the domains, and retry again.
[Mon Jan 23 09:21:59 EST 2017] Please add '--debug' or '--log' to check more details.
[Mon Jan 23 09:21:59 EST 2017] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh

Then you add the DNS records they requested in the zone file, for instance,

_acme-challenge.johnstechtalk.mobi IN TXT CDK_dACa_29apV30lc68Vo-mAx3e923ZOh6u-kyhXLo

Then you rerun acme.sh, but with the renew argument:
$ ./acme.sh ‐‐renew ‐d johnstechtalk.mobi
and you should get your SAN certificate issued to you! All the files – private key, intermediate CERT, newly-issued SAN certificate – in ~/.acme.sh/johnstechtalk.mobi/

Of course just put in your own domain names in place of mine. I don’t know how quickly you have to act to put in your TXT records for the DNS authentication challenge. I edited zone files by hand and got them in within a few minutes.

And note the order of the arguments in the original acme.sh command. Often the switch order is immaterial in Linux, but for this command it matters a bit. You have your first mentioned domain, then the dns switch, then your other domain names.

Authorization expires
I got into trouble today, these many months later because my authorization had expired. The error looks like this:

[Sat Feb  3 17:14:00 EST 2018] _CURL='curl -L --silent --dump-header /root/.acme.sh/http.header '
[Sat Feb  3 17:14:01 EST 2018] _ret='0'
[Sat Feb  3 17:14:01 EST 2018] code='404'
[Sat Feb  3 17:14:01 EST 2018] drjohnstechtalk.com:Challenge error: {"type":"urn:acme:error:malformed","detail":"Expired authorization","status": 404}
[Sat Feb  3 17:14:01 EST 2018] Skip for removelevel:
[Sat Feb  3 17:14:01 EST 2018] pid
[Sat Feb  3 17:14:01 EST 2018] _clearupdns
[Sat Feb  3 17:14:01 EST 2018] Dns not added, skip.
[Sat Feb  3 17:14:01 EST 2018] _on_issue_err
[Sat Feb  3 17:14:01 EST 2018] Please add '--debug' or '--log' to check more details.
[Sat Feb  3 17:14:01 EST 2018] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh
[Sat Feb  3 17:14:02 EST 2018] Diagnosis versions:

Turns out there was a 10-month validity to my one-time DNS authorization which I did not know. So I switched to web root authorization and that seemed to work. That went like this:

$ acme.sh ‐‐issue ‐d drjohnstechtalk.com ‐w /webroot/drjohns

And I had to run that as root.


References and related
Idea for free web server certificates: Let’s Encrypt
Info about acme.sh
A great review of the state of hosting companies preparedness for offering SSL circa March 2018 in their hosting packages – so you can compare side-by-side – is in this Naked Security article: https://nakedsecurity.sophos.com/2018/03/12/with-4-months-to-switch-on-https-are-web-hosting-companies-ready/

Categories
Admin Apache Network Technologies Security

drjohnstechtalk now uses HTTP Strict Transport Security, HSTS

Intro
I was reading about a kind of amazingly thorough exploit which could be done using a Raspberry Pi zero. Physical access is required, but the scope of what this guy has figured out and put together is really amazing.

Reading the description I decided is a good exercise in making sure I understand the underlying technologies. One was admittedly something I hadn’t seen before: DNS re-binding. That got me to reading about DNS re-binding, and that got me to looking at defenses against DNS rebinding.

HSTS to the rescue
Since in DNS rebinding you may have either a MITM (man in the middle) or a web site impersonated by a hacker, one defense against it is to use HTTPS. (The hacker will not have access to a web site’s private key and therefore has no way to fake a certificate). But what they can do is redirect users from HTTPS to HTTP, where no certificate is required.

HSTS is designed to make that move tip off the user by complaining to the user. Upon first visit the user gets a cookie that says this site should be https. Subsequent visits then are enforced by the user’s browser that the site accessed must be HTTPS.

drjohnstechtalk update
Two years ago I switched the default way I run my blog web site from HTTP to HTTPS due to the encryption offered by HTTPS, and the fact that search engines penalize HTTP sites.

It seems a natural progression in this age of increasing security awareness to up the ante and now also run HSTS. For me this was easy. Since I run my own apache server I simply needed to add the appropriate HTTP Response header to my server responses.

This is done within the virtual server section of the apache configuration like so:

# Guarantee HTTPS for 1/2 Year including Sub Domains  - DrJ 11/22/16
# see https://itigloo.com/security/how-to-configure-http-strict-transport-security-hsts-on-apache-nginx/
    Header always set Strict-Transport-Security "max-age=15811200; includeSubDomains; preload"

Of course this requires that the apache mod_headers is included.

Results
I test it form a linux server like this:

$ curl ‐i ‐k ‐s https://drjohnstechtalk.com/blog/|head ‐15

HTTP/1.1 200 OK
Date: Tue, 22 Nov 2016 20:30:56 GMT
Server: Apache/2
Strict-Transport-Security: max-age=15811200; includeSubDomains
Vary: Cookie,Accept-Encoding
X-Powered-By: PHP/5.4.43
X-Pingback: https://drjohnstechtalk.com/blog/xmlrpc.php
Last-Modified: Tue, 22 Nov 2016 20:30:58 GMT
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
 
<!DOCTYPE html>
<html lang="en-US">
<head>
<meta charset="UTF-8" />

See that new header Strict-Transport-Security: max-age=15811200; includeSubDomains; preload? That’s the result of what we did. But unless we put that preload at the end it doesn’t verify!

Capture-HSTS-verify-with-error

References and related
drjohnstechtalk is now encrypted – blog posting from 2014
This site has a good description of the requirements for a proper HSTS implementation, and I see that I missed something! https://hstspreload.appspot.com/
You can’t run https without a certificate. I soon will be using the free certificates offered by Let’s Encrypt. Here’s my write-up.