Tag: GoDaddy

Categories
Admin Apache Linux

Cloudflare: an added layer of protection for your personal web site

Intro

I was looking at what Cloudflare could do for my web site. A colleague pointed out that they have a free usage tier which supplies a web application firewall and some anti-bot measures. I checked it out and immedaitely signed up!

The details

What Cloudflare is supplying at no cost (for personal web sites like mine) is amazing. It’s not just a world-class dns service. That would already be amazing. Run dnscheker.org against drjohnstechtalk.com and you will see several different IPs mentioned around the world- just like the big guns! I also get for free some level of mitigation against dns-based attackes.

Web site protections

I don’t fully understand their products so I don’t know what level of protections I am getting in the free tier, but there are at least some! They say they’ve blocked 10 requests in the last few days

Web usage stats

I have to admin using raw linux tools against my apache access file hasn’t bee n the most illuminating until now. Now that I use Cloudflare I get a nice visual presentation showing where (which country) my visitors came from, where the bots come from, how much data was transmitted.

Certificate for HTTPS

Cloudflare automatically takes care of the web site certificate. I had to do nothing at all. So now I can forget my call out to LetsEncrypt. I wonder if GoDaddy is still charging $69 annually for their certificates.

Acceleration

Yeah my web site just feels faster now since the switch. It just does. And Cloudflare stats say that about 30% of the content has been served from their cache – all with zero setup effort on my part! I also believe they use certain tcp acceleration techniques to speed things up.

Cache

And Cloudflare caches some of my objects to boost performance. Considering that I pay for data transfer at Amazon AWS, it’s a fair question to ask if this caching could even be saving me money? I investigated this and found that I get billed maybe $ .02 per GByte, and in a busy month I might use .8 GB or so, so $ .02 per month. So I might occasionally save a penny or so – nothing substantial though!

geoDNS

Even with this free tier you get some geoDNS functionality for free, namely, visitors from around the world will see an IP address which is geographically close to where they are, bossting their performance when using your site. Stop to think about that. That’s a whole lot of infrastructure sophistication that they’re just giving you for free!

Why are they giving this much away?

I think they have the noble aim of improving the security posture of the Internet writ large. Much as letsencrypt greatly accelerated the adoptipon of web page encyrption (https) by making certificates free, Cloudflare hopes to accelerate the adoption of basic security measures for every web site, thereby lifting the security posture of the Internet as a whole. Count me as a booster!

What’s their business model. How will they ever make money?

Well, you’re only supposed to use the free tier for a personal web site, for one. My web sites don’t really have any usage and do not display ads so I think I qualify.

More importantly, the free security protections and acceleration are a kind of teaser and the path to upgrading to profesisonal tier is very visibly marked. So they’re not 100% altruistic.

Why I dislike GoDaddy

Let’s contrast this with offerings from GoDaddy. GoDaddy squeezes cents out of you at every turn. They make it somewhat mysterious what you are actually paying for so they’re counting on fear of screwing up (FOSU, to coin a term). After all, except for the small hit to your wallet, getting that upgraded tier – whois cloaking, anyone? – might be what you need. Who knows. Won’t hurt, right? But I get really tired of it. Amazon AWS is perhaps middle tier in this regards. They do have a free tier virtual server which I used initially. But it really doesn’t work except as a toy. My very modest web site overwhlemed it on too many occasions. So, basically useless. Everything else: you pay for it. But somehow they’re not shaking the pennies out of you at every turn unlike GoDaddy. And AWS even shows you how to optimize your spend.

How I converted my live site to Cloudflare

After signing up for Cloudflare I began to enter my dns domains, e.g., drjohnstechtalk.com, johnstechtalk.com, plsu a few others. They explained how at GoDaddy I had to update the nameserver records for these domains, which I did. Then Cloudflare has to verify these updates. Then my web sites basically stopped working. So I had to switch the encryption mode to full. This is done in Web sites > drjohnstechtalk.com > SSL/TLS > Overview. This mode encrypts the back-end data to my web server, but it accepts a self-signed certificate, no matter if it’s expired or not and no matter who issued it. That is all good because you still get the encrypted channel to your content server.

Then it began to work!

Restoring original visitor IPs to my apache web server logs

Very important to know from a technical standpoint that Cloudflare acts as a reverse proxy to your “content server.” Knowing this, you will also know that your content server’s apache logs get kind of boring because they will only show the Cloudflare IPs. But Cloudflare has a way to fix that so you can see the original IPs, not the Cloudlfare IPs in your apache logs.

Locking down your virtual server

If Internet users can still access the web server of your virtual server directly (bypassing Cloudflare), your security posture is only somewhat improved. To go further you need to use a local firewall. I debated whether to use AWS Network Security Groups or iptables on my centos virtual server. I went with iptables.

I lossely followed this developer article. Did I mention that Cloudflare has an extensive developer community? https://developers.cloudflare.com/fundamentals/get-started/setup/allow-cloudflare-ip-addresses/

Actually I had to install iptables first because I hadn’t been using it. So my little iptables script I created goes like this.

Copy Code 


#!/bin/bash
# from https://developers.cloudflare.com/fundamentals/get-started/setup/allow-cloudflare-ip-addresses/
# For IPv4 addresses
curl -s https://www.cloudflare.com/ips-v4|while read ip; do
 echo adding $ip to iptables restrictions
 iptables -I INPUT -p tcp -m multiport --dports http,https -s $ip -j ACCEPT
done
ip=127.0.0.1
iptables -I INPUT -p tcp -m multiport --dports http,https -s $ip -j ACCEPT
# maybe needed it just once??
#iptables -A INPUT -p tcp -m multiport --dports http,https -j DROP
# list all rules
iptables -S

I believe I just need to run it the one time, not, e.g., after every boot. We’ll soon see. The output looks like this:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -s 127.0.0.1/32 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 172.64.0.0/13 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 104.24.0.0/14 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 104.16.0.0/13 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 162.158.0.0/15 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 198.41.128.0/17 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 197.234.240.0/22 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 188.114.96.0/20 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 190.93.240.0/20 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 108.162.192.0/18 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 141.101.64.0/18 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 103.31.4.0/22 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 103.22.200.0/22 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 103.21.244.0/22 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 173.245.48.0/20 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 80,443 -j DROP

Note that this still leaves ssh open, but that’s ok since it is locked down via Network Security Group rules. No urgent need to change those.

Then I made sure that direct access to my content server freezes, which it does, and that access through the official DNS channels which use Cloudflare still works, which it did. So… all good. The setup was not hard at all. But since I have several hosted web sites for the iptables to make any sense I had to be sure to migrate all my hosted sites over to Cloudflare.

Not GoDaddy

I was dreading migrating my other zones (dns domains) over to Cloudflare. Still being in the GoDaddy mindframe I figured, sure, Cloudflare will permit me one zone for free, but then charge me for a second one.

So I plunged ahead. johnstechtalk.com. No charge!

And a third one: vmanswer.com. Also no charge!

And a fourth, and a fifth and a sixth.

I thought perhaps five will be the threshold. But it wasn’t. I only have six “zones” as Cloudflare now calls them. But they are all in my account and all free. Big relief. This is like the anti-GoDaddy.

DNS changes

Making DNS changes is quite fast. The changes are propagated within a minute or two.

api access

Everything you can do in the GUI you can do through the api. I had previously created and shared some model python api scripts.

ipv6

As if all the above weren’t already enough, I see Cloudflare also gives my web site accessibility via ipv6:

$ dig +short aaaa drjohnstechtalk.com

2606:4700:3035::ac43:ad17
2606:4700:3031::6815:3fea

I guess it’s accessible through ipv6 but I haven’t quite proven that yet.

Mail forwarding

I originally forgot that I had set up mail forwarding on GoDaddy. It was one of the few free things you could get. I think they switched native Outlook or something so my mail forwarding wasn’t working. On a lark I checked if Cloudflare has complementary mail forwarding for my domains. And they do! So that’s cool – another free service I will use.

Sending mail FROM this Cloudflare domain using your Gmail account

This is more tricky than simple mail forwarding. But I think I’ve got it working now. You use Gmail’s own server (smtp.gmail.com) as your relay. You also need to set up an app password for Gmail. Even though you need to specify a device such as Windows, it seems once enabled, you can send from this new account from any of your devices. I’ve found that you also need to update your TXT record (see link below) with an expanded SPF information:

v=spf1 include:_spf.google.com include:_spf.mx.cloudflare.net ~all

In words it means the Google and Cloudflare sending servers are authorized to sends emails with this domain in the sender field, mail from elsewhere will be marked.

Even after all that I wasn’t seeing my sent message at work where Microsoft 365 is in use. It landed in the Junk folder! Why? The sending email “appears similar to someone who previously sent you email, but may not be that person.” Since I am a former mail admin I am sympathetic to what they’re trying to do – help hapless users avoid phishing; because it’s true – the characters in my test email did bear similarities to my regular email. My regular email is first_name.last_name @ gmail.com, while mail from this domain was first_name @ last_name + s .com Mail sent to a fellow Gmail user suffered no such fate however. Different providers, different approaches. So I can accept that. Once it’s set up you get a drop-down menu of sending addresses every time you compose a new message! The detailed instructions are at the Cloudflare community site.

Cost savings using Cloudflare

Suppose like me you only use GoDaddy as your registrar and get all your other services in some other way. Well, Cloudflare began to pitch me on transferring my domains to them. I thought, Aha, this is the moment they will make money off me. So I read their pitch. Their offer is to bill me for the charges they incur from ICANN or wherever, i.e., pass-through charges without any additional middleman overhead. It’s like, what? So let’s say at GoDaddy I pay $22 per year per domain. Well with Cloudflare I’d be paying something like $10 per year. For one domain I wouldn’t bother, but since I have more than five, I will be bothering and gladly leaving GoDaddy in the dust. I have just transferred the first two domains. GoDaddy seems to drag out the process as long as possible. I found I could expedite it by approving the transfer in the GoDaddy portal (https://dcc.godaddy.com/control/transfers). The trick there is that that one URL looks very different depending on whether or not a domain transfer is pending. If GoDaddy perceives a domain transfer has been initiated by an other registrar, it will show that page with a Transfer In and Transfer Out tabs. Just select Transfer Out and approve your domain for transfer. Then the transfer happens within five minutes. Otherwise that page is shown with no possibility to do a transfer out. So I guess you have to be patiennt, refresh it, or I don’t know what to get it to draw correctly. Once approved in the GoDaddy transfer out portal, Cloudflare had them within 5 minutes. It’s not super-easy to do a transfer, but also not impossble.

In typical GoDaddy style, executing a domain transfer to another registrar seems essentially impossible if you use their latest Domain portfolio app. Fortunately I eventually noticed the option to switch from “beta” to the old Domain manager, which still has the option and looks a bit more like their documentation. I’ve generated auth codes and unlocked, etc. And I even see the correct domain status (ok as opposed to client transfer prohibited) when I do a whois, but now Cloudflare, which is usually so quick to execute, seems to be lagging in recognizing that the domains have been unlocked and suggests to check back in some hours. Weird. The solution here was to provide my credit card info. Even 12 hours later I was having this trouble where it said none of my domains were eligible for transfer. As soon as I provided my payment information, it recognized two of my domains as eligible for transfer. In other cases Cloudflare recognized that domains were unlocked in a matter of 15 minutes or so. It may help to first unlock the domain in GoDaddy, then to view it in Cloudflare. Not sure.

A plug for GoDaddy

As my favorite sport seems to be bashing GoDaddy I wanted to balance that out and say a few kind words about them. Someone in my houisehold just started a job with a startup who uses GoDaddy. It provides desktop Outlook Email, MS Teams, Sharepoint, helps with consulting, etc. And on day one this person was up and running. So if you use their services, they definitely offer value. My issue is that I tried to restrict my usage to just one service – domain registrar – and they pushed me to use it more extensively, which I resisted. But for a small business which needs those thnigs, it’s fine.

How many domains are you sharing your IP with?

The thnig with Cloudflare is that they assign you to a couple of their IP addresses, often beginning with either 172.67 or 104…. . Now did you ever wonder with how many other web sites you’re sharing those IPs? If not, you should! I found a tool that provides the answer: https://dnslytics.com/ So for this free tier they seem to keep the number around 500 unique domains per IP! Yes that’s a lot, but I’d only be concerned if there was evidence of service degradation, which so far I have not seen. What’s nice about the dnsyltics site is that it lists a few of the domains – far from all of them, but at least it’s 20 or 30 – associated with a given IP. That can be helpful during truobleshooting.

Conclusion

What Cloudflare provides for protective and performance services represents a huge forward advance in the state of the art. They do not niggle you for extra charges (entice is more the word here) for Fear of Screwing Up.

All in all, I am amazed, and I am something of an insider – a professional user of such services. So I heartily endorse using Cloudflare for all personal web servers. I have not been sponsored or even in contact with Cloudflare, by the way!

References and related

Cloudlfare tip: Restoring original visitor IPs to your apache web server.

Locking your virtual server down to just Cloudflare IPs: https://developers.cloudflare.com/fundamentals/get-started/setup/allow-cloudflare-ip-addresses/

Using the Cloudflare python api: working examples

Sending Gmail with your Cloudlflare domain as sending address

Cloudflare’s analysis of the exploit HTTP/2 Rapid Reset is extremely detailed. See https://blog.cloudflare.com/zero-day-rapid-reset-http2-record-breaking-ddos-attack/ and https://blog.cloudflare.com/technical-breakdown-http2-rapid-reset-ddos-attack/ .

I remember being so excited to discover free certificates from LetsEncrypt.

A good explanation of SPF records

Turn an IP addres into a list of associated domain names: https://dnslytics.com/

Categories
DNS Linux Perl Raspberry Pi Web Site Technologies

Roll your own domain drop catching service using GoDaddy

Intro
I’m after a particular domain and have been for years. But as a matter of pride I don’t want to overpay for it, so I don’t want to go through an auction. There are services that can help grab a DNS domain immediately after it expires, but they all want $$. That may make sense for high-demand domains. Mine is pretty obscure. I want to grab it quickly – perhaps within a few seconds after it becomes available, but I don’t expect any competition for it. That is a description of domain drop catching.

Since I am already using GoDaddy as my registrar I thought I’d see if they have a domain catching service. They don’t which is strange because they have other specialized domain services such as domain broker. They have a service which is designed for much the same purpose, however, called backorder. That creates an auction bid for the domain before it has expired. The cost isn’t too bad, but since I started down a different path I will roll my own. Perhaps they have an API which can be used to create my own domain catcher? It turns out they do!

It involves understanding how to read a JSON data file, which is new to me, but otherwise it’s not too bad.

The domain lifecycle
This graphic from ICANN illustrates it perfectly for your typical global top-level domain such as .com, .net, etc:
gtld-lifecycle

To put it into words, there is the
initial registration,
optional renewals,
expiration date,
auto-renew grace period of 0 – 45 days,
redemption grace period of 30 days,
pending delete of 5 days, and then
it’s released and available.

So in domain drop catching we are keenly interested in being fully prepared for the pending delete five day window. From an old discussion I’ve read that the precise time .com domains are released is usually between 2 -3 PM EST.

A word about the GoDaddy developer site
It’s developer.godaddy.com. It looks like one day it will be a great site, but for now it is wanting in some areas. Most of the menu items are duds and are placeholders. Really there are only three (mostly) working sections: get started, documentation and demo. Get started is only a few words and one slender snippet of Ajax code, and the demo itself also extremely limited, so the only real resource they provide is Documentation. Documentation is designed as an active documentation that you can try out functions with your data. You run it and it shows you all the needed request headers and data as well as the received response. The thing is that it’s very finicky. It’s supposed to show all the available functions but I couldn’t get it to work under Firefox. And with Internet Explorer/Edge it only worked about half the time. It seems to help to access it with a newly launched browser. The documentation, as good as it is, leaves some things unsaid. I have found:

https://api.ote-godaddy.com/ – use for TEST. Maybe ote stands for optional test environment?
https://api.godaddy.com/ – for production (what I am calling PROD)

The TEST environment does not require authentication for some things that PROD does. This shell script for checking available domains, which I call available-test.sh, works in TEST but not in PROD:

#!/bin/sh
# pass domain as argument
# apparently no AUTH is rquired for this one
curl -k 'https://api.ote-godaddy.com/v1/domains/available?domain='$1'&checkType=FAST&forTransfer=false'

In PROD I had to insert the authorization information – the key and secret they showed me on the screen. I call this script available.sh.

#!/bin/sh
# pass domain as argument
curl -s -k -H 'Authorization: sso-key *******8m_PwFAffjiNmiCUrKe******:**FF73L********' 'https://api.godaddy.com/v1/domains/available?domain='$1'&checkType=FULL&forTransfer=false'

I found that my expiring domain produced different results about five days after expiring if I used checkType of FAST versus checkType of FULL – and FAST was wrong. So I learned you have to use FULL to get an answer you can trust!

Example usage of an available domain

$ ./available.sh dr-johnstechtalk.com

{"available":true,"domain":"dr-johnstechtalk.com","definitive":false,"price":11990000,"currency":"USD","period":1}

2nd example – a non-available domain
$ ./available.sh drjohnstechtalk.com

{"available":false,"domain":"drjohnstechtalk.com","definitive":true,"price":11990000,"currency":"USD","period":1}

Example JSON file
I had to do a lot of search and replace to preserve my anonymity, but I feel this post wouldn’t be complete without showing the real contents of my JSON file I am using for both validate, and, hopefully, as the basis for my API-driven domain purchase:

{
  "domain": "dr-johnstechtalk.com",
  "renewAuto": true,
  "privacy": false,
  "nameServers": [
  ],
  "consent": {
    "agreementKeys": ["DNRA"],
    "agreedBy": "50.17.188.196",
    "agreedAt": "2016-09-29T16:00:00Z"
  },
  "period": 1,
  "contactAdmin": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    },
    "phone": "+1.5555551212"
  },
  "contactBilling": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    },
    "phone": "+1.5555551212"
  },
  "contactRegistrant": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "phone": "+1.5555551212",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    }
  },
  "contactTech": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "phone": "+1.5555551212",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    }
  }
}

Note the agreementkeys value: DNRA. GoDaddy doesn’t document this very well, but that is what you need to put there! Note also that the nameservers are left empty. I asked GoDaddy and that is what they advised to do. The other values are pretty much what you’d expect. I used my own server’s IP address for agreedBy – use your own IP. I don’t know how important it is to get the agreedAt date close to the current time. I’m going to assume it should be within 24 hours of the current time.

How do we test this JSON input file? I wrote a validate script for that I call validate.sh.

#!/bin/sh
# DrJ 9/2016
# godaddy-json-register was built using GoDaddy's documentation at https://developer.godaddy.com/doc#!/_v1_domains/validate
jsondata=`tr -d '\n' < godaddy-json-register`
 
curl -i -k -H 'Authorization: sso-key *******8m_PwFAffjiNmiCUrKe******:**FF73L********' -H 'Content-Type: application/json' -H 'Accept: application/json' -d "$jsondata" https://api.godaddy.com/v1/domains/purchase/validate

Run the validate script
$ ./validate.sh

HTTP/1.1 100 Continue
 
HTTP/1.1 100 Continue
Via: 1.1 api.godaddy.com
 
HTTP/1.1 200 OK
Date: Thu, 29 Sep 2016 20:11:33 GMT
X-Powered-By: Express
Vary: Origin,Accept-Encoding
Access-Control-Allow-Credentials: true
Content-Type: application/json; charset=utf-8
ETag: W/"2-mZFLkyvTelC5g8XnyQrpOw"
Via: 1.1 api.godaddy.com
Transfer-Encoding: chunked

Revised versions of the above scripts
So we can pass the domain name as argument I revised all the scripts. Also, I provide an agreeddAt date which is current.

The data file: godaddy-json-register

{
  "domain": "DOMAIN",
  "renewAuto": true,
  "privacy": false,
  "nameServers": [
  ],
  "consent": {
    "agreementKeys": ["DNRA"],
    "agreedBy": "50.17.188.196",
    "agreedAt": "DATE"
  },
  "period": 1,
  "contactAdmin": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    },
    "phone": "+1.5555551212"
  },
  "contactBilling": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    },
    "phone": "+1.5555551212"
  },
  "contactRegistrant": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "phone": "+1.5555551212",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    }
  },
  "contactTech": {
    "nameFirst": "Dr","nameLast": "John",
    "email": "[email protected]",
    "phone": "+1.5555551212",
    "addressMailing": {
      "address1": "555 Piney Drive",
      "city": "Smallville","state": "New Jersey","postalCode": "55555",
      "country": "US"
    }
  }
}

validate.sh

#!/bin/sh
# DrJ 10/2016
# godaddy-json-register was built using GoDaddy's documentation at https://developer.godaddy.com/doc#!/_v1_domains/validate
# pass domain as argument
# get date into accepted format
domain=$1
date=`date -u --rfc-3339=seconds|sed 's/ /T/'|sed 's/+.*/Z/'`
jsondata=`tr -d '\n' < godaddy-json-register`
jsondata=`echo $jsondata|sed 's/DATE/'$date'/'`
jsondata=`echo $jsondata|sed 's/DOMAIN/'$domain'/'`
#echo date is $date
#echo jsondata is $jsondata
curl -i -k -H 'Authorization: sso-key *******8m_PwFAffjiNmiCUrKe******:**FF73L********' -H 'Content-Type: application/json' -H 'Accept: application/json' -d "$jsondata" https://api.godaddy.com/v1/domains/purchase/validate

available.sh
No change. See listing above.

purchase.sh
Exact same as validate.sh, with just a slightly different URL. I need to test that it really works, but based on my reading I think it will.

#!/bin/sh
# DrJ 10/2016
# godaddy-json-register was built using GoDaddy's documentation at https://developer.godaddy.com/doc#!/_v1_domains/purchase
# pass domain as argument
# get date into accepted format
domain=$1
date=`date -u --rfc-3339=seconds|sed 's/ /T/'|sed 's/+.*/Z/'`
jsondata=`tr -d '\n' < godaddy-json-register`
jsondata=`echo $jsondata|sed 's/DATE/'$date'/'`
jsondata=`echo $jsondata|sed 's/DOMAIN/'$domain'/'`
#echo date is $date
#echo jsondata is $jsondata
curl -s -i -k -H 'Authorization: sso-key *******8m_PwFAffjiNmiCUrKe******:**FF73L********' -H 'Content-Type: application/json' -H 'Accept: application/json' -d "$jsondata" https://api.godaddy.com/v1/domains/purchase

Putting it all together
Here’s a looping script I call loop.pl. I switched to perl because it’s easier to do certain control operations.

#!/usr/bin/perl
#DrJ 10/2016
$DEBUG = 0;
$status = 0;
open STDOUT, '>', "loop.log" or die "Can't redirect STDOUT: $!";
                   open STDERR, ">&STDOUT"     or die "Can't dup STDOUT: $!";
 
                   select STDERR; $| = 1;      # make unbuffered
                   select STDOUT; $| = 1;      # make unbuffered
# edit this and change to your about-to-expire domain
$domain = "dr-johnstechtalk.com";
while ($status != 200) {
# show that we're alive and working...
  print "Now it's ".`date` if $i++ % 10 == 0;
  $hr = `date +%H`;
  chomp($hr);
# run loop more aggressively during times of day we think Network Solutions releases domains back to the pool, esp. around 2 - 3 PM EST
  $sleep = $hr > 11 && $hr < 16 ? 1 : 15;
  print "Hr,sleep: $hr,$sleep\n" if $DEBUG;
  $availRes = `./available.sh $domain`;
# {"available":true,"domain":"dr-johnstechtalk.com","definitive":false,"price":11990000,"currency":"USD","period":1}
  print "$availRes\n" if $DEBUG;
  ($available) = $availRes =~ /^\{"available":([^,]+),/;
  print "$available\n" if $DEBUG;
  if ($available eq "false") {
    print "test comparison OP for false result\n" if $DEBUG;
  } elsif ($available eq "true") {
# available value of true is extremely unreliable with many false positives. Confirm availability by making a 2nd call
    print "available.sh results: $availRes\n";
    $availRes = `./available.sh $domain`;
    print "available.sh re-test results: $availRes\n";
    ($available2) = $availRes =~ /^\{"available":([^,]+),/;
    next if $available2 eq "false";
# We got two available eq true results in a row so let's try to buy it!
    print "$domain is very likely available. Trying to buy it at ".`date`;
    open(BUY,"./purchase.sh $domain|") || die "Cannot run ./purchase.pl $domain!!\n";
    while() {
# print out each line so we can analyze what happened
      print ;
# we got it if we got back
# HTTP/1.1 200 OK
      if (/1.1 200 OK/) {
        print "We just bought $domain at ".`date`;
        $status = 200;
     }
    } # end of loop over results of purchase
    close(BUY);
    print "\n";
    exit if $status == 200;
  } else {
    print "available is neither false nor true: $available\n";
  }
  sleep($sleep);
}

Running the loop script
$ nohup ./loop.pl > loop.log 2>&1 &
Stopping the loop script
$ kill ‐9 %1

Description of loop.pl
I gotta say this loop script started out as a much simpler script. I fortunately started on it many days before my desired domain actually became available so I got to see and work out all the bugs. Contributing to the problem is that GoDaddy’s API results are quite unreliable. I was seeing a lot of false positives – almost 20%. So I decided to require two consecutive calls to available.sh to return true. I could have required available true and definitive true, but I’m afraid that will make me late to the party. The API is not documented to that level of detail so there’s no help there. But so far what I have seen is that when available incorrectly returns true, simultaneously definitive becomes false, whereas all other times definitive is true.

Results of running an earlier and simpler version of loop.pl
This shows all manner of false positives. But at least it never allowed me to buy the domain when it wasn’t available.

Now it's Wed Oct  5 15:20:01 EDT 2016
Now it's Wed Oct  5 15:20:19 EDT 2016
Now it's Wed Oct  5 15:20:38 EDT 2016
available.sh results: {"available":true,"domain":"dr-johnstechtalk.com","definitive":false,"price":11990000,"currency":"USD","period":1}
dr-johnstechtalk.com is available. Trying to buy it at Wed Oct  5 15:20:46 EDT 2016
HTTP/1.1 100 Continue
 
HTTP/1.1 100 Continue
Via: 1.1 api.godaddy.com
 
HTTP/1.1 422 Unprocessable Entity
Date: Wed, 05 Oct 2016 19:20:47 GMT
X-Powered-By: Express
Vary: Origin,Accept-Encoding
Access-Control-Allow-Credentials: true
Content-Type: application/json; charset=utf-8
ETag: W/"7d-O5Dw3WvJGo8h30TqR7j8zg"
Via: 1.1 api.godaddy.com
Transfer-Encoding: chunked
 
{"code":"UNAVAILABLE_DOMAIN","message":"The specified `domain` (dr-johnstechtalk.com) isn't available for purchase","name":"ApiError"}
Now it's Wed Oct  5 15:20:58 EDT 2016
Now it's Wed Oct  5 15:21:16 EDT 2016
Now it's Wed Oct  5 15:21:33 EDT 2016
available.sh results: {"available":true,"domain":"dr-johnstechtalk.com","definitive":false,"price":11990000,"currency":"USD","period":1}
dr-johnstechtalk.com is available. Trying to buy it at Wed Oct  5 15:21:34 EDT 2016
HTTP/1.1 100 Continue
 
HTTP/1.1 100 Continue
Via: 1.1 api.godaddy.com
 
HTTP/1.1 422 Unprocessable Entity
Date: Wed, 05 Oct 2016 19:21:36 GMT
X-Powered-By: Express
Vary: Origin,Accept-Encoding
Access-Control-Allow-Credentials: true
Content-Type: application/json; charset=utf-8
ETag: W/"7d-O5Dw3WvJGo8h30TqR7j8zg"
Via: 1.1 api.godaddy.com
Transfer-Encoding: chunked
 
{"code":"UNAVAILABLE_DOMAIN","message":"The specified `domain` (dr-johnstechtalk.com) isn't available for purchase","name":"ApiError"}
Now it's Wed Oct  5 15:21:55 EDT 2016
Now it's Wed Oct  5 15:22:12 EDT 2016
Now it's Wed Oct  5 15:22:30 EDT 2016
available.sh results: {"available":true,"domain":"dr-johnstechtalk.com","definitive":false,"price":11990000,"currency":"USD","period":1}
dr-johnstechtalk.com is available. Trying to buy it at Wed Oct  5 15:22:30 EDT 2016
...

These results show why i had to further refine the script to reduce the frequent false positives.

Review
What have we done? Our looping script, loop.pl, loops more aggressively during the time of day we think Network Solutions releases expired .com domains (around 2 PM EST). But just in case we’re wrong about that we’ll run it anyway at all hours of the day, but just not as quickly. So during the aggressive period we sleep just one second between calls to available.sh. When the domain finally does become available we call purchase.sh on it and exit and we write some timestamps and the domain we’ve just registered to our log file.

Performance
Miserable. This API seems tuned for relative ease-of-use, not speed. The validate call often takes, oh, say 40 seconds to return! I’m sure the purchase call will be no different. For a domainer that’s a lifetime. So any strategy that relies on speed had better turn to a registrar that’s tuned for it. GoDaddy I think is more aiming at resellers of their services.

Don’t have a linux environment handy?
Of course I’m using my own Amazon AWS server for this, but that needn’t be a barrier to entry. I could have used one of my Raspberry Pi’s. Probably even Cygwin on a Windows PC could be made to work.

Appendix A
How to remove all newline characters from your JSON file

Let’s say you have a nice JSON file which was created for you from the Documentation exercises called godaddy-json-register. It will contain lots of newline (“\n”) characters, assuming you’re using a Linux server. Remove them and put the output into a file called compact-json:

$ tr ‐d ‘\n'<godaddy‐json‐register>compact‐json

I like this because then I can still use curl rather than wget to make my API calls.

Appendix B
What an expiring domain looks like in whois
Run this from a linux server
$ whois <expiring‐domain.com>

Domain Name: expiring-domain.com
...
Creation Date: 2010-09-28T15:55:56Z
Registrar Registration Expiration Date: 2016-09-27T21:00:00Z
...
Domain Status: clientDeleteHold
Domain Status: clientDeleteProhibited
Domain Status: clientTransferProhibited
...

You see that Domain Status: clientDeleteHold? You don’t get that for regular domains whose registration is still good. They’ll usually have the two lines I show below that, but not that one. This is shown for my desired domain just a few days after its official expiration date.

2020 update

Four years later, still hunting for that domain – I am very patient! So I dusted off the program described here. Suprisingly, it all still works. Except maybe the JSON file. The onlything wrong with that was the lack of nameservers. I added some random GoDaddy nameservers and it seemed all good.

Conclusion
We show that GoDaddy’s API works and we provide simple scripts which can automate what is known as domain dropcatching. This approach should attempt to register a domain within a cople seconds of its being released – if we’ve done everything right. The GoDaddy API results are a little unstable however.

References and related
If you don’t mind paying, Fabulous.com has a domain drop catching service.
ICANN’s web site with the domain lifecycle infographic.
GoDaddy’s API documentation: http://developer.godaddy.com/
More about Raspberry Pi: http://raspberrypi.org/
I really wouldn’t bother with Cygwin – just get your hands on real Linux environment.
Curious about some of the curl options I used? Just run curl ‐‐help. I left out the description of the switches I use because it didn’t fit into the narrative.
Something about my Amazon AWS experience from some years ago.
All the perl tricks I used would take another blog post to explain. It’s just stuff I learned over the years so it didn’t take much time at all.
People who buy and sell domains for a living are called domainers. They are professionals and my guide will not make you competitive with them.

Categories
Admin Apache CentOS Security

drjohnstechtalk.com is now an encrypted web site

Intro
I don’t overtly chase search engine rankings. I’m comfortable being the 2,000,000th most visited site on the Internet, or something like that according to alexa. But I still take pride in what I’m producing here. So when I read a couple weeks ago that Google would be boosting the search rank of sites which use encryption, I felt I had to act. For me it is equally a matter of showing that I know how to do it and a chance to write another blog posting which may help others.

Very, very few people have my situation, which is a self-hosted web site, but still there may be snippets of advice which may apply to other situations.

I pulled off the switch to using https instead of http last night. The detail of how I did it are below.

The details
Actually there was nothing earth-shattering. It was a simple matter of applying stuff i already know how to do and putting it all together. of course I made some glitches along the way, but I also resolved them.

First the CERT
I was already running an SSL web server virtual server at https://drjohnstechtalk.com/ , but it was using a self-signed certificate. Being knowledgeable about certificates, I knew the first and easiest thing to do was to get a certificate (cert) issued by a recognized certificate authority (CA). Since my domain was bought from GoDaddy I decided to get my SSL certificate form them as well. It’s $69.99 for a one-year cert. Strangely, there is no economy of scale so a two-year cert costs exactly twice as much. i normally am a strong believer in two-year certs simply to avoid the hassle of renewing, etc, but since I am out-of-practice and feared I could throw my money away if I messed up the cert, I went with a one-year cert this time. It turns out I had nothing to fear…

Paying for a certificate at GoDaddy is easy. Actually figuring out how to get your certificate issued by them? Not so much. But I figured out where to go on their web site and managed to do it.

Before the CERT, the CSR
Let’s back up. Remember I’m self-hosted? I love being the boss and having that Linux prompt on my CentOS VM. So before I could buy a legit cert I needed to generate a private key and certificate signing request (CSR), which I did using openssl, having no other fancy tools available and being a command line lover.

To generate the private key and CSR with one openssl command do this:

$ openssl req -new -nodes -out myreq.csr

It prompts you for field values, e.g.:

Here’s how that dialog went:

Generating a 2048 bit RSA private key
.............+++
..............................+++
writing new private key to 'privkey.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:New Jersey
Locality Name (eg, city) [Default City]:
Organization Name (eg, company) [Default Company Ltd]:drjohnstechtalk.com
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:drjohnstechtalk.com
Email Address []:
 
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

and the files it created:

$ ls -ltr|tail -2

-rw-rw-r-- 1 john john     1704 Aug 23 09:52 privkey.pem
-rw-rw-r-- 1 john john     1021 Aug 23 09:52 myreq.csr

Before shipping it off to a CA you really ought to examine the CSR for accuracy. Here’s how:

$ openssl req -text -in myreq.csr

Certificate Request:
    Data:
        Version: 0 (0x0)
        Subject: C=US, ST=New Jersey, L=Default City, O=drjohnstechtalk.com, CN=drjohnstechtalk.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:e9:04:ab:7e:e1:c1:87:44:fb:fe:09:e1:8d:e5:
                    29:1c:cb:b5:e8:d0:cc:f4:89:67:23:ab:e5:e7:a6:
                    ...

What are we looking for, anyways? Well, the modulus should be the same as it is for the private key. To list the modulus of your private key:

$ openssl rsa -text -in privkey.pem|more

The other things I am looking for is the common name (CN) which has to exactly match the DNS name that is used to access the secure site.

I’m not pleased about the Default City, but I didn’t want to provide my actual city. We’ll see it doesn’t matter in the end.

For some CAs the Organization field also matters a great deal. Since I am a private individual I decided to use the CN as my organization and that was accepted by GoDaddy. So Probably its value also doesn’t matter.

the other critical thing is the length of the public key, 2048 bits. These days all keys should be 2048 bits. some years ago 1024 bits was perfectly fine. I’m not sure but maybe older openssl releases would have created a 1024 bit key length so that’s why you’ll want to watch out for that.

Examine the CERT
GoDaddy issued the certificate with some random alpha-numeric filename. i renamed it to something more suitable, drjohnstechtalk.crt. Let’s examine it:

$ openssl x509 -text -in drjohnstechtalk.crt|more

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            27:ab:99:79:cb:55:9f
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., OU=http://certs.godaddy.com/repository/, CN=Go Daddy Secure Certificate Authority - G2
        Validity
            Not Before: Aug 21 00:34:01 2014 GMT
            Not After : Aug 21 00:34:01 2015 GMT
        Subject: OU=Domain Control Validated, CN=drjohnstechtalk.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:e9:04:ab:7e:e1:c1:87:44:fb:fe:09:e1:8d:e5:
                    29:1c:cb:b5:e8:d0:cc:f4:89:67:23:ab:e5:e7:a6:
                     ...

So we’re checking that common name, key length, and what if any organization they used (in the Subject field). Also the modulus should match up. Note that they “cheaped out” and did not provide www.drjohnstechtalk.com as an explicit alternate name! In my opinion this should mean that if someone enters the URL https://www.drjohnstechtalk.com/ they should get a certificate name mismatch error. In practice this does not seem to happen – I’m not sure why. Probably the browsers are somewhat forgiving.

The apache side of the house
I don’t know if this is going to make any sense, but here goes. To begin I have a bare-bones secure virtual server that did essentially nothing. So I modified it to be an apache redirect factory and to use my brand shiny new legit certificate. Once I had that working I planned to swap roles and filenames with my regular configuration file, drjohns.conf.

Objective: don’t break existing links
Why the need for a redirect factory? This I felt as a matter of pride is important: it will permit all the current links to my site, which are all http, not https, to continue to work! That’s a good thing, right? Now most of those links are in search engines, which constantly comb my pages, so I’m sure over time they would automatically be updated if I didn’t bother, but I just felt better about knowing that no links would be broken by switching to https. And it shows I know what I’m doing!

The secure server configuration file on my server is in /etc/apache2/sites-enabled/drjohns.secure.conf. It’s an apache v 2.2 server. I put all the relevant key/cert/intermediate cert files in /etc/apache2/certs. the private key’s permissions were set to 600. The relevant apache configuration directives are here to use this CERT along with the GoDaddy intermediate certificates:

 
   SSLEngine on
    SSLCertificateFile /etc/apache2/certs/drjohnstechtalk.crt
    SSLCertificateKeyFile /etc/apache2/certs/drjohnstechtalk.key
    SSLCertificateChainFile /etc/apache2/certs/gd_bundle-g2-g1.crt

I initially didn’t include the intermediate certs (chain file), which again in my experience should have caused issues. Once again I didn’t observe any issues from omitting it, but my experience says that it should be present.

The redirect factory setup
For the redirect testing I referred to my own blog posting (which I think is underappreciated for whatever reason!) and have these lines:

# I really don't think this does anything other than chase away a scary warning in the error log...
        RewriteLock ${APACHE_LOCK_DIR}/rewrite_lock
<VirtualHost *:80>
 
        ServerAdmin webmaster@localhost
        ServerName      www.drjohnstechtalk.com
        ServerAlias     drjohnstechtalk.com
        ServerAlias     johnstechtalk.com
        ServerAlias     www.johnstechtalk.com
        ServerAlias     vmanswer.com
        ServerAlias     www.vmanswer.com
# Inspired by the dreadful documentation on http://httpd.apache.org/docs/2.0/mod/mod_rewrite.html
        RewriteEngine on
        RewriteMap  redirectMap prg:redirect.pl
        RewriteCond ${redirectMap:%{HTTP_HOST}%{REQUEST_URI}} ^(.+)$
# %N are backreferences to RewriteCond matches, and $N are backreferences to RewriteRule matches
        RewriteRule ^/.* %1 [R=301,L]

Pages look funny after the switch to SSL
One of the first casualties after the switch to SSL was that my pages looked funny. I know from general experience that this can happen if there is hard-wired links to http URLs, and that is what I observed in the page source. In particular my WP-Syntax plugin was now bleeding verbatim entries into the columns to the right if the PRE text contained long lines. Not pretty. The page source mostly had https includes, but in one place it did not. It had:

<link rel="stylesheet" href="http://drjohnstechtalk.com/blog/wp-content/plugins/wp-syntax/wp-syntax.css"

I puzzled over where that originated and I had a few ideas which didn’t turn out so well. For instance you’d think inserting this into wp-config.php would have worked:

define( 'WP_CONTENT_URL','https://drjohnstechtalk.com/blog/wp-content/');

But it had absolutely no effect. Finally I did an RTFM – the M being http://codex.wordpress.org/Editing_wp-config.php which is mentioned in wp-config.hp – and learned that the siteurl is set in the administration settings in the GUI, Settings|General WordPress Address URL and Site Address URL. I changed these to https://drjohnstechtalk.com/blog and bingo, my plugins began to work properly again!

What might go wrong when turning on SSL
In another context I have seen both those errors, which I feel are poorly documented on the Internet, so I wish to mention them here since they are closely related to the topic of this blog post.

SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:601

and

curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

I generated the first error in the process of trying to look at the SSL web site using openssl. What I do to test that certificates are being properly presented is:

$ openssl s_client -showcerts -connect localhost:443

And the second error mentioned above I generated trying to use curl to do something similar:

$ curl -i -k https://localhost/

The solution? Well, I began to suspect that I wasn’t running SSL at all so I tested curl assuming I was running on tcp port 443 but regular http:

$ curl -i http://localhost:443/

Yup. That worked just fine, producing all the usual HTTP response headers plus the content of my home page. So that means I wasn’t running SSL at all.

This virtual host being from a template I inherited and one I didn’t fully understand, I decided to just junk the most suspicious parts of the vhost configuration, which in my case were:

<IfDefine SSL>
<IfDefine !NOSSL>
...
</IfDefine>
</IfDefine>

and comment those guys out, giving,

#<IfDefine SSL>
#<IfDefine !NOSSL>
...
#</IfDefine>
#</IfDefine>

That worked! After a restart I really was running SSL.

Making it stronger
I did not do this immediately, but after the POODLE vulnerability came out and I ran some tests I realized I should have explicitly chosen a cipher suite in my apache server to make the encryption sufficiently unbreakable. This section of my working with ciphers post shows some good settings.

Mixed content
I forgot about something in my delight at running my own SSL web server – not all content was coming to the browser as https and Firefox and Internet Explorer began to complain as they grew more security-conscious over the months. After some investigation I found that what it was is that I had a redirect for favicon.ico to the WordPress favicon.ico. But it was a redirect to their HTTP favicon.ico. I changed it to their secure version, https://s2.wp.com/i/favicon.ico, and all was good!

I never use the Firefox debugging tools so I got lucky. I took a guess to find out more about this mixed content and clicked on Tools|Web developer|Web console. My lucky break was that it immediately told me the element that was still HTTP amidst my HTTPS web page. Knowing that it was a cinch to fix it as mentioned above.

Conclusion
Good-ole search engine optimization (SEO) has prodded us to make the leap to run SSL. In this posting we showed how we did that while preserving all the links that may be floating ou there on the Internet by using our redirect factory.

References
Having an apache instance dedicated to redirects is described in this article.
Some common sense steps to protect your apache server are described here.
Some other openssl commands besides the ones used here are described here.
Choosing an appropriate cipher suite and preventing use of the vulnerable SSLv2/3 is described in this post.
I read about Google’s plans to encrypt the web in this Naked Security article.