The IT Detective Agency: internal DNS queries getting clobbered after bind upgrade

Intro
We’ve upgraded BIND innumerable times over the years. There’s never really been an issue. The new version just picks up and behaves exactly like the old version and all is good. But this time, in upgrading from ISC’s BIND v 9.8.5-P2 to BIND v 9.9.5-P1 something was dramatically different.

The details
Look at these queries:

> dig ns 10.in-addr.arpa

; <<>> DiG 9.9.2-P2 <<>> ns 10.in-addr.arpa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60248
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;10.in-addr.arpa.               IN      NS
 
;; ANSWER SECTION:
10.in-addr.arpa.        0       IN      NS      10.IN-ADDR.ARPA.
 
;; Query time: 0 msec
;; SERVER: blah, blah
;; WHEN: Wed Jun 25 09:49:30 2014
;; MSG SIZE  rcvd: 73

> dig -x 10.100.208.10

; <<>> DiG 9.9.2-P2 <<>> -x 10.100.208.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 6088
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;10.208.100.10.in-addr.arpa.   IN      PTR
 
;; AUTHORITY SECTION:
10.IN-ADDR.ARPA.        86400   IN      SOA     10.IN-ADDR.ARPA. . 0 28800 7200 604800 86400
 
;; Query time: 0 msec
;; SERVER: blah, blah
;; WHEN: Wed Jun 25 09:49:56 2014
;; MSG SIZE  rcvd: 106

That is seriously bad and wrong – for us! This is a cache-only server and there are indeed RFC-1918 addresses defined on internal nameservers, such as that 10.100.200.10.

An email relay which relies on reverse lookups started to fail.

A DuckDuckGo search did not show anything relevant. Maybe Google would have.

I ultimately registered for an account at the knowledge base at isc.org, kb.isc.org, and quickly found my answer.

In fact they were crystal clear in explaining this very problem, so I hesitated to document it here, but I figure others might leap first and then read the documentation later like myself so it might do someone else some good.

They say:

...
Although this will be effective as a workaround, administrators are urged not to just specify empty-zones-enable no;
 
It is much better to use one or more disable-empty-zone option declarations to disable only the RFC 1918 empty zones that are in use internally.  
...

That empty-zones-enable no; by the way is a configuration option you can toss in your main configuration file in the options section.

Case closed.

Conclusion
Our reverse lookups on the Intranet began to fail after an innocuous upgrade of the ISC bind nameserver to version 9.9. A simple addition of an extra configuration statement resolved the matter. I guess it really is a good idea to RTFM.

References
ISC’s site is www.isc.org.
A different type of DNS clobbering was described in this case.
A word about Google’s awesome public DNS service.