Intro
This was a fascinating case insofar as it was my first encounter with a real life IPv6 application. So it was trial by fire.
The details
I think the title of the post makes clear what happened. The site people were saying they can ping hosts by IP but not by DNS name. So basically nothing was working. I asked them to do an ipconfig /all and send me the output. At the top of the list of DNS servers was this funny entry:
I asked them to run nslookup, and sure enough, it timed out trying to talk to that same IPv6 server. Yet they could PING it.
The DNS servers listed below the IPv6 one were the expected IPv4 our enterprise system hands out.
My quick conclusion: there is a rogue host on their subnet acting as an IPv6 DHCP server! It took some convincing on my part before they got on board with that idea.
But I goofed too. In my haste to move on, I confused an IPv6 address with a MAC address. Rookie IPv6 mistake I suppose. It looked strange, had letters and even colons, so it kind of looks like a MAC address, right? So I gave some quick advice to get rid of the problem: identify this address on the switch, find its port and disable it. So the guy looked for this funny MAC address and of course didn’t find it or anything that looked like it.
My general idea was right – there was a rogue IPv6 DHCP server.
My hypothesis as to what happened
The PCs have both an IPv4 as well as an IPv6 stack, as does just about everyone’s PC. These stacks run independently of each other. Everyone blissfully ignores the IPv6 communication, but that doesn’t mean it’s not occurring. I think these PCs got an IPv4 IP and DNS servers assigned to them in the usual way. All good. Then along came a DHCPv6 server and the PC’s IPv6 stack sent out a DHCPv6 request to the entire subnet (which it probably is doing periodically all along, there just was never a DHCPv6 server answering before this). This time the DHCPv6 server answered and gave out some IPv6-relevant information, including a IPv6 DNS server.
I further hypothesize that what I said above about the IPv4 and IPv6 stacks being independent is not entirely true. These stacks are joined in one place: the resolving nameservers. You only get one set of resolving namesevrers for your combined IPv4/IPv6 stacks, which sort of makes sense because DNS servers can answer queries about IPv6 objects if they are so configured. So, anyway, the DHCPv6 client decides to put the DNS server it has learned about from its DHCPv6 server at the front of the existing nameserver list. This nameserver is totally busted, however and sits on the request and the client’s error handling isn’t good enough to detect the problem and move on to the next nameserver in the list – an IPv4 nameserver which would have worked just great – despite the fact that it is designed to do just that. And all resolution breaks and breaks badly.
What was the offending device? They’re not saying, except we heard it was a router, hence, a host introduced by the LAN vendor who can’t or won’t admit to having made such an error, instead making a quiet correction. Quiet because of course they initially refused the incident and had us look elsewhere for the source of the “DHCP problem.”
Alternate theory
I see that IPv6 devices do not need to get DNS servers via DHCPv6. They can use a new protocol, NDP, neighbor discovery protocol. Maybe the IPv6 stack is periodically trying NDP and finally got a response from the rogue device and put that first on the list of nameservers. No DHCPv6 really used in that scenario, just NDP.
Useful tips for layer 2 stuff
Here’s how you can find the MAC of an IPv6 device which you have just PINGed:
netsh interface ipv6 show neighbors
from a CMD prompt on a Windows machine.
In Linux it’s
ip ‐6 neigh show
Conclusion
Another tough case resolved! We learn some valuable things about IPv6 in the process.
References and related
I found the relevant commands in this article: https://www.midnightfreddie.com/how-to-arp-a-in-ipv6.html