Intro
This is an active investigation and I cannot stamp case closed at the end as is my wont!
I have access to both source and destination servers. They are virtualy DNS servers running linux and BIND. pretty standard stuff.
We had been measuring response times of DNS queries and were all too often getting high values – like 3000 ms high! But not always. Also 50 ms.
And there were these ICMP messages to this effect:
ICMP 10.13.24.21 udp port 51343 unreachable, length 233
Fortunately I had the ability to do a packet tarce at both source and destination servers – that is a rare luxury these days in the age of granular access control.
Well, what I was seeing was the source DNS server sending a simple UDP query to the destination DNS server, just one UDP packet while the destination server saw that same packet four times! But not always. Slometimes it only saw one packet. Sometimes two packets. So it is random but occurring so often a trace for a few seconds shows the problem.
To be continued…