Intro
I haven’t published a new case in a while, not for lack of cases, but more that they they all fall into something I’ve already written about. But today there is definitely something new.
Some details
Thousandeyes agent-to-agent communication was generally working for all our enterprise agents after fixing firewall rules, etc, except for this one agent hosted in Azure US East. Was it something funny about the firewalls on either side of the vpn tunnel to this cloud? Ping tests were working. But a connection to tcp port 49153, which is used for agent-to-agent communication gave a response in the form of an ICMP type 3 code 10 packet which said something like host administratively prohibited. What?
The Cisco TAM suggested to look at iptables. I did a listing with iptables -L. The output is pretty long and I’m not experienced looking at it. Nothing much jumped out at me, but I did note the presence of this line:
REJECT all — anywhere anywhere reject-with icmp-host-prohibited
in a couple of the chains, which seemed suspicous.
An Internet search pointed towards firewalld since the agent is a Redhat 7.9 system. Indeed firewalld was running:
systemctl status firewalld ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2023-10-12 15:26:25 UTC; 5h 45min ago
The suggestion is to test with firewalld disabled. Indeed this produced correct results – no more ICMP packets back.
But it’s probably a good security measure to run firewalld, so how to modify it? This note from Redhat was particularly helpful in learning how to add a rule to the firewall. I pretty much just needed to do this to permanently add my rule:
firewall-cmd –add-port 49153/tcp –permanent
Afterwards the agent-to-agent tests began to be run successfully.
Which runs first, tcpdump or firewalld?
tcpdump
This is a good question to ask because if the order had been different, and who knows, you might have your packets dropped before you ever see them on tcpdump. But tcpdump seems to get a pretty clean mirror of what the network interface gets before application or kernel processing.
The new equivalent to netstat -an
If I want to see the listening processes in Redhat I might do a
ss -ln
In the old days I memorized using netstat -an, but that is now frowned upon.
Conclusion
We solved a case where tcp packets were getting returned with an ICMP packet which basically said: prohibited. This was due to the host, a Redhat 7 system, having restricted ports due to firewalld running. Once firewalld was modified this traffic was permitted and Thousandeyes Tests ran successfully. We also proved that tcpdump runs before firewalld.