Intro
The original premise of this article is that some Internet Service Providers were seen to block TCP port 22, used by ssh and sftp. However, as often happens during active IT investigations, this turns out to be completely wrong. In fact there was a block in this case we studied, but not by the ISPs. An overly aggressive ACL on the customer premise equipment Internet router is in fact the culprit.
The Problem
(IPs skewed to protect whatever) We asked a partner to do an sftp to drjohnstechtalk.com. All firewall and routing rules were in place. The partner tried it. He saw a SYN packet leaving, but no packets being returned. Here at drjohnstechtalk, we didn’t see any packets whatsoever! This partner makes sftp connections to other servers successfully. What the heck?
We had them try the following basic command:
nc -v host 22
where host is the IP of the target server. The response was:
nc: connect to host port 22 (tcp) failed: No route to host
But switching to port 21 (FTP) showed completely different behaviour: there was no message whatsoever and the session hanged. That’s good! That’s the usual firewalls dropping packets. But this No route to host needs more exploration.
Getting Closer
So we did an open trace. I mean a tcpdump without any limiting expression. The dump showed the SYN out to port 22, followed by this nugget:
13:09:14.279176 IP Sprint_IP > src_IP: ICMP host target_IP unreachable - admin prohibited filter, length 36
Next Steps
This well-intentioned filtering is causing a business problem. The Cisco IOS ACL that got them into trouble was this one:
ip access-list extended drop-spoof-and-telnet deny tcp any any eq 22 log-input
Solution
They liked the idea of this filtering, but apparently this was the first request for inbound ssh access. So they decided to keep this filter rule but precede it with more specific rules as required, essentially acting like a second firewall:
ip access-list extended drop-spoof-and-telnet permit tcp host IP_src host IP_dest eq 22 deny tcp any any eq 22 log-input