Intro
Our EDI group hails me last Friday and says they can’t reach their VANs, or at best intermittently. What to do, what to do… I go on the offensive and say they have to stop using FTP (and that’s literal FTP, not sftp, not FTPs, just plain old FTP), it’s been out of date for at least 15 years.
But that wasn’t really helping the situation, so I had to dig a lot deeper. And frankly, I was coincidentally having intermittent issues with my scripted speedtests. Could the two be related?
The details
We have a bunch of synthetic monitors we run though that same firewall. They were failing every few minutes, and then became good.
And these FTPs were like that as well. Some would work and then minutes later not work.
The firewall person on call looked at the firewall, saw some of the described traffic passing through, and declared firewall is fine.
So I got a more cooperative firewall colleague on this. And he got a really expert Checkpoint support person on the call. That guy led us to look at SYN DEFENDER which is part of IPS and enabled via fw accel. If it sees too many out of state packets in a given time it will shut down the interface where the problem was observed!
The practical effect is that even if you’re taking traces on the Checkpoint, checking the logs, etc, you won’t see the traffic! So that really throws most firewall admins is this situation is so unusual and they are not trained to look for it.
In this case it was an internal firewall and ir was comfortable to disable SYN DEFENDER on it. All problems went away after that.
Four months later…
Then four months later, after the firewall was upgraded to v 81.10, they must have set SYN DEFENDER (AKA synatk) up all over again. And of course no one was thinking about it or expecting what happened next, which is, these exact same problems started all over again. But there were different firewall colleagues involved, none with any first-hand experience of the issue. Then I got involved and just sort of tackled my way through it in a trouble-shooting session. No one was placing any judgments (my-stuff-is- fine,-yours-must-be-broken kind of thinking). Then I eventually recalled the old problem, and looked up this post to help name it – SYN DEFENDER – so that that would be meaningful to the firewall colleague. Yup, he took it from there. And we were good. I admonished the on-call guy who totally missed it, and he humbly admitted to not being familiar with this feature and how does it work. So I will explain it to him.
Results of running fwaccel synatk config:
enabled 0 enforce 0 global_high_threshold 10000 periodic_updates 1 cookie_resolution_shift 6 min_frag_sz 80 high_threshold 5000 low_threshold 1000 score_alpha 100 monitor_log_interval (msec) 60000 grace_timeout (msec) 30000 min_time_in_active (msec) 60000
These are probably the defaults as we haven’t messed with them. Right now you see it’s disabled. It spontaneously re-enabeld itself after only a few days, and the problems started all over again.
References and related
VAN: Value Added Network