Security Web Site Technologies

Who’s hacking Drjohnstechtalk lately?


This headline was inspired by years of listening to our managed service providers: overpromise and underdeliver! Who’s hacking my web site? I have no idea. But what I can deliver is a list of badly behaved IP addresses over the last 24 hours.

Let’s do it

So, here is a dynamically-compiled list of offenders who have “hacked” my web site over the last 24 hours. They are IP addresses caught trying to fetch non-existent web pages (such as the default login page) or post unauthorized content to the site such as spammy comments.

Without further ado, here are the latest IPs which include up-to-the-minute entries.

What are they?

I don’t think it’s anything glamorous like an actual black hat scheming to crack through my site’s defenses, which would probably fall pretty quickly! It looks like a lot of the same type of probes coming from different IPs. So I suspect the work of a botnet that crawls through promising-sounding WordPress sites, looking for weak ones. Probably thousands of bots – things like compromised security cameras and poorly configured routers (IoT) orchestrated by a Command and Control station under the control of a small group of bad actors.

And there is probably a bit of access from “security researchers” (ethical hackers) who look for weaknesses that they can responsibly disclose. I’m imagining this scenario: a security researcher discovers a 0-day WordPress vulnerability and wants to make a blanket statement to the effect: 30% of all WordPress sites are vulnerable to this 0-day exploit. So they have to test it. Well, I don’t want to be anyone’s statistic. So no thank you.

But I don’t have time to deal with any of that. It’s one strike and you’re out at my site: I block every single one of these IPs doing these things, even based on a single offense.

Actual example hacks

Here are some from November 2020: - - [22/Nov/2020:13:31:13 -0500] 704 "GET /blog/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4240.193 Safari/537.36" 818 - - [22/Nov/2020:13:31:14 -0500] 1 "GET /blog//wp-includes/wlwmanifest.xml HTTP/1.1" 200 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4240.193 Safari/537.36" 386 - - [22/Nov/2020:13:31:14 -0500] 409 "GET /blog//wp-login.php HTTP/1.1" 404 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4240.193 Safari/537.36" 371

Note the access at the end to /blog//wp-login.php, a link which does not exist on my site! I imagine the user agent is spoofed. Fate: never again to access my site. - - [22/Nov/2020:12:31:43 -0500] 26103 "POST /blog//xmlrpc.php HTTP/1.1" 200 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4240.193 Safari/537.36" 1094

This one (above) is an xmlrpc.php example. The next one is a bit more infuriating to me – a blatant command injection attempt: - - [22/Nov/2020:09:58:43 -0500] 673 "GET /blog/ HTTP/1.1" 200 "\\think\\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP21" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" 743

I caught it due to the presence of index.php – another string which does not have a legit reason to appear in my access log, AFAIK.

Then there’s the bot trying to pull a non-existent .env (which, if it existed, might have contained environment variables which might have provided hints about the inner workings of the site): - - [22/Nov/2020:09:48:59 -0500] 1248 "GET /.env HTTP/1.1" 404 "-" "python-requests/2.25.0" 184

The 404 status code means not found.

And this one may be trying to convey a message. I don’t like it: - - [12/Nov/2020:00:24:00 -0500] 623 "GET /blog/2011/08/http://Idonthaveanywebsite... HTTP/1.1" 301 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8;" 723


By looking for specific strings I realize I am implementing a very poor man’s version of a Web Application Firewall. Commercial WAFs are amazing to me – I know because i work with them. They have thousands of signatures, positive and negative matches, stuff you’d never even dream about. I can’t afford one for my self-hosted and self-funded site.

A word about command injection

If you look at the top 10 web site exploits, command injection is #1. A bunch of security vendors got together to help web site operators understand the most common threats by cataloging and explaining them in easy-to-understand terms. It’s pretty interesting.


Sadly, the most common visitor to me web site are bots up to no good. I have documented whose hitting me up in real time, in case this proves to be of interest to the security community. Actual offending lines from my access file have been provided to make everything more concrete.

I have offered a very brief security discussion.

I don’t know who’s hacking me, or what’s hacking me, but I have shared a lot of information not commonly shared.

References and related

A great commercial web application firewall (WAF) is offered by F5.

Here’s the link to the top 10 web site exploits in clear language:

Network Technologies Security

LDAP authentication on the F5 BigIP without Access Policy Manager

I recently received revised guidelines for dmz best practices which mentioned a requirement to implement application-independent authentication using the F5 web application firewall. I had never heard of it and didn’t think it was possible without buying the very expensive APM license. They insisted it was possible and even easy to do. So I investigated and found they were right!

The details
This is a feature added around version 11.4.

On the F5, go to Local Traffic|Profiles|Authentication|Configurations and create a new configuration. Here you put in the essential LDAP information and give these settings a name such as myLDAP. I needed to set Login Attribute to cn. Then go to …Authentication|Profiles and create a new one. Set parent profile as LDAP and associate the configvuration myLDAP to it. Rule can be _sys_auth_ldap.

In the virtual server Properties tab look for the section Authentication Profiles. Pick the profile you created.

That’s it! Your virtual server now has application-independent authentication using your preferred LDAP source.

So far I only tested against an LDAP source that doesn’t require an ldap bind. But I did successfully test against an ldaps source (which runs on port 636 and encrypts the communication using SSL. I got that to work setting SSL to Enabled and essentially taking the other SSL-related default values.

We show how to implement application-independent authentication on an F5 BigIP which only has the local traffic manager (LTM) license. We used an LDAP directory for the authentication source. I believe a certificate mechanism would also have been possible. As it happens our LDAP source was not an Active Directory (AD) tree, but I believe it would be possible to use that as well. We also did not limit access to any specific group, but that is probably possible as well.

Admin Network Technologies

The IT Detective Agency: the case of the mysterious reset

An F5 BigIP load balancer equipped with web application firewall worked for everyone, except one app used by one customer. What was going wrong?

Packet trace

I always do a packet trace when there is nothing else to go on, as is so often the case these days. Packet traces themselves are getting increasingly complex, what with encrypted communications and multiple connections, etc. In this case there seemed to be a single TCP connection which was of concern, but it was encrypted (SSL traffic on tcp port 443).

Well, I have access to the private key so I figured out how to insert that into Wireshark so it could decrypt the packets. Pretty cool – I’ve never done that before.

So the communication got a lot further than I had expected. What I had expected to learn is that there was an incompatibility between supported ciphers of the client and the server such that there were no overlapping ciphers. But no! That was not the case at all – the packet trace got well beyond that early stage of packet exchanges between client and server.

In fact the client got so far that it sent these (encrypted) HTTP headers, which I was able to decrypt with Wireshark and the servers’ private key:

POST /cgi-bin/java/ HTTP/1.0
Content-type: application/x-www-form-urlencoded
Content-length: 1816

Then right after that I saw the F5 BigIP device send the client a TCP reset (RST) as though it was unhappy about something and wanted to end it right there!

So, still stuck, I searched and found that you can enable logging of the reason for the TCP RST’s on F5 BigIPs:

Enable RST logging
To enable reset logging to the ltm log:
# tmsh
(tmos)# modify /sys db tm.rstcause.log value enable

And this is the error that was logged:

Logged error

Jun 16 08:09:19 local/tmm err tmm[5072]: 01230140:3: RST sent from to, [0x11d17ec:1804] No available pool member

It’s just a hint of what was wrong, but it was enough to jog my memory. A WAF (web application firewall) policy exists that matches based on hostname and otherwise exits. The hostname entered is Well, clearly that match is pretty darn literal. When we put the same URL into our browser or into curl we could not reproduce the error. But those clients produce a host header with value, not

So their client, a strange Java-base client, threw in the :443 into the host header and it was not matching the host header match in the WAF policy! So no pool was selected and the fall-through rule was executed, resulting in a TCP RST to the client!

I added an additional host header to match,

They tested and it worked!

Case closed.

A mysterious TCP RST sent from an F5 load balancer to just one client is explained in great detail. Some valuable networking tools were learned in the process, namely, how to decrypt an encrypted SSL packet trace.

Could I stand on my high horse and complain that they were sending a non-standard header and go fix their stupid client? Well that might have felt satisfying but when I looked at the HTTP standard it does permit the port number to be present in that form! So I was the one in the wrong, even from a protocol standpoint.

References and related
F5’s SOL 13223 describing enabling logging for TCP RST packets
HTTP Request header fields are described in this Wikipedia article.

Network Technologies

The IT Detective Agency: Why our forwarding vserver doesn’t route

F5 BigiP appliances are very versatile networking appliances. But sometimes you gotta know what you are doing!

The details
We set up a load-balanced Radius service using the same subnet for the radius servers as the load balancer itself. Setting up this service is moderately tricky. You have to set the default route of the Radius servers to be the load balancer, and on the load balancer SNAT (which I prefer to translate as “source NAT,” though technically it is “secure NAT”) and NAT should be disabled. And there are two services, AAA and audit (UDP ports 1812 and 1813).

So everything’s a bit different when all you'[re used to is creating load-balanced pools for web servers.

So with this setup an incoming packet comes in, its source is preserved, but its destination is NAT’d to the radius server by the load balancer. In the response, the source is the radius server. That gets NAT’d to the IP of the load-balanced service. So there are two stages for incoming request packets (pre- and -post NAT) and two for the responses. Here’s a trace which shows all this:

12:10:41.259073 IP > RADIUS, Access Request (1), id: 0x30 length: 260
12:10:41.259086 IP > RADIUS, Access Request (1), id: 0x30 length: 260
12:10:41.259735 IP > RADIUS, Access Reject (3), id: 0x30 length: 44
12:10:41.259745 IP > RADIUS, Access Reject (3), id: 0x30 length: 44

So all is good, right? Except that now we have a default route from the Radius server to the load balancer and so all response traffic is going through the load balancer, even things not related to Radius, such as a packets from an RDP session.

So we defined a forwarding_vserver to make the BigIP act as a router:

A forwarding vserver is a virtual server of type Forwarding (IP). In the bigip.conf file it looks like this:

virtual forwarding_vserver {
   ip forward
   destination any:any
   profiles route_friendly_fastL4 {}

But it doesn’t work! Packets from the Radius server come to the load balancer, and then they get source NAT’d to the floating self-IP of the load balancer. That’s no good. In TCP your response packets have to come from the IP you connected to! For simple PINGs you kind of get away with it, but with a warning. In TCP your PC will send a RST (reset) packet every time it gets a response packet with the wrong source IP, even if the other information is correct.

The solution
With the help of someone who understands snat auto-maps better than I do (evidently), I got the tip that I have a global snat-automap enabled, which is doing this. That’s how I’ve always run these LTMs (Local Traffic Managers). I had forgotten why or how I did it. Well the snat-automap pretty mcuh applies to all my other load-balanced services so I can’t simply chuck it. And I don’t have another subnet handy for use so I can’t simply exclude one of my vlans. They suggested that it could be turned off on my forwarding_vserver with an irule! Who would have figured? So I created a very simple irule:

# Turn off snat, i.e., for us in our forwarding_vserver
# inspired by
# DrJ, 11/2013
         snat none

and applied it to my forwarding_vserver, which now looks like this:

virtual forwarding_vserver {
   ip forward
   destination any:any
   rules snat-none
   profiles route_friendly_fastL4 {}

And voila, the LTM now routes those packets correctly without any address translation! And the Radius service still does its translations as desired.

Case closed!

We learned a little about F5 BigIPs today. The frustrating thing about the documentation is that they don’t really cover actual use cases so they introduce configuration settings without fully explaining why you would want to use them.

For the curious, the forwarding_vserver is accomodating an asymmetric routing situation. Incoming RDP (remote desktop protocol) packets get sent directly from a Cisco router to the Radius server. It’s just the response packets that flow from the Radius server, to the LTM, to the Cisco router.

References and related
In this post I show why a basic virtual server might not be working – a kind of rookie mistake we’ve all probably made at some point!
This post shows some non-trivial iRule examples.


The IT Detective Agency: strange ssl error explained

Fromm time-to-time I get an unusual ssl error when using curl to check one of my web sites. This post documents the error and how I recovered from it.

The details

I was bringing up a new web site on the F5 BigIP loadbalancer. It was a secure site. I typically use the F5 as an ssl acclerator so it terminates the ssl connection and makes an http connection back to the origin server.

So I tested my new site with curl:

$ curl -i -k

curl: (52) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104

Weird, I thought. I had taken the certificate from an older F5 unit and maybe I had installed it or its private key wrong?

I tested with openssl:

$ openssl s_client -showcerts -connect

SSL handshake has read 2831 bytes and written 453 bytes
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: E95AB5EA2D896D5B3A8BC82F1962FA4A68669EBEF1699DF375EEE95410EF5A0C
    Master-Key: EC5CA816BBE0955C4BC24EE198FE209BB0702FDAB4308A9DD99C1AF399A69AA19455838B02E78500040FE62A7FC417CD
    Key-Arg   : None
    Start Time: 1374679965
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)

This all looks pretty normal – the same as what you get from a healthy working site. So the SSL, contrary to what I was seeing from curl, seemed to be working OK.

OK, so, SSL is handled by the F5 itself we were saying. That leaves the origin server. Bingo!

In F5 you have virtual servers and pools. You configure the SSL CERTs and the public-facing IP and the pool on the virtual server. The pool is where you configure your origin server(s).

I had forgotten to associate a default pool with my virtual server! So the F5 had nowhere to go really with the request after handling the initial SSL dialog.

I don’t think the available help for this error is very good so I wanted to offer this specific example.

So I associated a pool with my virtual server and immediately the problem went away.

Case closed.

We solved a very specific case this week and hopefully provided some guidance to others who might be seeing this issue.

My favorite openssl commands

Admin Web Site Technologies

Tipsheet: How to run NTP on F5 BigIP

If you feel you’ve added your ntp servers correctly via the GUI (System|Configuration|Device|NTP), and yet you get an output like this:

# ntpq -p

     remote           refid      st t when poll reach   delay   offset  jitter
 ntp1.drj        .INIT.          16 -    - 1024    0    0.000    0.000   0.000
 ntp2.drj        .INIT.          16 -    - 1024    0    0.000    0.000   0.000

and you observe the time is off by seconds or even minutes, then you may have made the mistake I made. I used fully-qualified domain names (FQDN) for the ntp servers.

Switch from FQDNs to IP addresses and it will work fine:

# ntpq -p

     remote           refid      st t when poll reach   delay   offset  jitter
*ntp1.drj     3 u  783 1024  377    0.951   -8.830   0.775
+ntp2.drj     3 u  120 1024  377   20.963   -8.051   5.705

The date command will now give the correct time.

Having correct time is useful for the logging, especially if you are using ASM and are trying to correlate known activity against the reported errors.

Admin IT Operational Excellence Web Site Technologies

Virtual Server not Working in F5 BigIP

OK. This posting is only directly applicable to the small number of people who run BigIP load balancers. And of that set, only a certaIn subset will likely ever have this situation. Nevertheless, it’s useful to document it. There are lessons in it for the rest of us, it shows the creative problem-solving process used in IT, or rather the creative process that should be used.

So I had a virtual server associated with a certain pool and it was operating fine for years. Then something changes. We want to associate a new name with this virtual server, but test it first, all while keeping the old name working. Well, this is a secured site by which I mean it is running https rather than http. There’s nothing intrinsic in the web site itself that ties it to a particular name. If this were a run-of-the-mill non-secure site you would solve this problem with DNS. Set up an alias and you’re good to go. But secured sites are a wee bit trickier. They present a certificate after all. And the certificate has just one name, at least ours does. Guess I can address multi-name certificates known as Subject Alternative Name CERTs in a separate post. And that name is the original DNS name. What to do? Simple. As any BigIP admin would tell you you create a new virtual server and associate it with a new IP and a new SSL profile containing the new certificate you just bought but the old pool. In DNS assign this new IP to your new DNS name. That’s all pretty straightforward

Having done all that, I blithely tested with lynx (iI’s an old curses-based browser which runs on old Unix systems. The main point is to not test with a complex browser where like Internet Explorer where you are never 100% sure if the problem lies with the browser. If I had it, I would test with curl, but it’s not on that system.). And…it hangs.

Now I’ll admit a lot of stupid things I did (which is typical of any good debugging session of an IT professional – some self-created red herrings accompany any decent sleuthing) and I ratchet up the debugging a notch. Check the web server logs. I see no log of my lynx accesses. Dig a little deeper still. Fire up a trace. Here’s a little time-saver. BigIP does have a tcpdump program, but it is a little stunted. Typically you have multiple interfaces on a BigIP. In this case I felt it pertinent to know if packets were getting to the BigIP from lynx, and then again, if those packets were leaving the BigIP and going to the web server. So the tip is that whereas a “normal” tcpdump might allow you to use the switch -i any to listen on all interfaces, that doesn’t work on BigIP. Use -i 0.0 instead. And of course restrict it somehow so that your own shell session’s packets won’t be picked up by the trace, or else you could be in for a nasty surprise of exponentially increasing traffic (a devastating situation perhaps worthy of its own blog entry!). In this case I added an expression port 443. So I have:

tcpdump -i 0.0 port 443

And, somewhat to my surprise (You should always have a hypothesis, even if it’s just a gut feeling: will this little test work, or not. Why?) not only were packets going from lynx to BigIp and then again to the web server, I could even see returned packets back from the web server to BigIp to lynx. But it was not a lot of packets. A SYN, SYN-ACK and maybe a single data packet and that’s about it. It should have been more chatty.

The more tests you can think of, the better, especially ones that emphasize the marginal differences between the thing that works and the one that doesn’t. One test along those lines: take this same virtual server and associate it with a different pool. I did that, and that test worked!

Next, I tried to access the web server using curl on the BigIP itself. I could, but not at first. First I used the local web server URL http://web_server_ip:443/. It hung my curl command, just like using lynx on the other server. Hmm. I then looked on the web server again. I notice that it has a certificate installed. Ah. So it’s actually running https. So try curl from BigIP again, but this time with the -k switch (insecure, meaning don’t verify the certificate issuer) and a url beginning with https rather than http. Bingo. It comes back with the home page. Now we’re getting somewhere.

Finally I look more closely at the virtual server setup for the old name, the one that works. I see that the server profile is SSL. It basically means that the traffic is encrypted when it hits the BigIP, and the server CERT is associated with the external name. The BigIP decrypts the traffic, then re-encrypts it before sending it along to the web server. The CERT for the second leg is a self-signed CERT and is never seen by users.

I had forgotten to set up my new test virtual server with the server SSL profile, so the second leg of traffic was not being re-encyrpted by the BigIP, even though the web server was only willing to engage in SSL communication with the BigIP. Once I updated the server profile, it all worked fine! Of course after getting the expected results from lynx I went to my desktop browser, just like a regular user, and successfully tested it there as well. You want to make sure your final tests are a realistic approximation of what the user will be doing. If that’s not all possible under your own control, bring in a user for testing.

Liked this article? Here’s another of my IT operational excellence articles that has a somewhat wider applicability.