web application firewall – Dr John's Tech Talk

Intro

This headline was inspired by years of listening to our managed service providers: overpromise and underdeliver! Who’s hacking my web site? I have no idea. But what I can deliver is a list of badly behaved IP addresses over the last 24 hours.

Let’s do it

So, here is a dynamically-compiled list of offenders who have “hacked” my web site over the last 24 hours. They are IP addresses caught trying to fetch non-existent web pages (such as the default login page) or post unauthorized content to the site such as spammy comments.

Without further ado, here are the latest IPs which include up-to-the-minute entries.

What are they?

I don’t think it’s anything glamorous like an actual black hat scheming to crack through my site’s defenses, which would probably fall pretty quickly! It looks like a lot of the same type of probes coming from different IPs. So I suspect the work of a botnet that crawls through promising-sounding WordPress sites, looking for weak ones. Probably thousands of bots – things like compromised security cameras and poorly configured routers (IoT) orchestrated by a Command and Control station under the control of a small group of bad actors.

And there is probably a bit of access from “security researchers” (ethical hackers) who look for weaknesses that they can responsibly disclose. I’m imagining this scenario: a security researcher discovers a 0-day WordPress vulnerability and wants to make a blanket statement to the effect: 30% of all WordPress sites are vulnerable to this 0-day exploit. So they have to test it. Well, I don’t want to be anyone’s statistic. So no thank you.

But I don’t have time to deal with any of that. It’s one strike and you’re out at my site: I block every single one of these IPs doing these things, even based on a single offense.

Actual example hacks

Here are some from November 2020:

100.26.218.97 - - [22/Nov/2020:13:31:13 -0500] 704 "GET /blog/ HTTP/1.1" 200 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4240.193 Safari/537.36" 818
100.26.218.97 - - [22/Nov/2020:13:31:14 -0500] 1 "GET /blog//wp-includes/wlwmanifest.xml HTTP/1.1" 200 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4240.193 Safari/537.36" 386
100.26.218.97 - - [22/Nov/2020:13:31:14 -0500] 409 "GET /blog//wp-login.php HTTP/1.1" 404 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4240.193 Safari/537.36" 371

Note the access at the end to /blog//wp-login.php, a link which does not exist on my site! I imagine the user agent is spoofed. Fate: never again to access my site.

46.119.172.173 - - [22/Nov/2020:12:31:43 -0500] 26103 "POST /blog//xmlrpc.php HTTP/1.1" 200 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4240.193 Safari/537.36" 1094

This one (above) is an xmlrpc.php example. The next one is a bit more infuriating to me – a blatant command injection attempt:

45.146.164.211 - - [22/Nov/2020:09:58:43 -0500] 673 "GET /blog/ HTTP/1.1" 200 "https://50.17.188.196:443/index.php?s=/Index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP21" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" 743

I caught it due to the presence of index.php – another string which does not have a legit reason to appear in my access log, AFAIK.

Then there’s the bot trying to pull a non-existent .env (which, if it existed, might have contained environment variables which might have provided hints about the inner workings of the site):

54.226.98.220 - - [22/Nov/2020:09:48:59 -0500] 1248 "GET /.env HTTP/1.1" 404 "-" "python-requests/2.25.0" 184

The 404 status code means not found.

And this one may be trying to convey a message. I don’t like it:

69.30.226.234 - - [12/Nov/2020:00:24:00 -0500] 623 "GET /blog/2011/08/http://Idonthaveanywebsite... HTTP/1.1" 301 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" 723

Discussion

By looking for specific strings I realize I am implementing a very poor man’s version of a Web Application Firewall. Commercial WAFs are amazing to me – I know because i work with them. They have thousands of signatures, positive and negative matches, stuff you’d never even dream about. I can’t afford one for my self-hosted and self-funded site.

A word about command injection

If you look at the top 10 web site exploits, command injection is #1. A bunch of security vendors got together to help web site operators understand the most common threats by cataloging and explaining them in easy-to-understand terms. It’s pretty interesting. https://owasp.org/www-project-top-ten/

Conclusion

Sadly, the most common visitor to me web site are bots up to no good. I have documented whose hitting me up in real time, in case this proves to be of interest to the security community. Actual offending lines from my access file have been provided to make everything more concrete.

I have offered a very brief security discussion.

I don’t know who’s hacking me, or what’s hacking me, but I have shared a lot of information not commonly shared.

References and related

A great commercial web application firewall (WAF) is offered by F5.

Here’s the link to the top 10 web site exploits in clear language: https://owasp.org/www-project-top-ten/

Intro
When debugging F5’s Web Application Firewall (AKA ASM) you get all the HTTP request headers in the GUI which you can cut and paste. Sometimes it is not so obvious how to fix a particular SupportID and you don’t want to bother the user to reproduce the error. This is code that allows you to scrape the headers and send them back, assuming you have a Linux server on the Internet. It’s not quite exact – Perl wants to add a TE header and maybe some others. More importantly, F5 only gives you around the first 5 KB of the headers, and often the real headers + data can be much longer than that. But It’s pretty good at preserving the basic headers and the POSTed data, if any.

The details

#!/usr/bin/perl
# DrJ - 6/2015
# take waf Full Request as input from STDIN and spit it out
use LWP;
$DEBUG = 1;
# example:
#POST /cognosbi/cgi-bin/cognos.cgi HTTP/1.1
#Accept: */*
#Accept-Language: en-us
#Referer: https://ag-intelligence.drj.com/cognosbi/pat/rsapp.htm
#soapaction: http://www.ibm.com/xmlns/prod/cognos/reportService/201109/
#Content-Type: text/xml; charset=utf-8
#Accept-Encoding: gzip, deflate
#User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .
NET4.0C; .NET4.0E; Ion 3.3.0.125; InfoPath.3; MDPA101; OWASMIME/4.0200)
#Host: ag-intelligence.drj.com
#Content-Length: 161667
#DNT: 1
#Connection: Keep-Alive
#Cache-Control: no-cache
#Cookie: c8server=https%3A%2F%2Fag-intelligence.drj.com%3A443%2Fcognosbi%2Fcgi-bin%2Fcognos.cgi; mob_lang=en; TSe36f05=b6a15d33eff8797d98bc75f22d2c85e1f
20fa2f8dda9abed5570682f728766c9684dc6d2c7f603263f299dc03470ed0e5dd2c529d1af78e016f94f63ba1abc528f04a493fe2f9ff9dd4ea3b99a99de5024487a2a9a99de5024487a2a9
a99de5024487a2a9a99de5024487a2a; cc_state="null"; cam_passport=MTsxMDE6ZmVjNDEzYzQtMGVjOC03YjgyLTJlNTgtYjgxNWFhNTE0ZDQxOjMzNjQ5NDA0MDQ7MDszOzA7; CRN=col
umnsPerPage%3D3%26contentLocale%3Den-us%26showHiddenObjects%3Dtrue%26skin%3Dcorporate%26linesPerPage%3D35%26showWelcomePage%3Dtrue%26http%3A%2F%2Fdevelo
per.cognos.com%2Fceba%2Fconstants%2FsystemOptionEnum%23accessibilityFeatures%3Dfalse%26listViewSeparator%3Dbackground%26displayMode%3Dlist%26timeZoneID%
3DEST%26http%3A%2F%2Fdeveloper.cognos.com%2Fceba%2Fconstants%2FbiDirectionalOptionEnum%23biDirectionalFeaturesEnabled%3Dfalse%26format%3DHTML%26automati
cPageRefresh%3D30%26productLocale%3Den%26useAccessibilityFeatures%3Dfalse%26showOptionSummary%3Dtrue%26; cea-ssa=false; usersessionid="AQgAAAA/tHlVAAAA
AoAAACSXrdg581uvb1aFAAAAFJ3FMiV1XFHYYjmBPlmT/mXlCpnFAAAAFlzIt8pvs4xewrB8PBrB1omTHOq"; userCapabilities=f%3Bfdbffc6d%3Be000002f%3Bff077efa%26ARQAAABSdxTI
ldVxR2GI5gT5Zk%2F5l5QqZwtel%2FW97E5sO8sAbwnsxaQA5mLc; cc_session="s_cc:|s_conf:na|s_sch:td|s_hd:sa|s_serv:na|s_disp:na|s_set:|s_dep:na|s_dir:na|s_sms:dd
|s_ct:sa|s_cs:sa|s_so:sa|e_hp:CAMID(*22DRJ*3au*3aagarwap*22)|e_proot:Public*20Folders|prootid:i1E4054857A7F43398F82386788807209|e_mroot:My*20Folders|mr
ootid:i45E22CE51A2D4F9DA668810E67B44CD2|e_mrootpath:CAMID(*22DrJ*3au*3aagarwap*22)*2ffolder*5b*40name*3d*27My*20Folders*27*5d|e_user:John*20Doe|
e_tenantDisplayName:|e_showTenantInfo:true|cl:en-us|dcid:i1E4054857A7F43398F82386788807209|show_logon:true|uig:|ui:|rsuiprofile:all|lch:f|lca:f|ci:f|wri
te:true|eom:0|pp:3364940404|null:|cachestamp:2015-06-04T13:04:34|null:"; BIGipServerag-intelligence.drj.us.secure=1090633994.47873.0000; TS83f7ea=c439d
424e740dc4c1ed042aa7bafa59df20fa2f8dda9abed5570851f1e66c5c4bbcf97c1
 
#<SOAP-ENV:Envelope SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:
SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:bus='http://developer.cognos.com/schemas/bibus/3/' xmlns:rns1='http://developer.cognos.com/schemas/rep
ortService/1'><SOAP-ENV:Header><bus:biBusHeader xsi:type="bus:biBusHeader"><bus:CAM xsi:type="bus:CAM"><authenticityToken xsi:type="xsd:base64Binary">Vj
FZcyLfKb7OMXsKwfDwawdaJkxzqg==</authenticityToken></bus:CAM><bus:CAF xsi:type="bus:CAF"><contextID xsi:type="xsd:string">CAFW00000070Q0FGQTNjMDAwMDAwD
GQUFBQUZKM0ZNaVYxWEZIWVlqbUJQbG1ULW1YbENwbld5amlFRU5ObWJJcTZHaEdLNTdjVWZ0Zy1GWV8zOTgxNzd8cnM_</contextID></bus:CAF><bus:userPreferenceVars SOAP-ENC:arra
yType="bus:userPreferenceVar[]" xsi:type="SOAP-ENC:Array"><item><bus:name xsi:type="xsd:string">productLocale</bus:name><bus:value xsi:type="xsd:string"
>en</bus:value></item><item><bus:name xsi:type="xsd:string">contentLocale</bus:name><bus:value xsi:type="xsd:string">en-us</bus:value></item></bus:userP
referenceVars><bus:dispatcherTransportVars xsi:type="SOAP-ENC:Array" SOAP-ENC:arrayType="bus:dispatcherTransportVar[]"><item xsi:type="bus:dispatcherTra
nsportVar"><name xsi:type="xsd:string">rs</name><value xsi:type="xsd:string">true</value></item></bus:dispatcherTransportVars></bus:biBusHeader></SOAP-E
NV:Header><SOAP-ENV:Body><rns1:update><object xsi:type="bus:report"><searchPath><value xsi:type="xsd:string">/content/folder[@name=&apos;Dashboards and
Portals&apos;]/folder[@name=&apos;DRJOPS (DrJ)&apos;]/folder[@name=&apos;Dashboard Reports&apos;]/report[@name="Sales"]</value></searchPath><specific
ation><value xsi:type="xsd:string" xml:space="preserve">&lt;report xmlns=&quot;http://developer.cognos.com/schemas/report/9.0/&quot; useStyleVersion=&qu
ot;10&quot; expressionLocale=&quot;en-us&quot;&gt;
#&lt;modelPath&gt;/content/folder[@name=&apos;Shared Packages&apos;]/package[@name=&apos;DrJ_Dashboard_Cube&apos;]/model[@name=&apos;model&apos;]&lt;
/modelPath&gt;
#&lt;drillBehavior modelBasedDrillThru=&quot;true&quot;/&gt;
#&lt;layouts&gt;
#&lt;layout&gt;
#&lt;reportPages&gt;
 
#&lt;page name=&quot;DrJ Dashbord&quot;&gt;
#&lt;pageBody&gt;
#&lt;contents&gt;&lt;table&gt;&lt;style&gt;&lt;defaultStyles&gt;&lt;defaultStyle refStyle=&quot;tb&quot;/&gt;&lt;/defaultStyles&gt;&lt;CSS value=&quot;f
ont-family:Arial;font-size:8pt;font-weight:bold&quot;/&gt;&lt;/style&gt;&lt;tableRows&gt;&lt;tableRow&gt;&lt;tableCells&gt;&lt;tableCell colSpan=&quot;1
&quot;&gt;&lt;contents&gt;&lt;table&gt;&lt;style&gt;&lt;defaultStyles&gt;&lt;defaultStyle refStyle=&quot;tb&quot;/&gt;&lt;/defaultStyles&gt;&lt;CSS valu
e=&quot;border-collapse:collapse;width:100%;text-align:center&quot;/&gt;&lt;/style&gt;&lt;tableRows&gt;&lt;tableRow&gt;&lt;tableCells&gt;&lt;tableCell&g
t;&lt;contents&gt;&lt;textItem&gt;&lt;dataSource&gt;&lt;staticValue&gt;In Thousands, Data As of: &lt;/staticValue&gt;&lt;/dataSource&gt;&lt;style&gt;&lt
;CSS value=&quot;color:#0D3F53&quot;/&gt;&lt;/style&gt;&lt;/textItem&gt;&lt;textItem&gt;&lt;dataSource&gt;&lt;reportExpression&gt;CubeDataUpdatedOn ([Gr
ower_Dashboard].[Year])&lt;/reportExpression&gt;&lt;/dataSource&gt;&lt;style&gt;&lt;CSS value=&quot;color:#0D3F53&quot;/&gt;&lt;/style&gt;&lt;/textItem&
gt;&lt;/contents&gt;&lt;style&gt;&lt;CSS value=&quot;text-align:left;vertical-align:middle;font-family:Arial;font-size:9pt;font-weight:bold;border-top-s
tyle:none;border-right-style:none;border-bottom-style:none;border-left-style:none&quot;/&gt;&lt;/style&gt;&lt;/tableCell&gt;&lt;tableCell&gt;&lt;content
s&gt;&lt;image&gt;
#&lt;dataSource&gt;
#&lt;staticValue&gt;../samples/images/Dashboard_Excel.png&lt;/staticValue&gt;
#&lt;/dataSource&gt;
#&lt;conditionalStyles&gt;&lt;conditionalStyleCases refVariable=&quot;For ReportOutput&quot;&gt;&lt;conditionalStyle refVariableValue=&quot;1&quot;&gt;&
lt;CSS value=&quot;display:none&quot;/&gt;&lt;/conditionalStyle&gt;&lt;/conditionalStyleCases&gt;&lt;conditionalStyleDefault/&gt;&lt;/conditionalStyles&
gt;&lt;reportDrills&gt;&lt;reportDrill name=&quot;Excel&quot;&gt;&lt;drillLabel&gt;&lt;dataSource&gt;&lt;staticValue/&gt;&lt;/dataSource&gt;&lt;/drillLa
bel&gt;&lt;drillTarget method=&quot;execute&quot; outputFormat=&quot;spreadsheetML&quot; showInNewWindow=&quot;true&quot;&gt;&lt;reportPath path=&quot;/
content/folder[@name=&apos;Dashboards and Portals&apos;]/folder[@name=&apos;USCROP (Grower)&apos;]/folder[@name=&apos;Dashboard Reports&apos;]/report[@n
ame=&apos;Sales&apos;]&quot;&gt;&lt;XMLAttributes&gt;&lt;XMLAttribute name=&quot;ReportName&quot; value=&quot;Sales&quot; output=&quot;no&quot;/&gt;&lt;
XMLAttribute name=&quot;class&quot; value=&quot;report&quot; output=&quot;no&quot;/&gt;&lt;/XMLAttributes&gt;&lt;/reportPath&gt;&lt;drillLinks&gt;&lt;dr
illLink&gt;&lt;drillTargetContext&gt;&lt;parameterContext parameter=&quot;Select Area&quot;/&gt;&lt;/drillTargetContext&gt;&lt;drillSourceContext&gt;&lt
;parameterContext parameter=&quot;Select Area&quot;/&gt;&lt;/drillSourceContext&gt;&lt;/drillLink&gt;&lt;drillLink&gt;&lt;drillTargetContext&gt;&lt;para
meterContext parameter=&quot;Select DrJIdea Specialist&quot;/&gt;&lt;/drillTargetContext&gt;&lt;drillSourceContext&gt;&lt;parameterContext parameter=
&quot;Select Innovation Specialist&quot;/&gt;&lt;/drillSourceContext&gt;&lt;/drillLink&gt;&lt;drillLink&gt;&lt;drillTargetContext&gt;&lt;parameterContex
t parameter=&quot;AreaTgtBy&quot;/&gt;&lt;/drillTargetContext&gt;&lt;drillSourceContext&gt;&lt;parameterContext parameter=&quot;AreaTgtBy&quot;/&gt;&lt;
/drillSourceContext&gt;&lt;/drillLink&gt;&lt;drillLink&gt;&lt;drillTargetContext&gt;&lt;parameterContext parameter=&quot;ISTgtBy&quot;/&gt;&lt;/drillTar
getContext&gt;&lt;drillSourceContext&gt;&lt;parameterContext parameter=&quot;ISTgtBy&quot;/&gt;&lt;/drillSourceContext&gt;&lt;/drillLink&gt;&lt;drillLin
k&gt;&lt;drillTargetContext&gt;&lt;parameterContext parameter=&quot;Grower Type&quot;/&gt;&lt;/drillTargetContext&gt;&lt;drillSourceContext&gt;&lt;param
eterContext parameter=&quot;Grower Type&quot;/&gt;&lt;/drillSourceContext&gt;&lt;/drillLink&gt;&lt;/drillLinks&gt;&lt;/drillTarget&gt;&lt;/reportDrill&g
t;&lt;/reportDrills&gt;&lt;/image&gt;&lt;image&gt;
#&lt;dataSource&gt;
#&lt;staticValue&gt;../samples/images/Dashboard_PDF.png&lt;/staticValue&gt;
#&lt;/dataSource&gt;
#&lt;conditionalStyles&gt;&lt;conditionalStyleCases refVariable=&quot;For ReportOutput&quot;&gt;&lt;conditionalStyle refVariableValue=&quot;1&quot;&gt;&
lt;CSS value=&quot;display:none&quot;/&gt;&lt;/conditionalStyle&gt;&lt;/conditionalStyleCases&gt;&lt;conditionalStyleDefault/&gt;&lt;/conditionalStyles&
gt;&lt;reportDrills&gt;&lt;reportDrill name=&quot;PDF&quot;&gt;&lt;drillLabel&gt;&lt;dataSource&gt;&lt;staticValue/&gt;&lt;/dataSource&gt;&lt;/drillLabe
l&gt;&lt;drillTarget method=&quot;execute&quot; outputFormat=&quot;PDF&quot; showInNewWindow=&quot;true&quot;&gt;&lt;reportPath path=&quot;/content/fold
er[@name=&apos;Dashboards and Portals&apos;]/folder[@name=&apos;DRJOPS (Grower)&apos;]/folder[@name=&apos;Dashboard Reports&apos;]/report[@name=&apos;Sa
les&apos;]&quot;&gt;
@lines = <STDIN>;
for $line (@lines) {
  $_ = $line;
  chomp;
  if (/^(GET|POST)\s+(\/.*)\s(http|HTTP)/) {
    $uri = $2;
    $method = $1;
    next;
  } elsif (/^User-Agent:/ && ! $datasec) {
    ($agent) = /^User-Agent:\s+(.+)$/;
    print "User-Agent: $agent\n" if $DEBUG;
    next;
  } elsif (/^Content-Length:\s/ && ! $datasec) {
# LWP calculates its own content-length header
      next;
  } elsif  (! $datasec) {
# add header to hash
      ($hdr,$val) = /^(\S+):\s+(.+)$/;
      print "hdr,val: $hdr, $val\n" if $DEBUG;
      $myhash{$hdr} = $val if $hdr;
  }
  if (/^Host: (\S+)/i && ! $datasec) {
    $host = $1;
  }
  $data .= $line if $datasec;
# test if we are at the end of the headers
  $datasec = 1 if /^$/;
}
 
# Create a user agent object
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->agent($agent);
$req = HTTP::Request->new($method => "https://$host$uri");
# enter all the headers
foreach $key (keys %myhash) {
  $val = $myhash{$key};
  print "key,val: $key, $val\n" if $DEBUG;
  $req->header($key => $val);
}
print "data: $data\n" if $DEBUG;
$req->content($data);
# Pass request to the user agent and get a response back
my $res = $ua->request($req);
 
# Check the outcome of the response
if ($res->is_success) {
 print $res->content;
} else {
  print $res->status_line, "\n";
}

Conclusion
We’ve created a Perl-based HTTP user agent to preserve headers. It’s not perfect but it’s a help, for instance in debugging am F5 web application firewall policy.