Intro
When debugging F5’s Web Application Firewall (AKA ASM) you get all the HTTP request headers in the GUI which you can cut and paste. Sometimes it is not so obvious how to fix a particular SupportID and you don’t want to bother the user to reproduce the error. This is code that allows you to scrape the headers and send them back, assuming you have a Linux server on the Internet. It’s not quite exact – Perl wants to add a TE header and maybe some others. More importantly, F5 only gives you around the first 5 KB of the headers, and often the real headers + data can be much longer than that. But It’s pretty good at preserving the basic headers and the POSTed data, if any.
The details
#!/usr/bin/perl # DrJ - 6/2015 # take waf Full Request as input from STDIN and spit it out use LWP; $DEBUG = 1; # example: #POST /cognosbi/cgi-bin/cognos.cgi HTTP/1.1 #Accept: */* #Accept-Language: en-us #Referer: https://ag-intelligence.drj.com/cognosbi/pat/rsapp.htm #soapaction: http://www.ibm.com/xmlns/prod/cognos/reportService/201109/ #Content-Type: text/xml; charset=utf-8 #Accept-Encoding: gzip, deflate #User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; . NET4.0C; .NET4.0E; Ion 3.3.0.125; InfoPath.3; MDPA101; OWASMIME/4.0200) #Host: ag-intelligence.drj.com #Content-Length: 161667 #DNT: 1 #Connection: Keep-Alive #Cache-Control: no-cache #Cookie: c8server=https%3A%2F%2Fag-intelligence.drj.com%3A443%2Fcognosbi%2Fcgi-bin%2Fcognos.cgi; mob_lang=en; TSe36f05=b6a15d33eff8797d98bc75f22d2c85e1f 20fa2f8dda9abed5570682f728766c9684dc6d2c7f603263f299dc03470ed0e5dd2c529d1af78e016f94f63ba1abc528f04a493fe2f9ff9dd4ea3b99a99de5024487a2a9a99de5024487a2a9 a99de5024487a2a9a99de5024487a2a; cc_state="null"; cam_passport=MTsxMDE6ZmVjNDEzYzQtMGVjOC03YjgyLTJlNTgtYjgxNWFhNTE0ZDQxOjMzNjQ5NDA0MDQ7MDszOzA7; CRN=col umnsPerPage%3D3%26contentLocale%3Den-us%26showHiddenObjects%3Dtrue%26skin%3Dcorporate%26linesPerPage%3D35%26showWelcomePage%3Dtrue%26http%3A%2F%2Fdevelo per.cognos.com%2Fceba%2Fconstants%2FsystemOptionEnum%23accessibilityFeatures%3Dfalse%26listViewSeparator%3Dbackground%26displayMode%3Dlist%26timeZoneID% 3DEST%26http%3A%2F%2Fdeveloper.cognos.com%2Fceba%2Fconstants%2FbiDirectionalOptionEnum%23biDirectionalFeaturesEnabled%3Dfalse%26format%3DHTML%26automati cPageRefresh%3D30%26productLocale%3Den%26useAccessibilityFeatures%3Dfalse%26showOptionSummary%3Dtrue%26; cea-ssa=false; usersessionid="AQgAAAA/tHlVAAAA AoAAACSXrdg581uvb1aFAAAAFJ3FMiV1XFHYYjmBPlmT/mXlCpnFAAAAFlzIt8pvs4xewrB8PBrB1omTHOq"; userCapabilities=f%3Bfdbffc6d%3Be000002f%3Bff077efa%26ARQAAABSdxTI ldVxR2GI5gT5Zk%2F5l5QqZwtel%2FW97E5sO8sAbwnsxaQA5mLc; cc_session="s_cc:|s_conf:na|s_sch:td|s_hd:sa|s_serv:na|s_disp:na|s_set:|s_dep:na|s_dir:na|s_sms:dd |s_ct:sa|s_cs:sa|s_so:sa|e_hp:CAMID(*22DRJ*3au*3aagarwap*22)|e_proot:Public*20Folders|prootid:i1E4054857A7F43398F82386788807209|e_mroot:My*20Folders|mr ootid:i45E22CE51A2D4F9DA668810E67B44CD2|e_mrootpath:CAMID(*22DrJ*3au*3aagarwap*22)*2ffolder*5b*40name*3d*27My*20Folders*27*5d|e_user:John*20Doe| e_tenantDisplayName:|e_showTenantInfo:true|cl:en-us|dcid:i1E4054857A7F43398F82386788807209|show_logon:true|uig:|ui:|rsuiprofile:all|lch:f|lca:f|ci:f|wri te:true|eom:0|pp:3364940404|null:|cachestamp:2015-06-04T13:04:34|null:"; BIGipServerag-intelligence.drj.us.secure=1090633994.47873.0000; TS83f7ea=c439d 424e740dc4c1ed042aa7bafa59df20fa2f8dda9abed5570851f1e66c5c4bbcf97c1 #<SOAP-ENV:Envelope SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns: SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:bus='http://developer.cognos.com/schemas/bibus/3/' xmlns:rns1='http://developer.cognos.com/schemas/rep ortService/1'><SOAP-ENV:Header><bus:biBusHeader xsi:type="bus:biBusHeader"><bus:CAM xsi:type="bus:CAM"><authenticityToken xsi:type="xsd:base64Binary">Vj FZcyLfKb7OMXsKwfDwawdaJkxzqg==</authenticityToken></bus:CAM><bus:CAF xsi:type="bus:CAF"><contextID xsi:type="xsd:string">CAFW00000070Q0FGQTNjMDAwMDAwD GQUFBQUZKM0ZNaVYxWEZIWVlqbUJQbG1ULW1YbENwbld5amlFRU5ObWJJcTZHaEdLNTdjVWZ0Zy1GWV8zOTgxNzd8cnM_</contextID></bus:CAF><bus:userPreferenceVars SOAP-ENC:arra yType="bus:userPreferenceVar[]" xsi:type="SOAP-ENC:Array"><item><bus:name xsi:type="xsd:string">productLocale</bus:name><bus:value xsi:type="xsd:string" >en</bus:value></item><item><bus:name xsi:type="xsd:string">contentLocale</bus:name><bus:value xsi:type="xsd:string">en-us</bus:value></item></bus:userP referenceVars><bus:dispatcherTransportVars xsi:type="SOAP-ENC:Array" SOAP-ENC:arrayType="bus:dispatcherTransportVar[]"><item xsi:type="bus:dispatcherTra nsportVar"><name xsi:type="xsd:string">rs</name><value xsi:type="xsd:string">true</value></item></bus:dispatcherTransportVars></bus:biBusHeader></SOAP-E NV:Header><SOAP-ENV:Body><rns1:update><object xsi:type="bus:report"><searchPath><value xsi:type="xsd:string">/content/folder[@name='Dashboards and Portals']/folder[@name='DRJOPS (DrJ)']/folder[@name='Dashboard Reports']/report[@name="Sales"]</value></searchPath><specific ation><value xsi:type="xsd:string" xml:space="preserve"><report xmlns="http://developer.cognos.com/schemas/report/9.0/" useStyleVersion=&qu ot;10" expressionLocale="en-us"> #<modelPath>/content/folder[@name='Shared Packages']/package[@name='DrJ_Dashboard_Cube']/model[@name='model']< /modelPath> #<drillBehavior modelBasedDrillThru="true"/> #<layouts> #<layout> #<reportPages> #<page name="DrJ Dashbord"> #<pageBody> #<contents><table><style><defaultStyles><defaultStyle refStyle="tb"/></defaultStyles><CSS value="f ont-family:Arial;font-size:8pt;font-weight:bold"/></style><tableRows><tableRow><tableCells><tableCell colSpan="1 "><contents><table><style><defaultStyles><defaultStyle refStyle="tb"/></defaultStyles><CSS valu e="border-collapse:collapse;width:100%;text-align:center"/></style><tableRows><tableRow><tableCells><tableCell&g t;<contents><textItem><dataSource><staticValue>In Thousands, Data As of: </staticValue></dataSource><style>< ;CSS value="color:#0D3F53"/></style></textItem><textItem><dataSource><reportExpression>CubeDataUpdatedOn ([Gr ower_Dashboard].[Year])</reportExpression></dataSource><style><CSS value="color:#0D3F53"/></style></textItem& gt;</contents><style><CSS value="text-align:left;vertical-align:middle;font-family:Arial;font-size:9pt;font-weight:bold;border-top-s tyle:none;border-right-style:none;border-bottom-style:none;border-left-style:none"/></style></tableCell><tableCell><content s><image> #<dataSource> #<staticValue>../samples/images/Dashboard_Excel.png</staticValue> #</dataSource> #<conditionalStyles><conditionalStyleCases refVariable="For ReportOutput"><conditionalStyle refVariableValue="1">& lt;CSS value="display:none"/></conditionalStyle></conditionalStyleCases><conditionalStyleDefault/></conditionalStyles& gt;<reportDrills><reportDrill name="Excel"><drillLabel><dataSource><staticValue/></dataSource></drillLa bel><drillTarget method="execute" outputFormat="spreadsheetML" showInNewWindow="true"><reportPath path="/ content/folder[@name='Dashboards and Portals']/folder[@name='USCROP (Grower)']/folder[@name='Dashboard Reports']/report[@n ame='Sales']"><XMLAttributes><XMLAttribute name="ReportName" value="Sales" output="no"/>< XMLAttribute name="class" value="report" output="no"/></XMLAttributes></reportPath><drillLinks><dr illLink><drillTargetContext><parameterContext parameter="Select Area"/></drillTargetContext><drillSourceContext>< ;parameterContext parameter="Select Area"/></drillSourceContext></drillLink><drillLink><drillTargetContext><para meterContext parameter="Select DrJIdea Specialist"/></drillTargetContext><drillSourceContext><parameterContext parameter= "Select Innovation Specialist"/></drillSourceContext></drillLink><drillLink><drillTargetContext><parameterContex t parameter="AreaTgtBy"/></drillTargetContext><drillSourceContext><parameterContext parameter="AreaTgtBy"/>< /drillSourceContext></drillLink><drillLink><drillTargetContext><parameterContext parameter="ISTgtBy"/></drillTar getContext><drillSourceContext><parameterContext parameter="ISTgtBy"/></drillSourceContext></drillLink><drillLin k><drillTargetContext><parameterContext parameter="Grower Type"/></drillTargetContext><drillSourceContext><param eterContext parameter="Grower Type"/></drillSourceContext></drillLink></drillLinks></drillTarget></reportDrill&g t;</reportDrills></image><image> #<dataSource> #<staticValue>../samples/images/Dashboard_PDF.png</staticValue> #</dataSource> #<conditionalStyles><conditionalStyleCases refVariable="For ReportOutput"><conditionalStyle refVariableValue="1">& lt;CSS value="display:none"/></conditionalStyle></conditionalStyleCases><conditionalStyleDefault/></conditionalStyles& gt;<reportDrills><reportDrill name="PDF"><drillLabel><dataSource><staticValue/></dataSource></drillLabe l><drillTarget method="execute" outputFormat="PDF" showInNewWindow="true"><reportPath path="/content/fold er[@name='Dashboards and Portals']/folder[@name='DRJOPS (Grower)']/folder[@name='Dashboard Reports']/report[@name='Sa les']"> @lines = <STDIN>; for $line (@lines) { $_ = $line; chomp; if (/^(GET|POST)\s+(\/.*)\s(http|HTTP)/) { $uri = $2; $method = $1; next; } elsif (/^User-Agent:/ && ! $datasec) { ($agent) = /^User-Agent:\s+(.+)$/; print "User-Agent: $agent\n" if $DEBUG; next; } elsif (/^Content-Length:\s/ && ! $datasec) { # LWP calculates its own content-length header next; } elsif (! $datasec) { # add header to hash ($hdr,$val) = /^(\S+):\s+(.+)$/; print "hdr,val: $hdr, $val\n" if $DEBUG; $myhash{$hdr} = $val if $hdr; } if (/^Host: (\S+)/i && ! $datasec) { $host = $1; } $data .= $line if $datasec; # test if we are at the end of the headers $datasec = 1 if /^$/; } # Create a user agent object use LWP::UserAgent; $ua = LWP::UserAgent->new; $ua->agent($agent); $req = HTTP::Request->new($method => "https://$host$uri"); # enter all the headers foreach $key (keys %myhash) { $val = $myhash{$key}; print "key,val: $key, $val\n" if $DEBUG; $req->header($key => $val); } print "data: $data\n" if $DEBUG; $req->content($data); # Pass request to the user agent and get a response back my $res = $ua->request($req); # Check the outcome of the response if ($res->is_success) { print $res->content; } else { print $res->status_line, "\n"; } |
Conclusion
We’ve created a Perl-based HTTP user agent to preserve headers. It’s not perfect but it’s a help, for instance in debugging am F5 web application firewall policy.