Categories
Linux Perl

HTTP replay user-agent

Intro
When debugging F5’s Web Application Firewall (AKA ASM) you get all the HTTP request headers in the GUI which you can cut and paste. Sometimes it is not so obvious how to fix a particular SupportID and you don’t want to bother the user to reproduce the error. This is code that allows you to scrape the headers and send them back, assuming you have a Linux server on the Internet. It’s not quite exact – Perl wants to add a TE header and maybe some others. More importantly, F5 only gives you around the first 5 KB of the headers, and often the real headers + data can be much longer than that. But It’s pretty good at preserving the basic headers and the POSTed data, if any.

The details

#!/usr/bin/perl
# DrJ - 6/2015
# take waf Full Request as input from STDIN and spit it out
use LWP;
$DEBUG = 1;
# example:
#POST /cognosbi/cgi-bin/cognos.cgi HTTP/1.1
#Accept: */*
#Accept-Language: en-us
#Referer: https://ag-intelligence.drj.com/cognosbi/pat/rsapp.htm
#soapaction: http://www.ibm.com/xmlns/prod/cognos/reportService/201109/
#Content-Type: text/xml; charset=utf-8
#Accept-Encoding: gzip, deflate
#User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .
NET4.0C; .NET4.0E; Ion 3.3.0.125; InfoPath.3; MDPA101; OWASMIME/4.0200)
#Host: ag-intelligence.drj.com
#Content-Length: 161667
#DNT: 1
#Connection: Keep-Alive
#Cache-Control: no-cache
#Cookie: c8server=https%3A%2F%2Fag-intelligence.drj.com%3A443%2Fcognosbi%2Fcgi-bin%2Fcognos.cgi; mob_lang=en; TSe36f05=b6a15d33eff8797d98bc75f22d2c85e1f
20fa2f8dda9abed5570682f728766c9684dc6d2c7f603263f299dc03470ed0e5dd2c529d1af78e016f94f63ba1abc528f04a493fe2f9ff9dd4ea3b99a99de5024487a2a9a99de5024487a2a9
a99de5024487a2a9a99de5024487a2a; cc_state="null"; cam_passport=MTsxMDE6ZmVjNDEzYzQtMGVjOC03YjgyLTJlNTgtYjgxNWFhNTE0ZDQxOjMzNjQ5NDA0MDQ7MDszOzA7; CRN=col
umnsPerPage%3D3%26contentLocale%3Den-us%26showHiddenObjects%3Dtrue%26skin%3Dcorporate%26linesPerPage%3D35%26showWelcomePage%3Dtrue%26http%3A%2F%2Fdevelo
per.cognos.com%2Fceba%2Fconstants%2FsystemOptionEnum%23accessibilityFeatures%3Dfalse%26listViewSeparator%3Dbackground%26displayMode%3Dlist%26timeZoneID%
3DEST%26http%3A%2F%2Fdeveloper.cognos.com%2Fceba%2Fconstants%2FbiDirectionalOptionEnum%23biDirectionalFeaturesEnabled%3Dfalse%26format%3DHTML%26automati
cPageRefresh%3D30%26productLocale%3Den%26useAccessibilityFeatures%3Dfalse%26showOptionSummary%3Dtrue%26; cea-ssa=false; usersessionid="AQgAAAA/tHlVAAAA
AoAAACSXrdg581uvb1aFAAAAFJ3FMiV1XFHYYjmBPlmT/mXlCpnFAAAAFlzIt8pvs4xewrB8PBrB1omTHOq"; userCapabilities=f%3Bfdbffc6d%3Be000002f%3Bff077efa%26ARQAAABSdxTI
ldVxR2GI5gT5Zk%2F5l5QqZwtel%2FW97E5sO8sAbwnsxaQA5mLc; cc_session="s_cc:|s_conf:na|s_sch:td|s_hd:sa|s_serv:na|s_disp:na|s_set:|s_dep:na|s_dir:na|s_sms:dd
|s_ct:sa|s_cs:sa|s_so:sa|e_hp:CAMID(*22DRJ*3au*3aagarwap*22)|e_proot:Public*20Folders|prootid:i1E4054857A7F43398F82386788807209|e_mroot:My*20Folders|mr
ootid:i45E22CE51A2D4F9DA668810E67B44CD2|e_mrootpath:CAMID(*22DrJ*3au*3aagarwap*22)*2ffolder*5b*40name*3d*27My*20Folders*27*5d|e_user:John*20Doe|
e_tenantDisplayName:|e_showTenantInfo:true|cl:en-us|dcid:i1E4054857A7F43398F82386788807209|show_logon:true|uig:|ui:|rsuiprofile:all|lch:f|lca:f|ci:f|wri
te:true|eom:0|pp:3364940404|null:|cachestamp:2015-06-04T13:04:34|null:"; BIGipServerag-intelligence.drj.us.secure=1090633994.47873.0000; TS83f7ea=c439d
424e740dc4c1ed042aa7bafa59df20fa2f8dda9abed5570851f1e66c5c4bbcf97c1
 
#<SOAP-ENV:Envelope SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:
SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:bus='http://developer.cognos.com/schemas/bibus/3/' xmlns:rns1='http://developer.cognos.com/schemas/rep
ortService/1'><SOAP-ENV:Header><bus:biBusHeader xsi:type="bus:biBusHeader"><bus:CAM xsi:type="bus:CAM"><authenticityToken xsi:type="xsd:base64Binary">Vj
FZcyLfKb7OMXsKwfDwawdaJkxzqg==</authenticityToken></bus:CAM><bus:CAF xsi:type="bus:CAF"><contextID xsi:type="xsd:string">CAFW00000070Q0FGQTNjMDAwMDAwD
GQUFBQUZKM0ZNaVYxWEZIWVlqbUJQbG1ULW1YbENwbld5amlFRU5ObWJJcTZHaEdLNTdjVWZ0Zy1GWV8zOTgxNzd8cnM_</contextID></bus:CAF><bus:userPreferenceVars SOAP-ENC:arra
yType="bus:userPreferenceVar[]" xsi:type="SOAP-ENC:Array"><item><bus:name xsi:type="xsd:string">productLocale</bus:name><bus:value xsi:type="xsd:string"
>en</bus:value></item><item><bus:name xsi:type="xsd:string">contentLocale</bus:name><bus:value xsi:type="xsd:string">en-us</bus:value></item></bus:userP
referenceVars><bus:dispatcherTransportVars xsi:type="SOAP-ENC:Array" SOAP-ENC:arrayType="bus:dispatcherTransportVar[]"><item xsi:type="bus:dispatcherTra
nsportVar"><name xsi:type="xsd:string">rs</name><value xsi:type="xsd:string">true</value></item></bus:dispatcherTransportVars></bus:biBusHeader></SOAP-E
NV:Header><SOAP-ENV:Body><rns1:update><object xsi:type="bus:report"><searchPath><value xsi:type="xsd:string">/content/folder[@name=&apos;Dashboards and
Portals&apos;]/folder[@name=&apos;DRJOPS (DrJ)&apos;]/folder[@name=&apos;Dashboard Reports&apos;]/report[@name="Sales"]</value></searchPath><specific
ation><value xsi:type="xsd:string" xml:space="preserve">&lt;report xmlns=&quot;http://developer.cognos.com/schemas/report/9.0/&quot; useStyleVersion=&qu
ot;10&quot; expressionLocale=&quot;en-us&quot;&gt;
#&lt;modelPath&gt;/content/folder[@name=&apos;Shared Packages&apos;]/package[@name=&apos;DrJ_Dashboard_Cube&apos;]/model[@name=&apos;model&apos;]&lt;
/modelPath&gt;
#&lt;drillBehavior modelBasedDrillThru=&quot;true&quot;/&gt;
#&lt;layouts&gt;
#&lt;layout&gt;
#&lt;reportPages&gt;
 
#&lt;page name=&quot;DrJ Dashbord&quot;&gt;
#&lt;pageBody&gt;
#&lt;contents&gt;&lt;table&gt;&lt;style&gt;&lt;defaultStyles&gt;&lt;defaultStyle refStyle=&quot;tb&quot;/&gt;&lt;/defaultStyles&gt;&lt;CSS value=&quot;f
ont-family:Arial;font-size:8pt;font-weight:bold&quot;/&gt;&lt;/style&gt;&lt;tableRows&gt;&lt;tableRow&gt;&lt;tableCells&gt;&lt;tableCell colSpan=&quot;1
&quot;&gt;&lt;contents&gt;&lt;table&gt;&lt;style&gt;&lt;defaultStyles&gt;&lt;defaultStyle refStyle=&quot;tb&quot;/&gt;&lt;/defaultStyles&gt;&lt;CSS valu
e=&quot;border-collapse:collapse;width:100%;text-align:center&quot;/&gt;&lt;/style&gt;&lt;tableRows&gt;&lt;tableRow&gt;&lt;tableCells&gt;&lt;tableCell&g
t;&lt;contents&gt;&lt;textItem&gt;&lt;dataSource&gt;&lt;staticValue&gt;In Thousands, Data As of: &lt;/staticValue&gt;&lt;/dataSource&gt;&lt;style&gt;&lt
;CSS value=&quot;color:#0D3F53&quot;/&gt;&lt;/style&gt;&lt;/textItem&gt;&lt;textItem&gt;&lt;dataSource&gt;&lt;reportExpression&gt;CubeDataUpdatedOn ([Gr
ower_Dashboard].[Year])&lt;/reportExpression&gt;&lt;/dataSource&gt;&lt;style&gt;&lt;CSS value=&quot;color:#0D3F53&quot;/&gt;&lt;/style&gt;&lt;/textItem&
gt;&lt;/contents&gt;&lt;style&gt;&lt;CSS value=&quot;text-align:left;vertical-align:middle;font-family:Arial;font-size:9pt;font-weight:bold;border-top-s
tyle:none;border-right-style:none;border-bottom-style:none;border-left-style:none&quot;/&gt;&lt;/style&gt;&lt;/tableCell&gt;&lt;tableCell&gt;&lt;content
s&gt;&lt;image&gt;
#&lt;dataSource&gt;
#&lt;staticValue&gt;../samples/images/Dashboard_Excel.png&lt;/staticValue&gt;
#&lt;/dataSource&gt;
#&lt;conditionalStyles&gt;&lt;conditionalStyleCases refVariable=&quot;For ReportOutput&quot;&gt;&lt;conditionalStyle refVariableValue=&quot;1&quot;&gt;&
lt;CSS value=&quot;display:none&quot;/&gt;&lt;/conditionalStyle&gt;&lt;/conditionalStyleCases&gt;&lt;conditionalStyleDefault/&gt;&lt;/conditionalStyles&
gt;&lt;reportDrills&gt;&lt;reportDrill name=&quot;Excel&quot;&gt;&lt;drillLabel&gt;&lt;dataSource&gt;&lt;staticValue/&gt;&lt;/dataSource&gt;&lt;/drillLa
bel&gt;&lt;drillTarget method=&quot;execute&quot; outputFormat=&quot;spreadsheetML&quot; showInNewWindow=&quot;true&quot;&gt;&lt;reportPath path=&quot;/
content/folder[@name=&apos;Dashboards and Portals&apos;]/folder[@name=&apos;USCROP (Grower)&apos;]/folder[@name=&apos;Dashboard Reports&apos;]/report[@n
ame=&apos;Sales&apos;]&quot;&gt;&lt;XMLAttributes&gt;&lt;XMLAttribute name=&quot;ReportName&quot; value=&quot;Sales&quot; output=&quot;no&quot;/&gt;&lt;
XMLAttribute name=&quot;class&quot; value=&quot;report&quot; output=&quot;no&quot;/&gt;&lt;/XMLAttributes&gt;&lt;/reportPath&gt;&lt;drillLinks&gt;&lt;dr
illLink&gt;&lt;drillTargetContext&gt;&lt;parameterContext parameter=&quot;Select Area&quot;/&gt;&lt;/drillTargetContext&gt;&lt;drillSourceContext&gt;&lt
;parameterContext parameter=&quot;Select Area&quot;/&gt;&lt;/drillSourceContext&gt;&lt;/drillLink&gt;&lt;drillLink&gt;&lt;drillTargetContext&gt;&lt;para
meterContext parameter=&quot;Select DrJIdea Specialist&quot;/&gt;&lt;/drillTargetContext&gt;&lt;drillSourceContext&gt;&lt;parameterContext parameter=
&quot;Select Innovation Specialist&quot;/&gt;&lt;/drillSourceContext&gt;&lt;/drillLink&gt;&lt;drillLink&gt;&lt;drillTargetContext&gt;&lt;parameterContex
t parameter=&quot;AreaTgtBy&quot;/&gt;&lt;/drillTargetContext&gt;&lt;drillSourceContext&gt;&lt;parameterContext parameter=&quot;AreaTgtBy&quot;/&gt;&lt;
/drillSourceContext&gt;&lt;/drillLink&gt;&lt;drillLink&gt;&lt;drillTargetContext&gt;&lt;parameterContext parameter=&quot;ISTgtBy&quot;/&gt;&lt;/drillTar
getContext&gt;&lt;drillSourceContext&gt;&lt;parameterContext parameter=&quot;ISTgtBy&quot;/&gt;&lt;/drillSourceContext&gt;&lt;/drillLink&gt;&lt;drillLin
k&gt;&lt;drillTargetContext&gt;&lt;parameterContext parameter=&quot;Grower Type&quot;/&gt;&lt;/drillTargetContext&gt;&lt;drillSourceContext&gt;&lt;param
eterContext parameter=&quot;Grower Type&quot;/&gt;&lt;/drillSourceContext&gt;&lt;/drillLink&gt;&lt;/drillLinks&gt;&lt;/drillTarget&gt;&lt;/reportDrill&g
t;&lt;/reportDrills&gt;&lt;/image&gt;&lt;image&gt;
#&lt;dataSource&gt;
#&lt;staticValue&gt;../samples/images/Dashboard_PDF.png&lt;/staticValue&gt;
#&lt;/dataSource&gt;
#&lt;conditionalStyles&gt;&lt;conditionalStyleCases refVariable=&quot;For ReportOutput&quot;&gt;&lt;conditionalStyle refVariableValue=&quot;1&quot;&gt;&
lt;CSS value=&quot;display:none&quot;/&gt;&lt;/conditionalStyle&gt;&lt;/conditionalStyleCases&gt;&lt;conditionalStyleDefault/&gt;&lt;/conditionalStyles&
gt;&lt;reportDrills&gt;&lt;reportDrill name=&quot;PDF&quot;&gt;&lt;drillLabel&gt;&lt;dataSource&gt;&lt;staticValue/&gt;&lt;/dataSource&gt;&lt;/drillLabe
l&gt;&lt;drillTarget method=&quot;execute&quot; outputFormat=&quot;PDF&quot; showInNewWindow=&quot;true&quot;&gt;&lt;reportPath path=&quot;/content/fold
er[@name=&apos;Dashboards and Portals&apos;]/folder[@name=&apos;DRJOPS (Grower)&apos;]/folder[@name=&apos;Dashboard Reports&apos;]/report[@name=&apos;Sa
les&apos;]&quot;&gt;
@lines = <STDIN>;
for $line (@lines) {
  $_ = $line;
  chomp;
  if (/^(GET|POST)\s+(\/.*)\s(http|HTTP)/) {
    $uri = $2;
    $method = $1;
    next;
  } elsif (/^User-Agent:/ && ! $datasec) {
    ($agent) = /^User-Agent:\s+(.+)$/;
    print "User-Agent: $agent\n" if $DEBUG;
    next;
  } elsif (/^Content-Length:\s/ && ! $datasec) {
# LWP calculates its own content-length header
      next;
  } elsif  (! $datasec) {
# add header to hash
      ($hdr,$val) = /^(\S+):\s+(.+)$/;
      print "hdr,val: $hdr, $val\n" if $DEBUG;
      $myhash{$hdr} = $val if $hdr;
  }
  if (/^Host: (\S+)/i && ! $datasec) {
    $host = $1;
  }
  $data .= $line if $datasec;
# test if we are at the end of the headers
  $datasec = 1 if /^$/;
}
 
# Create a user agent object
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->agent($agent);
$req = HTTP::Request->new($method => "https://$host$uri");
# enter all the headers
foreach $key (keys %myhash) {
  $val = $myhash{$key};
  print "key,val: $key, $val\n" if $DEBUG;
  $req->header($key => $val);
}
print "data: $data\n" if $DEBUG;
$req->content($data);
# Pass request to the user agent and get a response back
my $res = $ua->request($req);
 
# Check the outcome of the response
if ($res->is_success) {
 print $res->content;
} else {
  print $res->status_line, "\n";
}

Conclusion
We’ve created a Perl-based HTTP user agent to preserve headers. It’s not perfect but it’s a help, for instance in debugging am F5 web application firewall policy.

Categories
DNS Linux Network Technologies Perl

Announcing a simple DNS web interface and code

Intro
For demonstration purposes I’ve written a WEB interface to do DNS queries. This can be used for light querying. Once it gets abused I will pull it from the web site.

Motivation
Some large enterprises are behind not only a corporate firewall, but also confined to a private namespace with no access to Internet name resolution. Users in such situations can use one of the many available tools to do DNS resolution through the web, but they all want to throw advertising at you and it’s not clear which can be trusted not to load you up with spyware. I am offering this ad-free DNS lookup using my position on the Internet as a trusted source.

And if you’re lucky and looking for code to do this yourself, you might find it. But nowhere will you find a site that’s running its own published code for DNS resolution. Except here.

The code
Admittedly very simple-minded, but hopefully not fatally flawed, here it is in Perl.

#!/usr/bin/perl
use CGI;
$query = new CGI;
%allowedArgs = (domainname =&gt; 'dum',type =&gt; 'dum',short =&gt; 'dum');
#
print "Content-type: text/html\n\n";
print "
\n";
foreach $key ($query->param) {
  exit(1) unless defined $allowedArgs{$key};
  exit(1) if $query->param($key) !~ /^([a-zA-Z0-9\.-]){2,256}$/;
  print "$key " . $query->param($key) . "\n";
}
# possible keys: domainname, type
$domainname = $query->param(domainname);
$type     = $query->param(type);
$type = "any" unless $type;
# argument validation checks
exit(1) if $domainname !~ /^([a-zA-Z0-9\.-]){2,256}$/ || $domainname =~ /\.\./ ||  ! $domainname;
exit(1) if $type !~ /^([a-zA-Z]){1,8}$/;

# short answer?
$short = "+short" if defined $query->param(short);

# authoritative request?
if (defined $query->param(authoritative)) {
# this will be a lot more complicated and so is not implemented. Perhaps someday if there is a request...
}

open(DIG,"dig $short $type $domainname|") || die "Cannot run dig!!\n";
while() {
  print ;
}

Yes it’s very old-school. I do not even use a DNS package. Why bother? It’s not rocket science. There’s a lot more to argument validation than it looks like – you would not believe the evil things people send to your web server. So you have to vigilant about injection attacks or shelling out by use of unexpected characters.

Usage

2020 Update

This URL has been deactivated since I moved to my new server. I’ll have to see if there’s time and interest to restore this functionality.

example 1

https://drjohnstechtalk.com/cgi-bin/digiface.cgi?domainname=johnstechtalk.com&type=a

domainname johnstechtalk.com
type a
 
; &lt;&lt;&gt;&gt; DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.4 &lt;&lt;&gt;&gt; a johnstechtalk.com
;; global options: +cmd
;; Got answer:
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 8711
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
 
;; QUESTION SECTION:
;johnstechtalk.com.		IN	A
 
;; ANSWER SECTION:
johnstechtalk.com.	3600	IN	A	50.17.188.196
 
;; Query time: 10 msec
;; SERVER: 172.16.0.23#53(172.16.0.23)
;; WHEN: Mon May  4 14:59:05 2015
;; MSG SIZE  rcvd: 51

example 2
https://drjohnstechtalk.com/cgi-bin/digiface.cgi?domainname=drjohnstechtalk.com&short

domainname drjohnstechtalk.com
short 
50.17.188.196

Familiarity with dig will help you determine the best switches to use as you can see that at the end of the day it is merely calling dig and sending back that output with a minimum of html markup. This will make it easy to parse the output programatically.

Conclusion
A simple DNS web interface is being announced today. Both the service and the code are being made available. The service may be pulled once it becomes abused.

References
A nice, not too commercial web interface to dig and traceroute that is more user-friendly than mine is http://www.kloth.net/services/dig.php
The dig man pages can be helpful.

Got a geoDNS entry? Although this link has ads, it’s quite interesting because it sends your query to open DNS servers around the world: https://dnschecker.org/.

You can explore some details behind Google’s public resolving server 8.8.8.8 by using the web site: https://dns.google.com/. It’s quite helpful.

I won’t paste the link to my service but you can see what it is from the examples above.

There’s a simple but effective DIG available for your Android smartphone from the Playstore. That’s DNS debugger from TurboBytes. No obnoxious ads and yet no cost.

Of course if you are on the Internet and have access to dig, Google’s DNS servers are available for you to use directly.

Want to learn if the Great Friewall of China is clobbering the expected DNS result? The site https://viewdns.info/chinesefirewall/ is designed to do just that.

Categories
Linux Python Raspberry Pi

What I’m working on: a Raspberry Pi digital photo frame

Intro
The idea is that for a display kiosk let’s have a Raspberry Pi drive a display like one of those electronic picture frames. Power the thing up, perhaps plug in a flash drive, leave off the mouse and keyboard, but have a display attached, and get it to where it just automatically starts a slideshow without more fuss.

Some discarded options
Obviously this is not breaking new ground. you can find many variants of this on the Internet. An early-on approach that caught my eye is flickrframe. I read the source code to learn that at the end of the day it relies on the fbi program (frame buffer imageviewer). I thought that perhaps I could rip out the part that connects to Flickr but it seemed like too much trouble. At the end of the day it’s just a question of whether to use fbi or not.

Then there’s Raspberry Pi slideshow. That’s a quite good write-up. That’s using pqiv. I think that solution is workable.

But the one I’m focusing on uses qiv. You would have thought that pqiv would rely on qiv (quick image viewer) but it appears not to. So qiv is a separate install. qiv has lots of switches so it’s been written with this kind of thing in mind it seems.

What it looks like so far

#!/bin/sh
# -f : full-screen; -R : disable deletion; -s : slideshow; -d : delay <secs>; -i : status-bar;
# -m : zoom; [-r : ranomdize]
# this doesn't handle filenames with spaces:
##cd /media; qiv -f -R -s -d 5 -i -m `find /media -regex ".+\.jpe?g$"`
# this one does:
if [ "$1" = "l" ]; then
# print out proposed filenames
  cd /media; find . -regex ".+\.jpe?g$"
else
  sleep 5
  cd /media; find . -regex ".+\.jpe?g$" -print0|xargs -0 qiv -f -R -s -d 5 -i -m
fi

The idea being, why not make a slideshow out of all the pictures found on a flash drive that’s been inserted into the Pi? That’s how a standard picture frame works after all. It’s a very convenient way to work with it. That’s the aim of the above script.

Requirements update
OK. Well this happens a lot in IT. We thought we were solving one problem but when we finally spoke with the visual arts team they had something entirely different in mind. They want to mix in movies as well. fbi, pqiv or qiv don’t handle movies. I have mplayer and vlc from my playing around with Raspberry Pi camera. mplayer runs like a dog on the movie files I tried, perhaps one frame update every two seconds. After more searching around I came across omxplayer. That actually works pretty well. It on the other hand doesn’t seem up to the task of handling a mixed multimedia stream of stills and movies. But it did handle the two movies types we had: .mov and .mp4 movie files. omxplayer is written specifically for the Pi so it uses its GPU for frame acceleration. mplayer just seems to rely on the CPU which just can’t keep up on a high-def quality movie. So as a result omxplayer will only play through a true graphical console. It doesn’t even bother you to get your DISPLAY environment variable set up correctly – it’s just going to send everything to the head display.

Overheard recently
And when using my TV as display omxplayer put out the sound, too, perfectly synchronized and of high quality.

I was thinking if we should kludge stitching together qiv and omxplayer. You know letting one lapse and starting up the other to transition from a still to a movie, but I don’t know how to make the transition smooth. So i searched around yet some more and found pipresents. I believe it is a python framework around omxplayer. It’s pretty sophisticated and yet free. It’s actually aimed at museums and can include reactions to pressed buttons as you have at museum displays. So far we got the example media show to loop through – it demonstrates a high-quality short movie and a still plus some captions at the beginning.

Pipresents isn’t perfect however
I quickly found some problems with pipresents so I went the official route and posted them to the github site, not really knowing what to expect. The first issue is that you are not allowed to import .mov files! That makes no sense since omxplayer plays them. So I post this bug and that very same day the author emails me back and explains that you simply edit pp_editor.py line 32 and add .mov as an additional video file type! Sure enough, that did it. Then I found that it wasn’t downsampling my images. These days everyone has a camera or phone that takes mmulti-megapixel images far exceeding a cheap display’s 1280×1024 resolution. So you only see a small portion of your jpeg. I just assumed pipresents would downsample these large pictures because the other packages like qiv do it so readily. Again the same day the author gets back to me and says no this isn’t supported – in pipresents. But there is a solution: I should use pipresents-next! It’s officially in beta but just about ready for production release. I don’t think I’ll go that route but it’s always nice to know your package continues to be developed. I’ve written my own downsampler which I will provide later on.

Screen turns off
The pipresents has a command-line switch, -b, to prevent screen blanking. But I think in general it’s better to not use that switch and instead disable screen blanking in general.

$ sudo nano /etc/kbd/config
– change BLANK_TIME=30 to BLANK-TIME=0
– and change POWERDOWN_TIME=30 to POWERDOWN_TIME=0
$ sudo nano /etc/lightdm/lightdm.conf
– below the [SeatDefault] line create this line:
xserver-command=X -s 0 dpms

How to get started with PiPresents
$ wget https://github.com/KenT2/pipresents/tarball/master -O – | tar xz

There should now be a directory ‘KenT2-pipresents-xxxx’ in your home directory. Rename the directory to pipresents:

$ mv KenT2* pipresents

To save time make sure you have two terminal windows open on your Pi and familiarize yourself with how to cut and paste text between them. Then from the one window you can:

$ cd pipresents; more README.md

while you execute the commands you’ve cut and paste from that window into the other, e.g.,

$ sudo apt-get install python-imaging
etc.

What happens if you forget to install the unclutter package
Not much. It’s just that you will see a mouse pointer in the center of the screen which won’t go away, which is not desirable for black box operation.

Python image downsizing program
This is also known as downsampling. Amazingly, you really don’t find a simple example program like this when you do an Internet search, at least not amongst the first few hits. I needed a program to reduce the large images to the size of the display while preserving the aspect ratio. My display, a run-of-the-mill Acer v173, is 1280 x 1024 pixels. Pretty standard stuff, right? yet the Pi sees it as 1232 x 992 pixels! Whoever would have thought that possible? And with no possible option to change that (at least from the GUI). So just put in the appropriate values for your display. This program just handles one single image file. also note that if it’s a small picture, meaning smaller than the display, it will be blown up to full screen and hence will make a thumbnail image look pixelated. The match doesn’t distinguish small from large images but I fel that is fine for the most part. So without further chatting, here it is. I called it resize3.py:

import Image
import sys
# DrJ 2/2015
# somewhat inspired by http://www.riisen.dk/dop/pil.html
# image file should be provided as argument
# Designed for Acer v173 display which the Pi sees as a strange 1232 x 992 pixel display
# though it really is 1 more run-of-the-mill 1280 x 1024
 
imageFile = sys.argv[1]
im1 = Image.open(imageFile)
 
def imgResize(im):
# Our display as seen by the Pi is a strange 1232 x 992 pixels
    width = im.size[0]
    height = im.size[1]
 
# If the aspect ratio is wider than the display screen's aspect ratio,
# constrain the width to the display's full width
    if width/float(height) > 1232.0/992.0:
      widthn = 1232
      heightn = int(height*1232.0/width)
    else:
      heightn = 992
      widthn  = int(width*992.0/height)
 
    im5 = im.resize((widthn, heightn), Image.ANTIALIAS) # best down-sizing filter
 
    im5.save("resize/" + imageFile)
 
imgResize(im1)

As I am not proficient in python I designed the above program to minimize file handling. That I do in a shell script which was much easier for me to write. Together they can easily handle downsampling all the image files in a particular directory. I call this script reduce.sh:

#!/bin/sh
echo "Look for the downsampled images in a sub-directory called resize
echo "JPEGs GIFs and PNGs are looked at in the current directory
mkdir resize 2>/dev/null
ls -1 *jpg *jpeg *JPG *png *PNG *gif *GIF 2>/dev/null|while read file; do
  echo downsampling $file
# downsample the image file
  python ~/resize3.py "$file"
done

Stopping the slideshow
Sometimes you just need to stop the thing and that’s not so easy when you’ve got it in blackbox mode and running at startup.

If you’re lucky enough to have a keyboard attached to the Pi we found that

<Alt> F4

from the keyboard stops it.

No keyboard? We assigned Our Pi a static IP address and leave an ethernet cable attached to it. Then we put a PC on the same subnet and ssh to it, e.g., using putty or teraterm. Then we run this simple kill script, which I call kill.sh:

#!/bin/sh
pkill -f pipresents.py
pkill omxplayer

Digital photo frame projects morphs to museum-style kiosk display
At times I was tempted to throw out this pipresents software but we persisted. It has a different emphasis from a digital photo frame where you plug in a USB stick and don’t care about the order the pictures are presented to you. pipresents is oriented towards museums and hence is all about curated displays, where you’ve pored over the presentation order and selected your mix of videos and images. And in the end that better matched our requirements.

The manual is wanting for clarity
It’s nice that a PDF manual is included, but it’s a pain to read it to extract the small bits of information you actually need. Here’s what you mostly need to know. An unattended slideshow mixture of images and videos is what he calls a mediashow. Make your own profile to hold your mediashow:

$ cd pipresents; python pp_editor.py

This brings up a graphical editor. Then follow these menus:

Profile|New from template|Mediashow

Choose a short easy-to-type name such as drjmedia.

Click on media.json and then you can start adding images and movies. These are known as “tracks.”

Remove the example track.

Add your own images and movies.

Do a Profile|Validate

There is no Save! Just kill it.

And to run it full screen from your home directory:

$ python pipresents/pipresents -ftop -p drjmedia

Autostarting your mediashow
The instructions provided in the manual.pdf worked on my older Pi, but not on the B+ model Pis. So to repeat it here, modifying it so that it is more correct (the author doesn’t seem comfortable with Linux). Manual.pdf has:

$ mkdir -p ~/.config/lxsession/LXDE
$ cd !$; echo "python pipresents/pipresents.py -ftop -pdrjmedia" > autostart
$ chmod +x autostart

And as I say this worked on my model B Pi, but not my B+. The following discussion about autostarting programs is specific to operating systems which use the LXDE desktop environment such as Raspbian. On the B+ this fairly different approach worked to get the media show automatically starting upon boot:

$ cd /etc/xdg/autostart

Create a file pipresents.desktop with these lines:

[Desktop Entry]
Type=Application
Name=pipresents
Exec=python pipresents/pipresents.py -ftop -pdrjmedia
Terminal=true

But I recommend this approach which also works:

$ mkdir ~/.config/autostart

Place a pipresents.desktop file in this directory with the contents shown above.

More sophisticated approach for better black box operations
We find it convenient to run pp_editor in a virtual display created by vnc. Then we still don’t need to attach keyboard or mouse to the Pi. But the problem is that pipresents will also launch in the vnc session and really slow things down. This is a solution I worked out to have only one instance of pipresents run, even if others X sessions are launched on other displays. Note that this is a general solution and applies to any autostarted program.

The main idea is to test in a simple shell script if our display is the console (:0.0) or not.

I should interject I haven’t actually tested this but I think it’s going to work! Update: Yes, it did work!

Put startpipresents.sh in /home/pi with these contents:

#!/bin/bash
# DISPLAY environment variable is :0.0 for the console display
echo $DISPLAY|grep :0 > /dev/null 2>&1
if [ "$?" == "0" ]; then
#  matched. start pipresents in this xsession, but not any other one
  python pipresents/pipresents.py -ftop -pdrjmedia
fi

Then pipresents.desktop becomes this:

[Desktop Entry]
Type=Application
Name=pipresents
Exec=/home/pi/startpipresents.sh
Terminal=true

To install the vnc server:

$ sudo apt-get install tightvncserver

And to auto-launch it make a vnc.desktop file in ~/.config/autostart like this:

[Desktop Entry]
Type=Application
Name=vncserver
Exec=/home/pi/startvncserver.sh
Terminal=false

and put this in the file /home/pi/startvncserver.sh:

#!/bin/bash
# DISPLAY environment variable is :0.0 for the console display
echo $DISPLAY|grep :0 > /dev/null 2>&1
if [ "$?" == "0" ]; then
#  matched. start vncserver in this xsession, but not any other one
  vncserver
fi

You need to launch vncserver by hand once to establish the password.

And we may as well pre-launch the pp_editor because we’re likely to need that. So make a file in the home directory called startppeditor.sh with these contents:

#!/bin/bash
# DISPLAY environment variable is :1.0 for the vnc display
echo $DISPLAY|grep :1 > /dev/null 2>&1
if [ "$?" == "0" ]; then
#  matched. start ppeditor in this xsession, but not any other one
  python pipresents/pp_editor.py
fi

and in ~/.config/autostart a file called ppeditor.desktop with these contents:

[Desktop Entry]
Type=Application
Name=ppeditor
Exec=/home/pi/startppeditor.sh
Terminal=true

Similarly we can pre-launch an lxterminal because we’ll probably need one of those. Here’s an example startlxterminal.sh:

#!/bin/bash
# DISPLAY environment variable is :1.0 for the vnc display
echo $DISPLAY|grep :1 > /dev/null 2>&1
if [ "$?" == "0" ]; then
#  matched. start a large lxterminal in this xsession, but not any other one
  lxterminal --geometry=100x40
fi

and the autostart file:

[Desktop Entry]
Type=Application
Name=lxterminal
Exec=/home/pi/startlxterminal.sh
Terminal=true

A note about Powerpoint slides
With a Macbook we were able to read in a Powerpoint slideshow and export it to JPEG images, one image per slide. That was pretty convenient. We have done the same directly from Microsoft Powerpoint – it’s a save option.

A note about Mpeg4 videos
Some videos overwhelm these older Pis that we use. Maybe on the Pi 3 they’d be OK? A creative student would hand us his 2 minute movie in mpeg4 format. The Pi would never be able to display it. We learned you can reduce the resolution to get the Pi to display it. A student was doing this on his Macbook, but when he left i had to figure out a way.

The original mpeg4 video had resolution of 1920 x 1080. I wanted to have horizontal resolution of no more than 1232, but maybe even smaller, while preserving the aspect ratio (widescreen format).

I used good ole’ Microsoft Movie Maker. I don’t think it’s available any longer except from dodgy sites, but in the days of Windows 7 you could get it for free through Windows Live Update. Then, if you upgraded that Windows 7 PC to Windows 10, it allowed you to keep Movie Maker. That’s the only way I know of. Not that it’s a good program. It’s not. Very basic. But it does permit resizing a video stream to custom resolution, so I have to give it that. I tried various resolutions nd played them back. i finally settled on the smallest I tried: 800×450. In fact I couldn’t really tell the difference in video quality between all the samples. And of corse 800×450 made for the smallest file. So we took that one. Fortunately, pipresents blew it up to occupy the full screen width (1232 pixels) while preserving the aspect ratio. So it looks great and no further action was needed.

The sound of silence
You want the video sound to come out the stereo mini-jack because you’re not using an HDMI monitor? PiPresents tries to send audio out through HDMI by default so you won’t hear the sounds if you have a VGA monitor. But you can change that. If you want to do this in raw omxplayer the switch which sends the sound out through the mini-jack is:

omxplayer -o local

In pipresents this option is available in the pp_editor. It’s a property of the profile. So you edit the profile, look for omx-audio, and change its value in the drop-down box from hdmi to local. That’s it!

A word about DHCP
We use a PC to connect to the four Pis. They are connected to a hub and there is an Ethernet cable connected to the hub and ready to be connected to a PC with an Ethernet port. The Pis all have private IP addresses: 10.31.42.1, 10.31.42.2, 10.31.42.3 and 10.31.42.4. For convenience, we set up a DHCP server on Pi 1 so that when the PC connects, it gets assigned an IP address on that subnet. DHCP is a service that dynamically assigns IP addresses. Turns out this is dead easy. You simply install dnsmasq (sudo apt-get install dnsmasq) and make sure it is enabled. That’s it! More sophisticated setups require modification of the file /etc/dnsmasq.conf, but for our simple use case that is not even needed – it just picks reasonable values and assigns an appropriate IP to the laptop that allows it to communicate to any of the four Pis.

References and related
I worked on this project with a student. Building a Four Monitor Media Show using Raspberry Pis
Pipresents has its own wordpress site.

LXDE has its own official site.
Read about a first look at the custom-built 7″ Raspberry Pi touch display in this blog post.

An alternative slideshow program to pipresents is to leverage qiv. I put something together and demo it in this post, but with a twist: I pull all the photos from my own Google Drive, where I store 40,000+ pictures!

Categories
Admin Linux Security

Citrix problems with SHA2 certificates SSL error 61

Intro
Basically all certificates issued these days use the SHA2 signing algorithm whereas a year ago or for some CAs just a few months ago this was not the case and the SHA1 signing algorithm was being used. This change causes some compatibility problems.

The details
It can be a little hard to test a new certificate with Citrix Secure Gateway. If you try it and pray, you may well find that a majority of Citrix clients can connect your Secure Gateway but some cannot. They may even see SSL error 61.

So if you dutifully go to this Citrix support page, TID 101990, you read a very convincing description of the problem and why it happens. The only thing is, it is probably totally wrong for your case! Because in it they argue that your certificate is faulty and go back to your CA and get a good one! Ridiculous! I’ve dealt with lots of CAs and gotten lots of certificates. Never had a faulty one like that.

So what’s the real explanation? I think it is that their own Citrix client is out-of-date on the PC where it isn’t working and doesn’t support SHA2! This is still an unfolding story so that involves a little speculation. Upgrade the Citrix Receiver client and try again.

But of course you need to do your basic homework and make sure the basic stuff is in order. Use openssl to fetch your certificate and certificate chain and have a look at them to make sure you’ve really set it up right. A beginner’s mistake is to forget to include the intermediate CERT. Perhaps that could cause the SSL error 61 as well. And of course you need a certificate issued by a legitimate CA. A self-signed certificate will probably definitely give you an SSL error 61.

Given time I’ll show how to check if your certificate – or any other reference certificate you want to compare it to- uses SHA1 or SHA2.

To be updated if I get more conclusive information…

Conclusion
Citrix is giving out misleading or wrong advice about SSL error 61.

References and related articles
This site seems to confirm the widespread problem with many Citrix clients and SHA2 certificates.
http://www.p2vme.com/2014/02/sha2-certificates-and-citrix-receiver.html
This site talks about the dangers of SHA1 certificates and what Microsoft is doing about it.

Categories
Admin Linux

Cognos stopped working. But nothing changed!

Intro
Once again we got into this undesirable situation in which for no apparent reason Cognos logins simply stopped working. As administrator of the Cognos gateway piece and only that piece I was ready to swear up and down it cuoldn’t be my fault, but I took a closer look anyways, just in case. Here is what I found.

The details
We had been running Cognos v 10 without incident for over a year now when the word came from the application owner that logins stopped working in the morning. That is, through the gateway. If they tested through the application server directly that was still still working.

I knew I hadn’t changed anything so I assumed they did, or their LDAP back-end authentication server wasn’t working right.

Checking the apache web server logs showed nothing out of the ordinary. And my gateway had been running for a couple months straight (no reboot) when the problem occurred. I tested myself and saw for myself. You would get the initial pop-up log=in screen, then after submitting your authentication information it would just come back to you blank.

I managed to get a trace to the back-end dispatcher. Even that was pretty unspectacular. There was some HTTP communication back-and-forth in a couple of independent streams. The 2nd stream even contained something that looked promising:

CAMUsername=drjohn&CAMPazzword=NEwBCScGSIb3DQ...

And the server response to that was already to return the user to the login page as though already something had gone wrong.

The app owner said she observed the same problem on the test system, but ignore that comment because that isn’t tested or used much. That’s another environment I hadn’t touched in a long while – 16 months.

I confirmed I could log on to the development dispatcher but not through my gateway. What the heck?

So I decided to look at my own blog posts for inspiration. It seems the thing to do in times like this is to save the configuration. So I tried that on the test system – that’s what it’s for, after all. I breathed a sigh of relief when in fact the save went through – you just never know. But, yes, I got the green checks. And I’ll be darned if that didn’t fix it! So with that small victory, I saved the configuration on my two gateway servers and, yup, logins started working there as well!

I am very annoyed at IBM for making such a faulty product. My private speculation as to what happened is that when you save the configuration you generate cryptographic information, which means PKI which includes certificates which have expiration dates. I suppose the certificates being used to exchange information securely between gateway and dispatcher simply expired and the software inconsiderately produced no errors about the matter other than to stop working. Even when I launched the cogconfig program no errors were displayed initially.

IBM’s role
I strongly suggested to the application owner to open a case with IBM about this ridiculous behaviour. But since it’s IBM I’m not too confident it will go anywhere.

cogconfig.sh
The details of launching the configuration tool in linux is described in the references. But note that unlike that article, this time I did not need to delete any key files or any other files at all. Just save the configuration.

August 16th, 2016 update

Well almost exactly two years later I stepped on that same rake again! Nothing changed yet authentications stopped working. I even took a a packet trace to prove that the gateway was sending packets to the app server, which it was. The app server only reported this very misleading message: unable to authenticate because credentials are invalid. Worse, I was vaguely aware there could be a problem with the cryptographic keys, but I assumed such a problem would scramble all communication to the app server. yet that same app server log file showed the userid of the user correctly! So I was really misled by that.

When they logged on directly to the app server it was fine. See that date? That’s almost exactly two years after my original posting of this article. So I guess the keys were good for two years this last time, then they expired without warning and without any proper logging. Thanks IBM! The solution was exactly the same: run cogconfig.sh and save. That’s it… I obviously forgot my own advice and I did not regenerate the config from time to time.


For the future

I think to avoid a repetition of this problem I may save the configuration every six months or so. 16 months is a strange time in the world of certificates for an expiration to occur. I don’t get that. So maybe my explanation as to what happened is bogus, but it’s all we have for now.

Conclusion
Another mysterious Cognos error. This one we resolved a tad faster than usual because prior experience told us there was one possible action we could reasonably take to help the situation out. There were no panicked reboots of any servers, by the way. Did you have this problem? Welcome to enterprise software!

References
The details on where the configuration program, cogconfig.sh is to be found and run is described in this article.
If you forgot which is your dispatcher, you can grep the file cogstartup.xml for 9300, which is the port it runs on, to give you some hints.

Categories
Admin Linux

Things that went wrong during the HP SiteScope upgrade

Intro
These are my notes of all the stupid things I did during my attempt to upgrade to HP SiteScope version 11.24 in response to a security problem in earlier releases. When you don’t do these things very often you totally forget what you did last time!

The details
I managed to download the “patch,” SIS_00314, from the HP Passport site. That was relatively straightforward.

First mistake: winging it
After baking up my configuration I looked into the zip file and saw an rpm that seemed like the thing I needed: packages/HPSiS1124Core-11.24.241-Linux2.4.rpm. So I decided to just install it directly. That created deep directories like this one:

/opt/HP/SiteScope/installation/HPSiS1124/flvr/SiteScope

and it really looked like it wasn’t going to do anything. And when I tried that with the Java package all kinds of dependent libraries were not found. So I removed that package and went back to the manual.

So I identified the included deployment guide and skimmed through that for inspiration. It seems you should drive the installation through the file HPSiS1124_11.24_setup.bin. So I tried it. First I had to set up my X-Windows stuff:

# vncserver :2
# export DISPLAY=:2.0
(then connect to that display using VNC client)
# xhost +
and to test it (the following command should pop up a new window):
# gnome-terminal

Then finally:

# ./HPSiS1124_11.24_setup.bin

Preparing to install...
Extracting the JRE from the installer archive...
Unpacking the JRE...
Extracting the installation resources from the installer archive...
Configuring the installer for this system's environment...
 (i) Checking display ...
 (-) Display may not be properly configured
Please make sure the display is set properly...

I don’t know what went wrong, but I knew a GUI install was out of the question without a lot of digging. I also knew about silent or console installs. So I looked to that part of the deployment manual.

# ./HPSiS1124_11.24_setup.bin -i silent

Preparing to install...
Extracting the JRE from the installer archive...
Unpacking the JRE...
Extracting the installation resources from the installer archive...
Configuring the installer for this system's environment...
Preparing SILENT Mode Installation...
 
===============================================================================
                                       (created with InstallAnywhere by Zero G)
-------------------------------------------------------------------------------
 
=======================================================
 
Installer User Interface Mode Not Supported
 
Unable to load and to prepare the installer in console or silent mode.
 
=======================================================

I learned this happens if you run silent mode but have a DISPLAY environment variable set! Just unset DISPLAY and re-run.

But that didn’t work out for me either. In /tmp/HPOvInstaller/HPSiS1124_11.24 the file HPSiS1124_11.24_2014.08.11_15_39_HPOvInstallerLog.txt showed:


2014-08-11 15:39:40,065 INFO – Checking free disk space… [FAILED]

However since it was a silent installation it never complained! I only knew there was a problem when I restarted SiteScope and saw it was still showing me version 11.23.

I cleared some more space and the install finally completed successfully.

Conclusion
We ran into quite a few problems doing a simple minor HP SiteScope upgrade, most of our own making. But we persevered and are now running version 11.24.

References
Yearning for the old Freshwater SiteScope? You’ll want to read this blog posting and comments.
The Secunia advisory.

Categories
Admin Linux Network Technologies

The IT Detetecive Agency: the case of the unreliable FTP

Intro
So one of my power users complains that his FTPs to a particular site fail frequently, but not always. I rolled up my sleeves and set to work. The thing I do best is find the essence of a problem – what is the bare minimum sequence of events that reproduces it. I’m still getting my head around this one and I haven’t cracked the case yet, but I’ve learned about a few obscure packet generation tools.

The details
I may flesh this out later. The essence of the thing is that a packet trace (using tcpdump) shows that randomly no SYN-ACK packet is returned for our SYN packet to the FTP server on port 21. The FTP server resides on the Amazon cloud, but on the West Coast. We are on the East Coast. Not that that matters.

So I learned to reproduce the problem myself with the built-in ftp client. But I wanted even more control.

Packet generation tools
My trajectory went kind of like this:

ftp -> ping -> nmap -> hping3 -> mausezahn (-> scapy)

I had to compile mausezahn but I did manage to get it to work. I guess the developer has passed away. It doesn’t offer complete control over tcp packet generation, but nearly so.

I just discovered scapy. It appears to offer complete control over packet generation, including the tcp options such as mss, but the proof is in the pudding and I haven’t had time to check it out.

See the references fo links to further information about where to find these packages.

Preliminary findings
I began to see that with nmap and hping3 I was getting SYN-ACKs back consistently. What’s the difference between their SYN packets and ftp’s? They don’t use any options whereas my ftp client does.
And that is the essence of the problem. A tcp SYN packet which sets options like SACK, wscale and MSS is not being responded to around 30% of the time. No options set? SYN-ACKs come back 100% of the time. Pings are answered 99 – 100% of the time. mausezahn (mz) allows to set the window size. The window size is irrelevant.

Is it one particular tcp option that is the culprit, or just the fact of using any of them? Unfortunately that’s where you reach the limits of mz. mz only allows you to turn on or off all options. scapy promises to be more granular. So at least with mz by itself I can turn of/off the problem at will. That is getting to the essence of the problem.

Another wrinkle? Only certain source IPs have the problem! I have an identical system using a different ISP and it works all day long.

Conclusion
A lot of work and only modest progress to show for it. I need cooperation of the ftp administrator to do a simultaneous trace. Either the packet never gets to him, or his infrastructure discards it, or he responds but I never see the response. A two-sided trace will narrow down which of these three things is happening.

But I did learn that fine-control packet generation is a bit difficult to come by, which comes as a surprise in this day and age. You have to do some work to get full control over your packets.

I have nos stomach for writing my own C++ code to have total control.

It’s still an open case.

References
nmap.org talks about nmap. nmap is a pretty standard package available for major distributions. But it is not sufficiently configurable.
I’ve written about hping3 before, showing how to compile it.
I used this site for mausezahn source code and documentation.
scapy is well-documented here.

Categories
Admin Linux Network Technologies

Querying AD via LDAP – reference documentation

Intro
Suppose you managed to stuff the user’s ID into the description field of every computer object. Then independently the name of the computer object appears in a log such as a web server log and you want to know the user associated with that.

These examples show how to get at that description field from the computer object name.

The details

ldapsearch is a useful tool. I have two versions of it installed on Windows 7 and different Unix/Linux versions. The syntax is slightly different in all cases. Let’s assume the AD domain DRJOHNSAD is mapped to DNS domain drjohnsad.drjohns.net, and the user is drj. Then we have:

Linux
> ldapsearch -h drjohnsad.drjohns.net -b dc=drjohnsad,dc=drjohns,dc=net -D ‘drjohnsad\drj’ -W cn=computerName description

The -W switch prompts for the password. That is a nice switch, and not available in all versions of ldapsearch. If not, use -w password instead. drjohnsad\drj needs the single quote to prevent the “\” character from being treated as a special character by the shell. Windows doesn’t need that.

Windows 7 CMD Window

Oracle-provided ldapsearch

> ldapsearch -h drjohnsad.drjohns.net -b dc=drjohnsad,dc=drjohns,dc=net -D drjohnsad\drj -q cn=computerName description

So -q is used to prompt for a password instead of Linux’s -W.

Lotus Notes ldapsearch

> ldapsearch -h drjohnsad.drjohns.net -b dc=drjohnsad,dc=drjohns,dc=net -D drjohnsad\drj -w password cn=computerName description

You gotta put in the password on the command line.

Of course Windows also has applications which can be used for ldap queries in a GUI, but I don’t use them.

Conclusion
The syntax for a simple ldap query against an AD domain controller is shown.

Categories
Admin Internet Mail Linux

Sendmail: getting mailertable, smarttable and virtusertable to play nice together

Intro
Remember when I posted about some obscure but cool sendmail features? Well in that posting I mentioned having trouble with a catch-all mailertable entry co-existing with a smarttable. I recently got an incentive to work through that and I am sharing my results here. Plus use of virtusertable thrown in for good measure.

The setup
I have a Secure Mail Gateway (SMGW). Some of my users use it, most do not. So for the few who do I wish to divert my outgoing mail, if it sent by one of them, to the SMGW1. Everything else that’s outbound gets forwarded to an Internet-facing relay2. And for inbound mail I also want to divert their email over to it so it can do S/MIME or PGP decryption3. All other inbound emails should go to my native email system4.

So consider the outbound stream, requirement 1. Sender-based routing. That’s the normal thing in sendmail. That’s why you use smarttable, which Andrzej Filip has developed. See my references for more details on that. But my mailertable has a catch-all entry like:

.              relay:emgw.drj.com

I found if I got rid of this entry my smarttable entries began to work, but they stopped working if I put it back. I rolled up my sleeves and tried to understand ruleset 0. I became pretty convinced that mailertable runs before smarttable, and that if mailertable made a successful lookup of the recipie4nt domain that’s it. You’re done. Then I read a very brief comment in Andrzej’s writeup of smarttable. He bascially said mailertable runs first, and I guess form that brief comment you’re supposed to know that this setup I am trying will never work. And yet I have a colleague with a similar setup who says it does work, which got my competitive juices flowing – if the other guy can do it, then I know it can be done and I can do it as well! I knew I had to eliminate the catch-all entry, but still needed a catch-all feature. What to do? Fudge DNS to the point where all TLDs have a fake MX entry pointing at my external mail server?? Sounds too kludgy.

I was reading here and there in the Sendmail book and the cf/README file when SMARTHOST caught my eye. A smarthost can be defined to deliver all email to a specified relay. I always viewed it as an alternative to the mailertable, where you can specify much more specific delivery rules. But maybe they could co-exist? Mailertable for domain-specific delivery instructions, and smarthost for everything else? Yes, you can indeed do that! And with the catchall entry gone from mailertable does smarttable begin to work? Yes! It does! So our outbound stream is in good shape and all requirements are met. Each user of the SMGW is entered in the smarttable:

[email protected]    relay:smgw.drj.com
[email protected]    relay:smgw.drj.com
etc.

The mc configuration for smarthost looks like this:

dnl smarthost: to take care of the everything-else delivery case - DrJ 2/28/14
define(`SMART_HOST',`internet-facing-relay.drj.com')

Inbound
For inbound we can use virtusertable to rewrite the user domain to a fictional domain, then a mailertable entry which describes that this fictional domain should be routed over to the SMGW! Like this.

Virtusertable

and

Mailertable

drj.com   relay:native-mail-system-gw.drj.com
.smgw     relay:smgw.drj.com

Although this sounded good on paper, I found it was not enough by itself. In addition I needed to throw in a virtuser domain file which included the domain drj.com.

The mc configuration for virtusertable and virtuserdomain look like this.

dnl
dnl add virtusertable to do recipient re-writing to accomodate SMGW routing -DrJ 3/3/14
FEATURE(`virtusertable',`hash -o /etc/mail/virtusertable')dnl
dnl virtuser domain file location - DrJ 2/27/14
VIRTUSER_DOMAIN_FILE(`/etc/mail/virtuserdomains')dnl

virtuserdomains is a standard text file which contains drj.com.

Testing
It is helpful to know how to debug this stuff. Smarttable is probably the hardest. Here’s how to see the before and after effect that smarttable has:

$ sendmail -Csendmail-test.cf -bt<<END
> 3,0 [email protected]
> [email protected]
> 3,0 [email protected]
> END

The 3,0 lines show the result of running rulesets 3 and 0, I guess. The .Df…line defines the sender, so the last 3,0 line shows how delivery to the sender is now altered if the defined sender exists in the smarttable table.

Regular addresses can be tested with the same -bt switch if you want to see the gory details, or simply -bv:

$ sendmail -Csendmail-test.cf -bt< 3,0 [email protected]
> END

$ sendmail -Csendmail-test.cf -bv [email protected]

It’s assumed you put your sendmail config in sendmail-test.cf to not interfere with production.

Even if all the tests succeed, what I found is that smarthost did not take effect dynamically. I needed to re-start sendmail.

Conclusion
By digging into the innards of sendmail we learned enough to see how things should work together and found that it is indeed possible for smarttable, virtusertable and mailertable to peacefully co-exist, but only with a helping of smarthost and virtuserdomains!

References
I describe smarttable here.
Andrzej’s smarttable page is here.

Categories
Admin Linux Network Technologies Raspberry Pi

Using your Raspberry Pi as a router

Intro
Most Raspberry Pi router HowTos describe how to make the Pi act like an Access Point. That can be very useful, and kind of tricky. Here I’ve turned wireless and wired network roles around and show how to make it route traffic from a wired LAN over its WiFi connection.

I bought the EDIMAX nano USB adapter to play around with wireless on my Pi. It wasn’t long before the network lover in me realized, Hey I got an ethernet port on one end, a wireless on the other – that sounds like a potential router – let’s have some fun! So I hooked up my cantankerous Sony Blueray player to it. The thing has always been touchy about using wireless but it also has a wired LAN connection option – in a room where I don’t have a wired ethernet network available. Enter the routing Pi…

WiFi on the Pi
There are many ways to get the thing going. The important thing to note is that this EDIMAX is tested and is known to be compatible with the Raspberry Pi. I guess they even included the appropriate driver for it, because there is no need at any time to read the mini-CD that the EDIMAX comes with and try to pull of, or worse, compile, a Linux driver like I initially feared. I think you merely need to do one of these numbers:

$ sudo apt-get update

to ensure you have the latest of everything on your Pi and that includes the driver for this WiFi adapter.

Also note that the Pi, being a tiny computer, is perfectly complemented by this nano adapter. The thing is so tiny you barely have enough surface area to pull it out of the USB slot.

As I said there are probably many ways to get your WiFi going. Being a command line lover I present that way, and even then there are alternative setups to choose from, of which I present only one here.

If you do one of these numbers:

$ cd /etc/network; sudo nano interfaces

I would recommend to add a line towards the top:

auto wlan0

and at the bottom:

# following http://antael.blogspot.com/2013/01/wifi-is-live-and-kicking.html
# to see Wifi signal strength and available signals run iwlist wlan0 scan
#allow-hotplug wlan0
iface wlan0 inet static
wpa-ssid 
wpa-psk 
wpa-key_mgmt WPA-PSK
address 192.168.0.90
gateway 192.168.0.254
netmask 255.255.255.0
broadcast 192.168.0.255

substitute the name of your WiFi SSID and the password (WPA Pre-Shared Key) in place of <MYSSID> and <MYSSIDPASSWORD>.

I prefer static IPs to dynamic ones. That way I ssh to the IP and it is the same every time.

As I am setting up a router this is not my complete interfaces file, just the part I want to emphasize for now. The complete file is listed below.

A word on what we are setting out to do – our network architecture

OK. So I am going to put the Pi next to the Blueray Player. The Pi will connect to my TP-LINK WiFi network. The Pi’s ethernet interface will connect to the LAN port on the Blueray player using a standard ethernet patch cable (these days it is not necessary to use a crossover cable as there’s always a device that auto-senses how it is wired). I will create a new network for this interface on the Pi, and make sure I’ve assigned a valid IP on this network to the Blueray player. The network I choose is 10.31.42.0/24 – something completely different from anything I may already have in use at home.

The Blueray player can also be assigned a static IP address. I give it 10.31.42.13 and the Pi I assign 10.31.42.11. The gateway of the Blueray player is our routing Pi, so that makes the gateway IP 10.31.42.11.

But I have another requirement: I want to conveniently put the Pi back on my wired home network in case something goes wrong with wireless. so I want to keep its valid static IP (192.168.2.100) which it used to have when it was wired to my switch. The way to accomplish all this is to use virtual IPs on the eth0 interface. See the full /etc/network/interfaces file below which shows eht0 still assigned to IP 192.168.2.200, but as well a virtual interface eth0:0 assigned to 10.31.42.11.

Convenient way to test things
Initially I was focused on the architecture I’ve outline above and getting all my interfaces up and running. And I could run, for instance,

$ sudo service networking restart

to make my changes to the interfaces file be dynamically enabled, and I can do a

$ ifconfig -a

to show all my interfaces, their IPs and other information. And I especially like

$ iwlist wlan0 scan

to show the available WiFi networks and their signal strength. But how do I know that routing is working?? Here’s how. You know the ping command, right? Normally you do

$ ping 8.8.8.8

to show you can reach the Internet. Why? because that’s a valid IP address on the Internet that responds to PING – thank you Google – and it’s easy to remember!

But since our default route is out of our WiFi-connected interface, the IP it picks for the source of that PING is the IP assigned to wlan0, namely 192.168.0.90. There’s no Pi-routing involved in that so far. But now, you can choose a different source IP for your pings. So we pick our new virtual IP, 10.31.42.11 like this:

$ ping -I 10.31.42.11 8.8.8.8

but instead of a nice result like

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_req=1 ttl=44 time=172 ms
64 bytes from 8.8.8.8: icmp_req=2 ttl=44 time=92.7 ms
64 bytes from 8.8.8.8: icmp_req=3 ttl=44 time=69.6 ms
64 bytes from 8.8.8.8: icmp_req=4 ttl=44 time=70.6 ms
64 bytes from 8.8.8.8: icmp_req=5 ttl=44 time=70.3 ms
64 bytes from 8.8.8.8: icmp_req=6 ttl=44 time=70.5 ms
^C
--- 8.8.8.8 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5006ms
rtt min/avg/max/mdev = 69.676/91.121/172.740/37.409 ms

we don’t get much encouragement:

PING 8.8.8.8 (8.8.8.8) from 10.31.42.11 : 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms

The ^C above means <CTRL-C> was typed in at the keyboard.

So we’re seeing 100% packet loss and we can safely conclude no routing is occurring. By the way, as a reality and typo check on this approach, try to specify as source your wlan0 IP, as in

$ ping -I 192.168.0.90 8.8.8.8

This should work.

Turn on routing
I feared it would be mess to get routing turned on, but it turns out to be pretty tidy, considering. Probably your Pi is already enabled for routing. Mine was. Just

$ cat /proc/sys/net/ipv4/ip_forward

and make sure the value is 1. That means it is set up to do routing. If you’re not so lucky and you have 0, enable routing:

$ sudo sysctl -w net.ipv4.ip_forward=1

But of course there’s more we have to do or else our source ping would have worked!

We need to turn on network address translation. This is the part I was worried about, never having done it before on Linux, but only with expensive products like commercial firewalls. But it’s really not bad at all.

I created a file iptables-NAT in the home directory of the pi user with these contents:

#!/bin/sh
# This is a one-time script - DrJ
# 2/2014
# explained nicely in http://www.karlrupp.net/en/computer/nat_tutorial
# and seems to even work!
 
# flush old iptables stuff. Need to specify nat table specifically
iptables -t nat -F
iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
 
# make it permanent
#/usr/sbin/service iptables save
 
# list it
iptables -t nat -L
 
# persist it:
 
iptables-save &gt; /etc/iptables.conf
 
# note that in /etc/network/interfaces we added this line to read in
# iptables.conf upon reboot:
# pre-up iptables-restore &lt; /etc/iptables.conf

and ran it:

$ cd; sudo ./iptables-NAT

It outputs this:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
 
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
 
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
 
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  anywhere             anywhere

Now run our source ping:

$ ping -I 10.31.42.11 8.8.8.8

PING 8.8.8.8 (8.8.8.8) from 10.31.42.11 : 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_req=1 ttl=44 time=76.3 ms
64 bytes from 8.8.8.8: icmp_req=2 ttl=44 time=71.4 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 71.471/73.922/76.373/2.451 ms

Voila! It worked. We now are NATing our source address of 10.31.42.11 to our wlan IP of 192.168.0.90 before sending the packet along to the next router. But as skeptics, we’d like to see some more empirical proof that all that is really happening. You can. Here’s how using tcpdump.

The thing is that tcpdump doesn’t have full access to pre-processing, it only has “post processing” access. So we need to hook up the Pi to a device we are going to route traffic for, in my case the Sony Blueray player. I don’t have much control, but I can do a network diagnostics which I know tries to reach the configured DNS server (because it complained that it could not reach the DNS server 8.8.8.8 during one of my early tests. So tee up tcpdump to monitor all interfaces’ traffic to 8.8.8.8 like this:

$ sudo tcpdump -n -i any host 8.8.8.8

Then run ping (or a DNS query) to 8.8.8.8 on that device, and voila, this is the result:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
20:22:09.041922 IP 10.31.42.13.54693 &gt; 8.8.8.8.53: 5+ A? m.root-servers.net. (36)
20:22:09.042144 IP 192.168.0.90.54693 &gt; 8.8.8.8.53: 5+ A? m.root-servers.net. (36)
20:22:09.163618 IP 8.8.8.8.53 &gt; 192.168.0.90.54693: 5 1/0/0 A 202.12.27.33 (52)
20:22:09.163777 IP 8.8.8.8.53 &gt; 10.31.42.13.54693: 5 1/0/0 A 202.12.27.33 (52)

The -i any switch told it to listen on all interfaces. Unfortunately the output doesn’t show which interface, but you can easily deduce it. the traffic comes in from my Blueray player at 10.31.42.13 on eth0, and then leaves the Pi on wlan0 having had its source IP translated to 192.168.0.90. And the reverse happens to the response from 8.8.8.8. In fact I see from this tcpdump output that the Blueray player does not do a PING, it does an actual DNS query to 8.8.8.8. Makes sense since it is a DNS server.

Be persistent
But a reboot will undo this nice NATing we have achieved. Try it. So we have to find a way to make our iptables entry persist across reboots. I’m sure there are many ways to do this. I liked this one. You add this line to the end of your interfaces file:

pre-up iptables-restore &lt; /etc/iptables.conf

And of course we already anticipated this manner of proceeding in our iptables-NAT file when we included this line at the bottom:

iptables-save &gt; /etc/iptables.conf

Now reboot again, and try that source ping as the first command you issue. Now it should work.

Ready for the real test
Now I can hook up my Blueray player with some confidence that at least the networking should be working. Assign its IP (10.31.42.13 as mentioned above), its gateway (the Pi’s eth0:0 virtual IP, namely 10.31.42.11 in our example), a DNS server (8.8.8.8, of course!) and see if:

– we can ping 10.31.42.13 from the Pi
– the Blueray player’s network tester shows OK (there aren’t many fine debugging utilities, just this single “network test”)

If the Blueray player can’t reach the configured DNS server then it its test fails.

Why this big long explanation for something that no one else in the world wants to do?
All this sounds like a very, very specific application that doesn’t apply to anyone else. But once you understand some of the core networking principles involved that I’ve touched on here, you will begin to realize that this can be very easily generalized to something much more applicable and powerful: a full-blown replacement for a standard wireless router such as my TP-LINK nano router. After all, we’ve set up almost all the utilities and facilities that you get from a standard router (NATing and routing), perhaps with one glaring exception: a DHCP service. I personally didn’t want one, but nothing prevents you from setting that up on the wired side. I give some suggestions below on how to do that in the next section.

A few words on a DHCP service
I did get that running on the Pi as well, though for an entirely different purpose. I used dnsmasq:

$ sudo apt-get install dnsmasq

and I think if you edit /etc/dnsmasq.conf and put these lines at the bottom:

interface=eth0
dhcp-range=10.31.42.14,10.31.42.254,10h
dhcp-option=3,10.31.42.11

I think you will pretty much have a working DHCP service as well and could use that instead of assigning static IPs.

What I fear about the DHCP service – and I think I have seen this – is that if I put the Pi back to the wired network, its DHCP server competes with the normal router’s DHCP service, and I start to lose connectivity to my devices! So be careful. If devices like your PC start to pick up 10.31.42.x addresses on a network where 192.168.2.x is expected, there could be some connectivity troubles ahead until you disconnect the Pi!

Finally, the full /etc/network/interfaces file

auto lo
auto eth0
auto eth0:0
auto wlan0
 
iface lo inet loopback
# DrJ change: make IP static
# somewhat inspired by http://www.techiecorner.com/486/how-to-setup-static-ip-in-debian/ - DrJ 1/8/13
#iface eth0 inet dhcp
iface eth0 inet static
address 192.168.2.100
#gateway  192.168.2.1
netmask 255.255.255.0
network 192.168.2.0
broadcast 192.168.2.255
 
# for network 3142
iface eth0:0 inet static
address 10.31.42.11
netmask 255.255.255.0
network 10.31.42.0
broadcast 10.31.42.255
 
# following http://antael.blogspot.com/2013/01/wifi-is-live-and-kicking.html
# to see Wifi signal strength and available signals run iwlist wlan0 scan
#allow-hotplug wlan0
iface wlan0 inet static
wpa-ssid 
wpa-psk 
wpa-key_mgmt WPA-PSK
address 192.168.0.90
gateway 192.168.0.254
netmask 255.255.255.0
broadcast 192.168.0.255
# wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf
#iface default inet dhcp
 
# to read in iptable configuration. See ~pi/iptables-NAT
# added - DrJ 3/2/14
pre-up iptables-restore &lt; /etc/iptables.conf

Conclusion
Yes, we got our Pi to act as a router, and it wasn’t too bad. We demonstrated it for this specific application, but also showed how the simple addition of a DHCP service would make this a very general solution.
Did it help with the problem at hand – getting smoother streaming on the Sony Blueray player? Well, actually, yes, it did seem to help.

References and related

An idea for temporarily replacing a busted home router with a Raspberry Pi router which uses your hotspot is described in this article.

Getting started on a Pi without a dedicated console is described here.
A look at playing around with real-time video using the Pi’s camera is described here.
A more esoteric project, using the Pi to monitor your home’s Internet/power connection, is presented here.
Turning your Raspberry Pi into a transparent networking bridge is described here.
Interested to run some real networking protocols on your Pi like RIP, OSPF or BGP? I suggest to look into Quagga. Quagga is a networking suite. In full disclosure I haven’t had time or motivation to experiment with it myself (yet).