Categories
Admin Ajax Image Manipulation Web Site Technologies

How to create a Progressive Scrolling Web Gallery

Intro
I know, I know, there are thousands of ways to display your pictures on the web. I did a 60 second search and settled on one approach that looked interesting to me. Then I quickly ran into some limits and made some improvements. That’s why there are now thousands plus one more as of today! The app I improved upon is good for previewing pictures in a directory where there are lots of nice pictures. It makes the downloading more pleasant and shows large-ish thumbnail images that can be enjoyed in their own right while you wait for more images to download.

Thankyou Alexandru Pitea
So I just downloaded the stuff from his fine tutorial, How to Create an Infinite Scrolling Web Gallery. I unpacked his the downloaded zip file: worked first time. That’s a good sign, right? That doesn’t always happen. Then I started to make changes and ruined it.

As previously documented I use Goodsync to sync all my home pictures to my server. So all pictures are present in various folders. But they’re big. I needed thumbnails for this gallery app. I wrote a very crude thumbnail generator. I basically have to edit it each time I work on a different directory. One day I’ll fix it up. I call it createthumbs.php:

<?php
function createThumbs( $pathToImages, $pathToThumbs, $thumbWidth )
{
  // open the directory
  $dir = opendir( $pathToImages );
 
  // loop through it, looking for any/all JPG files:
  while (false !== ($fname = readdir( $dir ))) {
    // parse path for the extension
    $info = pathinfo($pathToImages . $fname);
    // continue only if this is a JPEG image
    if ( strtolower($info['extension']) == 'jpg' )
    {
      echo "Creating thumbnail for {$fname} <br />";
 
      // load image and get image size
      $img = imagecreatefromjpeg( "{$pathToImages}{$fname}" );
      $width = imagesx( $img );
      $height = imagesy( $img );
 
      // calculate thumbnail size
      $new_width = $thumbWidth;
      $new_height = floor( $height * ( $thumbWidth / $width ) );
 
      // create a new temporary image
      $tmp_img = imagecreatetruecolor( $new_width, $new_height );
 
      // copy and resize old image into new image
      imagecopyresized( $tmp_img, $img, 0, 0, 0, 0, $new_width, $new_height, $width, $height );
 
      // save thumbnail into a file
      imagejpeg( $tmp_img, "{$pathToThumbs}{$fname}" );
    }
  }
  // close the directory
  closedir( $dir );
}
// call createThumb function and pass to it as parameters the path
// to the directory that contains images, the path to the directory
// in which thumbnails will be placed and the thumbnail's width.
// We are assuming that the path will be a relative path working
// both in the filesystem, and through the web for links
createThumbs("img/2012_05/","thumb/",200);
?>

Notice these are pretty big thumbnails – 200 pixels. That’s how the gallery program works best, and I think it is a good size for how you will want to browse your pictures.

Then I moved the original img directory to img.orig and made a symbolic link to one of my pictures’s folders (which I had run through the thumbnail generator).

img -> /homepic/pictures_chronological/2012_05/

It worked. But there were a couple annoying things. First, the picture order seemed nearly random. Apparently the order reflected the timestamp of the file, but not a sort by name order. I found it was simple to sort them by name, which produced a nice sensible order, by adding:

...
// sensible sort
$sortbool = sort($files,SORT_STRING);
...

to getImages.php.

The other annoying thing was the infinite scroll. Not sure what the attrtaction was to that. Many comments on his post asked how to turn it off. Turns out that was easy:

// prevent annoying infinite scroll
//$response = $response.$files[$i%count($files)].’;’;
$response = $response.$files[$i].’;’;

in the same file.

One astute user noticed the lack of input validation in the argument to GET, which should always be a non-negative integer. So I incorporated his suggestion for argument validation as well.

The full getImages.php file is here:

<?php
// input argument validation - only numbers permitted
function filter($data) {
if(is_numeric($data)) {
  return $data;
}
  else { header("Location: index.html"); }
}
 
        $dir = "thumb";
        if(is_dir($dir)){
                if($dd = opendir($dir)){
                        while (($f = readdir($dd)) !== false)
                                if($f != "." && $f != "..")
                                        $files[] = $f;
                        closedir($dd);
                }
// sensible sort
$sortbool = sort($files,SORT_STRING);
 
 
        $n = filter($_GET["n"]);
        $response = "";
                for($i = $n; $i<$n+12; $i++){
// prevent annoying infinite scroll
                        //$response = $response.$files[$i%count($files)].';';
                        $res = $files[$i];
                        if  (isset($res)) $response = $response.$res.';';
                }
                echo $response;
        }
?>

I’ve only done a couple tests a couple folders but in those tests they both showed all the pictures and then stopped scrolling, as you naturally would want. So that’s why what I have produced is a progressive scroll, not an infinite scroll the useful progressive scrolling part of the original code was preserved.

I think he even used bigger thumbnails than 200 pixels. For these smaller ones it makes more sense to grab pictures 12 at-a-time. So I made a few changes in index.html to take care of that.

Alexandru also had his first nine images hard-coded into his index.html. Again, I don’t see the point in that – makes it a lot harder to generalize. So I chucked that and appropriately modified some offsets, etc, without any terrible side-effects.

Putting it all together that code now looks like this:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, user-scalable=no initial-scale=1.0, minimum-scale=1.0" />
<title>Web Gallery | Progressive Sroll</title>
<link rel="stylesheet" href="style.css" />
</head>
 
<body onload="setInterval('scroll();', 250);">
<div id="header">Web Gallery | Progressive Scroll</div>
<div id="container">
</div>
</body>
</html>
<script>
//var contentHeight = 800;
var contentHeight = document.getElementById('container').offsetHeight;
var pageHeight = document.documentElement.clientHeight;
var scrollPosition;
var n = 0;
var xmlhttp;
 
function putImages(){
 
        if (xmlhttp.readyState==4)
          {
                  if(xmlhttp.responseText){
                         var resp = xmlhttp.responseText.replace("\r\n", "");
                         var files = resp.split(";");
                          var j = 0;
                          for(i=0; i<files.length; i++){
                                  if(files[i] != ""){
                                         document.getElementById("container").innerHTML += '<a href="img/'+files[i]+'"><img
 src="thumb/'+files[i]+'" /></a>';
                                         j++;
 
                                         if(j == 3 || j == 6 || j == 9)
                                                  document.getElementById("container").innerHTML += '<br />';
                                          else if(j == 12){
                                                  document.getElementById("container").innerHTML += '<p>'+(n-1)+" Images Di
splayed | <a href='#header'>top</a></p><br /><hr />";
                                                  j = 0;
                                          }
                                  }
                          }
                          if (i < 12) document.getElementById("container").innerHTML += '<p>'+(n-13+i)+" Images Displayed |
 <a href='#header'>top</a></p><br />";
                  }
          }
}
 
 
function scroll(){
 
        if(navigator.appName == "Microsoft Internet Explorer")
                scrollPosition = document.documentElement.scrollTop;
        else
                scrollPosition = window.pageYOffset;
 
        if((contentHeight - pageHeight - scrollPosition) < 200){
 
                if(window.XMLHttpRequest)
                        xmlhttp = new XMLHttpRequest();
                else
                        if(window.ActiveXObject)
                                xmlhttp = new ActiveXObject("Microsoft.XMLHTTP");
                        else
                                alert ("Bummer! Your browser does not support XMLHTTP!");
 
                var url="getImages.php?n="+n;
 
                xmlhttp.open("GET",url,true);
                xmlhttp.send();
 
// 12 pictures at a time...
                n += 12;
                xmlhttp.onreadystatechange=putImages;
                contentHeight = document.getElementById('container').offsetHeight;
                //contentHeight += 800;
        }
}
 
</script>

Notice I also played around with the scrolling function because that gave me difficulty. I set the condition contentHeight – pageHeight – scrollPosition to be less than 700, a requirement that is easier to meet, since in my tests I was often getting no scrolling whatsoever.

That’s it!

So to use my improvements you could download the source files from Alexandru’s site, then overwrite getImages.php and index.html from a cut-and-paste from this page.

To do list…
Naturally the first person to try it tried from an Android Smartphone using the Opera browser and it only showed him the first 12 pictures and didn’t do any scrolling. I developed for IE/FF on PC. I’ve just now tried Opera on PC and that worked fine. I’ll have to understand what is happening on Smartphones. So…I learned there is webkit for Smartphone compatibility. I added a meta tag concerning viewport (which I’ve already included in the html source file above). Now the pictures are a little large on my Android browser, and the progressive scrolling takes a nudge to get going, but it basically does work, which is an improvement. But still not on Opera mini! And not that well on Blackberry…

I’d also like to add a folder-browser plug-in.

Conclusion
Pages load fast initially in a progressive scroll approach. So this could be a useful program as a way to display your pictures on your own web site. We fixed up some of the undesirable behaviour of Alexandru’s original version.

Categories
Admin

Mysql Exploit: v. 5.1.6 on CentOS 6 does not appear vulnerable

Intro
As this story makes crystal clear, the test for the mysql password bug is ridiculously simple to run for yourself:

$ for i in `seq 1 1000`; do mysql -u root –password=bad -h 127.0.0.1 2>/dev/null; done

More on that
I am at version 5.1.61:

$ mysql –version

I fully expected to get a mysql> prompt from the above exploit code but I did not.

The Amazon cloud has some decent protections in place.

For instance I tried

$ mysql -u root –password=mysecretpassword -h 127.0.0.1 2>/dev/null

and of course I got in. But modified slightly, to

$ mysql -u root –password=mysecretpassword -h drjohnstechtalk.com 2>/dev/null

and it’s a no go. It just hangs. I can’t believe I never did this earlier, but I wanted to see the routing for my own elastic IP:

$ traceroute drjohnstechtalk.com

traceroute to drjohnstechtalk.com (50.17.188.196), 30 hops max, 60 byte packets
 1  ip-10-10-216-2.ec2.internal (10.10.216.2)  0.343 ms  0.506 ms  0.504 ms
 2  ip-10-1-54-93.ec2.internal (10.1.54.93)  0.571 ms ip-10-1-42-93.ec2.internal (10.1.42.93)  0.565 ms ip-10-1-52-93.ec2.internal (10.1.52.93)  0.366 ms
 3  ip-10-1-39-14.ec2.internal (10.1.39.14)  0.457 ms ip-10-1-41-14.ec2.internal (10.1.41.14)  0.515 ms ip-10-1-37-14.ec2.internal (10.1.37.14)  0.605 ms
 4  216.182.224.84 (216.182.224.84)  0.662 ms 216.182.224.86 (216.182.224.86)  0.606 ms  0.608 ms
 5  216.182.232.53 (216.182.232.53)  0.837 ms 216.182.224.89 (216.182.224.89)  0.924 ms 216.182.232.53 (216.182.232.53)  1.030 ms
 6  ip-10-1-41-13.ec2.internal (10.1.41.13)  0.869 ms ip-10-1-39-13.ec2.internal (10.1.39.13)  1.082 ms ip-10-1-43-13.ec2.internal (10.1.43.13)  1.154 ms
 7  ip-10-1-36-94.ec2.internal (10.1.36.94)  1.481 ms ip-10-1-54-94.ec2.internal (10.1.54.94)  1.351 ms ip-10-1-42-94.ec2.internal (10.1.42.94)  1.173 ms
 8  * * *
 9  * * *
10  * * *
...

So there’s quite a few hops before I hit my own IP! That’s plenty of hops in which to insert a firewall, which I suppose they do, to enforce my personal security policy.

My eth0 IP is 10.10.219.96. Using that:

$ mysql -u root –password=mysecretpassword -h 10.10.21.96

I get:

ERROR 1130 (HY000): Host 'ip-10-10-219-96.ec2.internal' is not allowed to connect to this MySQL server

even though my my.cnf file does not have this apparent restriction and the mysql daemon is listening on all interfaces:

$ netstat -an|grep LISTEN

...
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN

Conclusion
I don’t recall taking special steps to secure my msql installation though it’s not out of the question. So I conclude that inspite of the articles that cite my version as being vulnerable, it is not, at least under CentOS 6, and even if it were, it would be especially hard to exploit for an Amazon cloud server.

Categories
Admin Linux Network Technologies Raspberry Pi

Compiling hping on CentOS

Intro
hping was recommend to me as a tool to stage a mock DOS test against one of our servers. I found that I did not have it installed on my CentOS 6 instance and could not find it with a yum search. I’m sure there is an rpm for it somewhere, but I figured it would be just as easy to compile it myself as to find the rpm. I was wrong. It probably was a _little_ harder to compile it, but I learned some things in doing so. So I’ll share my experience. It wasn’t too bad. I have nothing original to add here to what you find elsewhere, except that I didn’t find anywhere else with all these problems documented in one place. So I’ve produced this blog post as a convenient reference.

I’ve also faced this same situation on SLES – can’t find a package for hping anywhere – and found the same recipe below works to compile hping3.

The Details
I downloaded the source, hping3-20051105.tar.gz, from hping.org. Try a ./configure and…

error can not find the byte order for this architecture, fix bytesex.h

After a few quick searches I began to wonder what the byte order is in the Amazon cloud. Inspired I wrote this C program to find out and remove all doubt:

/* returns true if system is big_endian. See http://unixpapa.com/incnote/byteorder.html - DrJ */
#include<stdio.h>
 
main()
{
    printf("Hello World");
    int ans = am_big_endian();
    printf("am_big_endian value: %d",ans);
 
}
 
int am_big_endian()
  {
     long one= 1;
     return !(*((char *)(&one)));
  }

This program makes me realize a) how much I dislike C, and b) how I will never be a C expert no matter how much I dabble.

The program returns 0 so the Amazon cloud has little endian byte order as we could have guessed. All Intel i386 family chips are little endian it seems. Back to bytesex.h. I edited it so that it has:

#define BYTE_ORDER_LITTLE_ENDIAN
/* # error can not find the byte order for this architecture, fix bytesex.h */

Now I can run make. Next error:

pcap.h No such file or directory.

I installed libpcap-devel with yum to provide that header file:

$ yum install libpcap-devel

Next error:

net/bpf.h no such file or directory

For this I did:

$ ln -s /usr/include/pcap-bpf.h /usr/include/net/bpf.h

TCL
Next error:

/usr/bin/ld: cannot find -ltcl

I decided that I wouldn’t need TCL anyways to run in simple command-line fashion, so I excised it:

./configure --no-tcl

Then, finally, it compiled OK with some warnings.

hping3 for Raspberry Pi
On the Raspberry Pi it was simple to install hping3:

$ sudo apt-get install hping3

That’s it!

Raspberry Pi’s are pretty slow to generate serious traffic, but if you have a bunch I suppose they could amount to something in total.

Conclusion
Now I’m ready to go to use hping3 for a SYN_FLOOD simulated attack or whatever else we want to test.

Categories
Admin Internet Mail Linux Perl

The IT Detective Agency: last letter of attachment name is missing!

Intro
Today we bring you an IT whodunit thriller. A user using Lotus Notes informs his local IT that a process that emails SQL reports to him and a few others has suddenly stopped working correctly. The reports either contain an HTML attachment where the attachment type has been chopped to “ht” instead of “htm,” or an MHTML attachment type which has also been chopped, down to “mh” instead of “mht.” They get emailed from the reporting server to a sendmail mail relay. Now the convenient ability to double-click on the attachment and launch it stopped working as a result of these chopped filenames. What’s going on? Fix it!

Let’s Reproduce the Problem
Fortunately this one was easier than most to reproduce. But first a digression. Let’s have some fun and challenge ourselves with it before we deep dive. What do you think the culprit is? What’s your hypothesis? Drawing on my many years of experience running enterprise-class sendmail servers, and never before having seen this problem despite the hundreds of millions of delivered emails, my best instincts told me to look elsewhere.

The origin server, let’s call it aspen, sends few messages, so I had the luxury to turn on tracing on my sendmail server with a filter limiting the traffic to its IP:

$ tcpdump -i eth0 -s 1540 -w /tmp/aspen.cap host aspen

Using wireshark to analyze asp.cap and following the tcp stream I see this:

...
Content-Type: multipart/mixed;
		 boundary="CSmtpMsgPart123X456_000_C800C42D"
 
This is a multipart message in MIME format
 
--CSmtpMsgPart123X456_000_C800C42D
Content-Type: text/plain;
		 charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
 
SQLplus automated report
--CSmtpMsgPart123X456_000_C800C42D
Content-Type: application/octet-stream;
		 name="tower status_2012_06_04--09.25.00.htm"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
		 filename="tower status_2012_06_04--09.25.00.htm
 
<html><head></head><body><h1>Content goes here...</h1></body>
</html>
 
--CSmtpMsgPart123X456_000_C800C42D--

Result of trace of original email as received by sendmail

But the source as viewed from within Lotus Notes is:

...
Content-Type: multipart/mixed;
		 boundary="CSmtpMsgPart123X456_000_C800C42D"
 
 
--CSmtpMsgPart123X456_000_C800C42D
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
		 charset="iso-8859-1"
 
SQLplus automated report
--CSmtpMsgPart123X456_000_C800C42D
Content-Type: application/octet-stream;
		 name="tower status_2012_06_04--09.25.00.htm"
Content-Disposition: attachment;
		 filename="tower status_2012_06_04--09.25.00.ht"
Content-Transfer-Encoding: base64
 
PGh0bWw+PGhlYWQ+PC9oZWFkPjxib2R5PjxoMT5Db250ZW50IGdvZXMgaGVyZS4uLjwvaDE+PC9i
b2R5Pg0KPC9odG1sPg==
 
--CSmtpMsgPart123X456_000_C800C42D--

Same email after being trasferred to Lotus Notes

I was in shock.

I fully expected the message source to go through unaltered all the way into Lotus Notes, but it didn’t. The trace taken before sendmail’s actions was not an exact match to the source of the message I received. So either sendmail or Lotus Notes (or both) were altering the source in significant ways.

At the same time, we got a big clue as to what is behind the missing letter in the file extension. To highlight it, compare this line from the trace:

filename=”tower status_2012_06_04–09.25.00.htm

to that same line as it appears in the Lotus Notes source:

filename=”tower status_2012_06_04–09.25.00.ht

So there is no final close quote (“) in the filename attribute as it comes from the aspen server! That can’t be good.

But it used to work. What do we make of that fact??

I had to dig farther. I was suddenly reminded of the final episode of House where it is apparent that the solving the puzzle of symptoms is the highest aspiration for Doctor House. Maybe I am similarly motivated? Because I was definitely willing to throw the full weight of my resources behind this mystery. At least for the half-day I had to spare on this.

First step was to reproduce the problem myself. For sending an email you would normally use sendmail or mailx or such, but I didn’t trust any of those programs – afraid they would mess with my headers in secret, undocumented ways.

So I wrote my own mail sending program using Perl/Expect. Now I’m not advocating this as a best practice. It’s just that for me, given my skillset and perceived difficulty in finding a proper program to do what I wanted (which I’m sure is out there), this was the path of least resistance, the best and most efficient use of my time. You see, I already had the core of the program written for another purpose, so I knew it wouldn’t be too difficult to finish for this purpose. And I admit I’m not the best at Expect and I’m not the best at Perl. I just know enough to get things done and pretty quickly at that.

OK. Enough apologies. Here’s that code:

#!/usr/bin/perl
# drjohnstechtalk.com - 6/2012
# Send mail by explicit use of the protocol
$DEBUG = 1;
use Expect;
use Getopt::Std;
getopts('m:r:s:');
$recip = $opt_r;
$sender = $opt_s;
$hostname = $ENV{HOSTNAME};
chop($hostname);
print "hostname,mailhost,sender,recip: $hostname,$opt_m,$sender,$recip\n" if $DEBUG;
$telnet = "telnet";
@hosts = ($opt_m);
$logf = "/var/tmp/smtpresults.log";
 
$timeout = 15;
 
$data = qq(Subject: test of strange MIME error
X-myHeader: my-value
From: $sender
To: $recip
Subject: SQLplus Report - tower status
Date: Mon, 4 Jun 2012 9:25:10 --0400
Importance: Normal
X-Mailer: ATL CSmtp Class
X-MSMail-Priority: Normal
X-Priority: 3 (Normal)
MIME-Version: 1.0
Content-Type: multipart/mixed;
        boundary="CSmtpMsgPart123X456_000_C800C42D"
 
This is a multipart message in MIME format
 
--CSmtpMsgPart123X456_000_C800C42D
Content-Type: text/plain;
        charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
 
SQLplus automated report
--CSmtpMsgPart123X456_000_C800C42D
Content-Type: application/octet-stream;
        name="tower status_2012_06_04--09.25.00.htm"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
        filename="tower status_2012_06_04--09.25.00.htm
 
<html><head></head><body><h1>Content goes here...</h1></body>
</html>
--CSmtpMsgPart123X456_000_C800C42D--
 
.
);
sub myInit {
# This structure is ugly (p.148 in the book) but it's clear how to fill it
@steps = (
        { Expect => "220 ",
          Command => "helo $hostname"},
# Envelope sender
        { Expect => "250 ",
          Command => "mail from: $sender"},
# Envelope recipient
        { Expect => "250 ",
          Command => "rcpt to: $recip"},
# data command
        { Expect => "250 ",
          Command => "data"},
# start mail message
        { Expect => "354 Enter ",
          Command => $data},
# end session nicely
        { Expect => "250 Message accepted ",
          Command => "quit"},
);
}       # end sub myInit
#
# Main program
open(LOGF,">$logf") || die "Cannot open log file!!\n";
foreach $host (@hosts) {
  login($host);
}
 
# create an Expect object by spawning another process
sub login {
($host) = @_;
myInit();
#@params = ($host," 25");
$init_command = "$telnet $host 25";
#$Expect::Debug = 3;
my $exp = Expect->spawn("$init_command")
         or die "Cannot spawn $command: $!\n";
#
# Now run all the other commands
foreach $step (@steps) {
  $i++;
  $expstr = %{$step}->{Expect};
  $cmd = %{$step}->{Command};
#  print "expstr,cmd: $expstr, $cmd\n";
# Logging
#$exp->debug(2);
#$exp->exp_internal(1);
  $exp->log_stdout(0);  # disable stdout for each command
  $exp->log_file($logf);
  @match_patterns = ($expstr);
  ($matched_pattern_position, $error, $successfully_matching_string, $before_match, $after_match) = $exp->expect($timeout,
@match_patterns);
  unless ($matched_pattern_position == 1) {
    $err = 1;
    last;
  }
  #die "No match: error was: $error\n" unless $matched_pattern_position == 1;
  # We got our match. Proceed.
  $exp->send("$cmd\n");
}       # end loop over all the steps
 
#
# hard close
$exp->hard_close();
close(LOGF);
#unlink($logf);
}       # end sub login

Code for sendmsg2.pl

Invoke it:

$ ./sendmsg2.pl -m sendmail_host -s [email protected] -r [email protected]

The nice thing with this program is that I can inject a message into sendmail, but also I can inject it directly into the Lotus Notes smtp gateway, bypassing sendmail, and thereby triangulate the problem. The sendmail and Lotus Notes servers have slightly different responses to the various protocol stages, hence I clipped the Expect strings down to the minimal common set of characters after some experimentation.

This program makes it easy to test several scenarios of interest. Leave the final quote and inject into either sendmail or Lotus Notes (LN). Tack on the final quote to see if that really fixes things. The results?

Missing final quote

with final quote added

inject to sendmail

ht” in final email to LN; extension chopped

htm” and all is good

inject to LN

htm in final email; but extension not chopped

htm” and all is good

I now had incontrovertible proof that sendmail, my sendmail was altering the original message. It is looking at the unbalanced quote mark situation and recovering as best as possible by replacing the terminating character “m” with the missing double quote “. I was beginning to suspect it. After that shock drained away, I tried to check the RFCs. I figured it must be some well-meaning attempt on its part to make things right. Well, the RFCs, 822 and 1806 are a little hard to read and apply to this situation.

Let’s be clear. There’s no question that the sender is wrong and ought to be closing out that quote. But I don’t think there’s some single, unambiguous statement from the RFCs that make that abundantly apparent. Nevertheless, of course that’s what I told them to do.

The other thing from reading the RFC is that the whole filename attribute looks optional. To satisfy my curiosity – and possibly provide more options for remediation to aspen – I sent a test where I entirely left out the offending filename=”tower… line. In that case the line above it should have its terminating semicolon shorn:

Content-Disposition: attachment

After all, there already is a name=”tower…” as a Content-type parameter, and the string following that was never in question: it has its terminating semicolon.

Yup, that worked just great too!

Then I thought of another approach. Shouldn’t the overriding definition of the what the filetype is be contained in the Content-type header? What if it were more correctly defined as

Content-type: text/html

?

Content-type appears in two places in this email. I changed them both for good measure, but left the unbalanced quotations problem. Nope. Lotus Notes did not know what to with the attachment it displays as tower status_2012_06_04–09.25.00.ht. So we can’t recommend that course of action.

What Sendmail’s Point-of-View might be
Looking at the book, I see sendmail does care about MIME headers, in particular it cares about the Content-Disposition header. It feels that it is unreliable and hence merely advisory in nature. Also, some years ago there was a sendmail vulnerability wherein malformed multipart MIME messages could cause sendmail to crash (see http://www.kb.cert.org/vuls/id/146718. So maybe sendmail is just a little sensitive to this situation and feels perfectly comfortable and justified in right-forming a malformed header. Just a guess on my part.

Case closed.

Conclusion
We battled a strange email attachment naming error which seemed to be an RFC violation of the MIME protocols. By carefully constructing a testing program we were easily able to reproduce the problem and isolate the fault and recommend corrective actions in the sending program. Now we have a convenient way to inject SMTP email whenever and wherever we want. We feel sendmail’s reputation remains unscathed, though its corrective actions could be characterized as overly solicitous.

Categories
Admin CentOS Linux Raspberry Pi

A few RPM and YUM commands and equivalent on Raspberry Pi

Intro
This post adds nothing to the knowledge out there and readily available on the Internet. I just got tired of looking up elsewhere the few useful rpm and yum commands that I employ. Here’s how I installed a missing binary on one system when I have a similar system that has it.

RPM is the Redhat Package Manager. It is also used on Suse Linux (SLES). A much better resource than this page (Hey, we can’t all be experts!) is http://www.idevelopment.info/data/Unix/Linux/LINUX_RPMCommands.shtml

List all installed packages:

$ rpm −qa
dmidecode-2.11-2.el6.x86_64
libXcursor-1.1.10-2.el6.x86_64
basesystem-10.0-4.el6.noarch
plymouth-core-libs-0.8.3-24.el6.centos.x86_64
libXrandr-1.3.0-4.el6.x86_64
ncurses-base-5.7-3.20090208.el6.x86_64
python-ethtool-0.6-1.el6.x86_64

Same as above – list all installed packages – but list the most recently installed packages first (Wish I had discovered this command sooner)!

$ rpm −qa −−last

libcurl-devel-7.19.7-35.el6                   Mon Apr  1 20:00:47 2013
curl-7.19.7-35.el6                            Mon Apr  1 20:00:47 2013
libidn-devel-1.18-2.el6                       Mon Apr  1 20:00:46 2013
libcurl-7.19.7-35.el6                         Mon Apr  1 20:00:46 2013
libssh2-1.4.2-1.el6                           Mon Apr  1 20:00:45 2013
ncurses-static-5.7-3.20090208.el6             Mon Apr  1 19:59:24 2013
ncurses-devel-5.7-3.20090208.el6              Mon Apr  1 19:58:40 2013
gcc-c++-4.4.7-3.el6                           Fri Mar 15 07:59:36 2013
gcc-gfortran-4.4.7-3.el6                      Fri Mar 15 07:59:34 2013
...

Which package owns a command:

$ rpm −qf `which make`
make-3.81-3.el5

(This was run on an older Redhat 5.6 system which has make.)

Similarly, which package owns a file:

$ rpm −qf /usr/lib64/libssh2.so.1
libssh2-1-1.2.9-4.2.2.1

List files in (an installed) package:
$ rpm −ql freeradius-client-1.1.6-40.1

List files in an rpm package file:
$ rpm −qlp packages/HPSiS1124Core-11.24.241-Linux2.4.rpm

Get history of the package versions on this server:

$ yum history list te-agent

Get history of the list of changes to this package:

$ rpm -q -changelog te-agent

Install a package from a local RPM file:
$ rpm −i openmotif-libs-32bit-2.3.1-3.13.x86_64.rpm

Uninstall a packge:
$ rpm −e package
$ rpm −e freeradius-server-libs-2.1.1-7.12.1

How will you install the missing make in CentOS? Use yum to search for it:

$ yum search make

Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: mirror.umd.edu
 * extras: mirror.umd.edu
 * updates: mirror.cogentco.com
============================== N/S Matched: make ===============================
automake.noarch : A GNU tool for automatically creating Makefiles
...
imake.x86_64 : imake source code configuration and build system
...
make.x86_64 : A GNU tool which simplifies the build process for users
makebootfat.x86_64 : Utility for creation bootable FAT disk
mendexk.x86_64 : Replacement for makeindex with many enhancements
...

How to install it:

$ sudo yum install make.x86_64

Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.umd.edu
 * extras: mirror.umd.edu
 * updates: mirror.cogentco.com
Setting up Install Process
Resolving Dependencies
--&gt; Running transaction check
---&gt; Package make.x86_64 1:3.81-19.el6 will be installed
--&gt; Finished Dependency Resolution
 
Dependencies Resolved
 
===========================================================================================================================
 Package                   Arch                        Version                             Repository                 Size
===========================================================================================================================
Installing:
 make                      x86_64                      1:3.81-19.el6                       base                      389 k
 
Transaction Summary
===========================================================================================================================
Install       1 Package(s)
 
Total download size: 389 k
Installed size: 1.0 M
Is this ok [y/N]: y
Downloading Packages:
make-3.81-19.el6.x86_64.rpm                                                                         | 389 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : 1:make-3.81-19.el6.x86_64                                                                               1/1
 
Installed:
  make.x86_64 1:3.81-19.el6
 
Complete!

make should now be in your path.

If we were dealing with SLES I would use zypper instead of yum, but the idea of searching and installing is similar.

Debian Linux, e.g. Raspberry Pi

Find which package a file belongs to:

> dpkg -S filepath

List installed packages:

> dpkg -l

List all files belonging to the package iperf3:

> dpkg -L iperf3

Transferring packages from one system to another

When I needed to transfer Debian packages from one system with Internet access to another without, I would do:

apt download apache2

Then sftp the file to the other system and on it do

apt install ./apache2_2.4.53-1~deb11u1_amd64.deb

In fact that only worked after I installed all dependencies. This web of files covered all dependencies:

apache2-bin_2.4.53-1~deb11u1_amd64.deb
apache2-data_2.4.53-1~deb11u1_all.deb
apache2-utils_2.4.53-1~deb11u1_amd64.deb
apache2_2.4.53-1~deb11u1_amd64.deb
libapr1_1.7.0-6+deb11u1_amd64.deb
libaprutil1-dbd-mysql_1.6.1-5_amd64.deb
libaprutil1-dbd-odbc_1.6.1-5_amd64.deb
libaprutil1-dbd-pgsql_1.6.1-5_amd64.deb
libaprutil1-dbd-sqlite3_1.6.1-5_amd64.deb
libaprutil1-ldap_1.6.1-5_amd64.deb
libaprutil1_1.6.1-5_amd64.deb
libgdbm-compat4_1.19-2_amd64.deb
libjansson4_2.13.1-1.1_amd64.deb
liblua5.3-0_5.3.3-1.1+b1_amd64.deb
libmariadb3_1%3a10.5.15-0+deb11u1_amd64.deb
libperl5.32_5.32.1-4+deb11u2_amd64.deb
mailcap_3.69_all.deb
mariadb-common_1%3a10.5.15-0+deb11u1_all.deb
mime-support_3.66_all.deb
mysql-common_5.8+1.0.7_all.deb
perl-modules-5.32_5.32.1-4+deb11u2_all.deb
perl_5.32.1-4+deb11u2_amd64.deb
ssl-cert_1.1.0+nmu1_all.deb

Categories
Admin CentOS Security

How to Set up a Secure sftp-only Service

Intro
Updated Jan, 2015.

Usually I post a document because I think I have something to add. This time I found a link that covers the topic better than I could. I just wanted to have it covered here. What if you want to offer an sftp-only jailed account? Can you do that? How do you do it?

The Answer
Well, it used to be all here: http://blog.swiftbyte.com/linux/allowing-sftp-access-while-chrooting-the-user-and-denying-shell-access/. But that link is no longer valid.

I tried it, appropriately modified for CentOS and it worked perfectly. A few notes. Presumably you will already have ssh installed. Who can imagine a server without it? So there’s typically no need to install openssh-server.

I was leery mucking with subsystem sftp. What if it prevented me from doing sftp to my own account and having full access like I’m used to? Turns out it does no harm in that regard.

Very minor point. His documentation might be good for Ubuntu. To restart the ssh daemon in CentOS/Fedora, I recommend a sudo service sshd restart. Do you wonder if that will knock you out of your own ssh session? I did. It does not. Not sure why not!

These groupadd/useradd/usermod functions are “cute.” I’m old school and used to editing the darn files by hand (/etc/passwd, /etc/group). I suppose it’s safer to use the cute functions – less chance a typo could render your server inoperable (yup, done that).

Let’s call my sftp-only user is joerg.

I did the chown root:root thing, but initially the files weren’t accessible to the joerg user. The permissions were 700 on the home directory, now owned by root. That produces this error when you try to sftp:

$ sftp joerg@localhost
sftp> dir

Couldn't get handle: Permission denied

That’s no good, so I liberalized the permissions:

$ sudo chmod go+rx /home/joerg

My /etc/passwd line for this user looks like this:

joerg:x:1004:901:Joerg, etherip author:/home/joerg:/bin/false

So note the unusual shell, /bin/false. That’s the key to locking things down.

In /etc/group I have this;

joerg:x:1004:

If you want to add the entries by hand to passwd and group then if I recall correctly you run a pwconv to generate an appropriate entry for it in /etc/shadow, and a sudo passwd joerg to set up a desired password.

Does it work? Yeah, it really does.

$ sftp joerg@localhost
Connecting to localhost…
sftponlyuser@localhost’s password:
sftp> pwd
Remote working directory: /
sftp> cd ..
sftp> pwd
Remote working directory: /
sftp> cd /etc
Couldn’t canonicalise: No such file or directory
sftp> ls -l
[shows files in /home/joerg]

Moreover, ssh really is shut out:

$ ssh joerg@localhost
joerg@localhost’s password:

This hangs and never returns with a prompt!

Cool, huh?

Locking out this same account
Now suppose you only intended joerg to temporarily have access and you want to lock the account out without actually removing it. This can be done with:

$ sudo passwd -l joerg

This puts an invalid character in that account’s shadow file entry.

Conclusion
We have an easy prescription to make a jailed sftp-only account that we tested and found really works. Regular accounts were not affected. The base article on which I embellished is now kaput so I’ve added a few more details to make up for that.

Categories
Admin Web Site Technologies

Tipsheet: How to run NTP on F5 BigIP

If you feel you’ve added your ntp servers correctly via the GUI (System|Configuration|Device|NTP), and yet you get an output like this:

# ntpq -p

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 ntp1.drj        .INIT.          16 -    - 1024    0    0.000    0.000   0.000
 ntp2.drj        .INIT.          16 -    - 1024    0    0.000    0.000   0.000

and you observe the time is off by seconds or even minutes, then you may have made the mistake I made. I used fully-qualified domain names (FQDN) for the ntp servers.

Switch from FQDNs to IP addresses and it will work fine:

# ntpq -p

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*ntp1.drj         10.23.34.1     3 u  783 1024  377    0.951   -8.830   0.775
+ntp2.drj         10.23.35.1     3 u  120 1024  377   20.963   -8.051   5.705

The date command will now give the correct time.

Having correct time is useful for the logging, especially if you are using ASM and are trying to correlate known activity against the reported errors.

Categories
Admin

Highlights from the AWS Summit in NYC April 19 2012

Intro
It’s interesting to attend popular IT events. I haven’t been to one for awhile. As informative as the presentations are what you can learn by observing the other participants. Here are my observations from the Amazon AWS Summit yesterday in NYC.

Amazon has some really knoweldgeable managers addressing the audience. These guys had clearly mastered the details.

Learned by Observing Others
I’m a follower not a leader when it comes to consumer gadgets. At this event basically everyone had a Smartphone. Blackberries were well-represented though not predominant. iPhones and other Android phones seemed be in great use (unfortunately I cannot tell one from the other). iPads were pretty common, too. As were small notebooks and the occasional standard laptop. Probably 15% of the audience had one of those devices. The conference gave out WiFi access, which required a username/password to be entered in a web page. That was pretty important as I could not get signal otherwise.

I learned that people now take pictures of slides they want to remember for later. Guess I have to get better at using my Smartphone camera! It’s a great idea, really.

It used to be that these events would be very male-dominated. That’s still the case, but not quite as much. Our lunch tables help 10 people each and there was on average one woman per table, so about 10%.

Now about AWS Itself
Lots and lots of companies are doing cool things with Amazon cloud services. Pinterest, PBS, Exfm, Vox even old-line companies like Shell. I don’t think any serious start-up would buy their own infrastructure today when Amazon has made it so easy and economical to put things in the cloud.

Amazon started the service in 2006 and thus are celebrating six years of cloud service. They have lowered prices 19 times. For me the most significant of those price drops was in early March when it became competitive, no, more than competitive to be considered against the pure hosting companies.

Amazon cloud services not counts about 60 service offerings. I don’t claim to understand even most of them, but the ones I do seem really well thought-out. They really deserve to be that Magic Quadrant leader that Gartner paints them out to be, leading in both vision and ability to execute.

S3 is for storing objects. You don’t care how the object is stored. Your reference to the object will be a URL. One example actual usage: storing images. Each image stored as an object in S3. This makes it easy to begin to use a content distribution network (CDN).

I use EC2 so I have some familiarity with it. Users are supposed to have some familiarity with the author of their image. Oops. I do not. News to me: they have a facility to import VM images.

In virtual private cloud (VPC) the user gets to extend his own private address space into Amazon Cloud. Cool. There is a Direct Connect option to avoid traditional VPN tunnels. Security groups offer a decent way to enforce security but I don’t totally get it.

I’m interested in EtherIP. From this summit I learned of a company that has a product for that. The company is Astaro and the appliance is RED (Remote Ethernet Device). Cool.

One molecular discovery company chained together 30000 Amazon cpu cores, ran it for three hours, for a total of 100,000 hours of compute time, and only paid $5,000/hour. All that horsepower together makes for the 50th fastest supercomputer in the world. The equipment value was about $22,000,000.

I guess attendance was at least 2,000 people. There were lots of younger people there as well as an ecosystem of supporting third-party companies.

The Python language was mentioned three or four times. Its star is on the rise.

I came as a private user of AWS. This is my story. I was disappointed that in all their use cases and examples, there was not one mention of individual users such as myself. It was all geared towards organizations. You get worried when you don’t see anyone like yourself out there. I even asked one of the speakers about that but he said that they take users like me as a given. The hard work was getting the enterprises comfortable with it. I’m not entirely convinced.

Conclusion
Amazon has an exciting cloud offering and they get it. Their customers include probably the most interesting companies on the planet.

Categories
Admin DNS Proxy

The IT Detective Agency: Browsing Stopped Working on Internet-Connected Enterprise Laptops

Intro
Recall our elegant solution to DNS clobbering and use of a PAC file, which we documented here: The IT Detective Agency: How We Neutralized Nasty DNS Clobbering Before it Could Bite Us. Things were going smoothly with that kludge in place for months. Then suddenly it wasn’t. What went wrong and how to fix?? Read on.

The Details
We began hearing reports of increasing numbers of users not being able to access Internet sites on their enterprise laptops when directly connected to Internet. Internet Explorer gave that cryptic error to the effect Internet Explorer cannot display the web page. This was a real problem, and not a mere inconvenience, for those cases where a user was using a hotel Internet system that required a web page sign-on – that web page itself could not be displayed, giving that same error message. For simple use at home this error may not have been fatal.

But we had this working. What the…?

I struggled as Rop demoed the problem for me with a user’s laptop. I couldn’t deny it because I saw it with my own eyes and saw that the configuration was correct. Correct as in how we wanted it to be, not correct as in working. Basically all web pages were timing out after 15 seconds or so and displaying cannot display the web page. The chief setting is the PAC file, which is used by this enterprise on their Intranet.

On the Internet (see above-mentioned link) the PAC file was more of an annoyance and we had aliased the DNS value to 127.0.0.1 to get around DNS clobbering by ISPs.

While I was talking about the problem to a colleague I thought to look for a web server on the affected laptop. Yup, there it was:

C:\Users\drj>netstat -an|more

Active Connections

  Proto  Local Address          Foreign Address        State
  TCP    0.0.0.0:80             0.0.0.0:0              LISTENING
  TCP    0.0.0.0:135            0.0.0.0:0              LISTENING
  ...

What the…? Why is there a web server running on port 80? How will it respond to a PAC file request? I quickly got some hints by hitting my own laptop with curl:

$ curl -i 192.168.3.4/proxy.pac

HTTP/1.1 404 Not Found
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Tue, 17 Apr 2012 18:39:30 GMT
Connection: close
Content-Length: 315


Not Found

Not Found


HTTP Error 404. The requested resource is not found.

So the server is Microsoft-HTTPAPI, which upon invetsigation seems to be a Microsoft Web Deployment Agent Service (MsDepSvc).

The main point is that I don’t remember that being there in the past. I felt it’s presence, probably a new “feature” explained the current problem. What to do about it however??

Since this is an enterprise, not a small shop with a couple PCs, turning off MsDepSvc is not a realistic option. It’s probably used for peer-to-peer software distribution.

Hmm. Let’s review why we think our original solution worked in the first place. It didn’t work so well when the DNS was clobbered by my ISP. Why? I think because the ISP put up a web server when it encountered a NXDOMAIN DNS response and that web server gave a 404 not found error when the browser searched for the PAC file. Turning the DNS entry to the loopback interface, 127.0.0.1, gave the browser a valid IP, one that it would connect to and quickly receive a TCP RST (reset). Then it would happily conclude there was no way to reach the PAC file, not use it, and try DIRECT connections to Internet sites. That’s my theory and I’m sticking to it!

In light of all this information the following option emerged as most likely to succeed: set the Internet value of the DNS of the PAC file to a valid IP, and specifically, one that would send a TCP RST (essentially meaning that TCP port 80 is reachable but there is no listener on it).

We tried it and it seemed to work for me and my colleague. We no loner had the problem of not being able to connect to Internet sites with the PAC file configured and directly connected to Internet.

I noticed however that once I got on the enterprise network via VPN I wasn’t able to connect to Internet sites right away. After about five minutes I could.

My theory about that is that I had a too-long TTL on my PAC file DNS entry. The TTL was 30 minutes. I shortened it to five minutes. Because when you think about it, that DNS value should get cached by the PC and retained even after it transitions to Intranet-connected.

I haven’t retested, but I think that adjustment will help.

Conclusion
I also haven’t gotten a lot of feedback from this latest fix, but I’m feeling pretty good about it.

Case: again mostly solved.

Hey, don’t blame me about the ambiguity. I never hear back from the users when things are working 🙂

References
A closely related case involving Verizon “clobbering” TCP RST packets also bit us.

Categories
Admin Security

User Add/Delete Jython Script for RSA Authentication Manager

Intro
I thought I might save someone else the trouble of re-creating a respectable program using the Authentication Manager SDK for version 7.1 which can simply add and delete users to keep the local database in sync with an external database.

History Lesson
We had this problem licked under Authentication Manager v 6.1.2. In that version the sdk was TCL-based and for whatever reason, it seemed a whole lot simpler to understand the model and get working code. When we began to look at v 7.1 we saw we were confronted with a whole different animal that required new understanding and new skills to master.

The Details
Jython is Python plus Java. I really don’t know either language so I used a technique you might call programming by extrapolation. Here is the code. Not really understanding python I preserved as much as possible for fear of breaking something. I nevertheless had to be a little innovative and create a new function.

'''
 * Jython class demonstrating the Administration API
 * usage from a Jython script.
 *
 * Run this script in the utils directory of the Authentication Manager installation.
 *
 * Execute the command "rsautil jython AdminAPIDemos.py create <admin user name> <password>"
 * Execute the command "rsautil jython AdminAPIDemos.py assign <admin user name> <password>"
 * Execute the command "rsautil jython AdminAPIDemos.py update <admin user name> <password>"
 * Execute the command "rsautil jython AdminAPIDemos.py delete <admin user name> <password>"
 *
 * If you are executing this script in an environment other than the predefined
 * rsautil scripting tool you must make the CommandClientAppContext.xml file
 * available in the end of the classpath for this script. You must also configure
 * the necessary connection parameters in a properties file located in the process
 * working directory. See the provided samples for more information.
'''
 
# imports
from jarray import array
import sys
# DrJ required import
# Not Workign! from org.python.modules import re
from java.util.regex import *
from java.lang import *
 
 
from java.util import Calendar,Date
from java.lang import String
 
from org.springframework.beans import BeanUtils
 
from com.rsa.admin import AddGroupCommand
from com.rsa.admin import AddPrincipalsCommand
from com.rsa.admin import DeleteGroupCommand
from com.rsa.admin import DeletePrincipalsCommand
from com.rsa.admin import LinkGroupPrincipalsCommand
from com.rsa.admin import LinkAdminRolesPrincipalsCommand
from com.rsa.admin import SearchAdminRolesCommand
from com.rsa.admin import SearchGroupsCommand
from com.rsa.admin import SearchPrincipalsCommand
from com.rsa.admin import SearchRealmsCommand
from com.rsa.admin import SearchSecurityDomainCommand
from com.rsa.admin import UpdateGroupCommand
from com.rsa.admin import UpdatePrincipalCommand
from com.rsa.admin.data import AdminRoleDTOBase
from com.rsa.admin.data import GroupDTO
from com.rsa.admin.data import IdentitySourceDTO
from com.rsa.admin.data import ModificationDTO
from com.rsa.admin.data import PrincipalDTO
from com.rsa.admin.data import RealmDTO
from com.rsa.admin.data import SecurityDomainDTO
from com.rsa.admin.data import UpdateGroupDTO
from com.rsa.admin.data import UpdatePrincipalDTO
from com.rsa.authmgr.admin.agentmgt import AddAgentCommand
from com.rsa.authmgr.admin.agentmgt import DeleteAgentsCommand
from com.rsa.authmgr.admin.agentmgt import LinkAgentsToGroupsCommand
from com.rsa.authmgr.admin.agentmgt import SearchAgentsCommand
from com.rsa.authmgr.admin.agentmgt import UpdateAgentCommand
from com.rsa.authmgr.admin.agentmgt.data import AgentConstants
from com.rsa.authmgr.admin.agentmgt.data import AgentDTO, ListAgentDTO
from com.rsa.authmgr.admin.hostmgt.data import HostDTO
from com.rsa.authmgr.admin.principalmgt import AddAMPrincipalCommand
from com.rsa.authmgr.admin.principalmgt.data import AMPrincipalDTO
from com.rsa.authmgr.admin.tokenmgt import GetNextAvailableTokenCommand
from com.rsa.authmgr.admin.tokenmgt import LinkTokensWithPrincipalCommand
from com.rsa.authn import SearchPasswordPoliciesCommand
from com.rsa.authn import UpdatePasswordPolicyCommand
from com.rsa.authn.data import PasswordPolicyDTO
from com.rsa.command import ClientSession
from com.rsa.command import CommandException
from com.rsa.command import CommandTargetPolicy, ConnectionFactory
from com.rsa.command.exception import DataNotFoundException, DuplicateDataException
from com.rsa.common.search import Filter
 
'''
 * This class demonstrates the usage patterns of the
 * Authentication Manager 7.1 API.
 *
 * <p>
 * The first set of operations performed if the first
 * command line argument is equal to "create".
 * The sample creates a restricted agent, a group, and a user.
 * Links the user to the group and the group to the agent.
 * </p>
 * <p>
 * The second set of operations performed if the first
 * command line argument is equal to "delete".
 * Lookup the user, group and agent created above.
 * Delete the user, group and agent.
 * </p>
 * <p>
 * A third set of operations is performed if the first
 * command line argument is equal to "assign".
 * Lookup the user and assign the next available
 * SecurID token to the user.
 * Lookup the SuperAdminRole and assign it to the user.
 * </p>
 * <p>
 * A fourth set of operations performed if the first
 * command line argument is equal to "update".
 * Update the Agent, Group, and User objects.
 * </p>
 * <p>
 * A fifth set of operations performed if the first
 * command line argument is equal to "disable".
 * Lookup a password policy with a name that starts
 * with "Initial" and then disable the password history
 * for that policy. Use this to allow the sample to
 * perform multiple updates of the user password using
 * the same password for each update.
 * </p>
 * <p>
 * The APIs demonstrated include the use of the Filter
 * class to generate search expressions for use with
 * all search commands.
 * </p>
'''
class AdminAPIDemos:
 
    '''
     * We need to know these fairly static values throughout this sample.
     * Set the references to top level security domain (realm) and system
     * identity source to use later.
     *
     * @throws CommandException if something goes wrong
    '''
    def __init__(self):
        searchRealmCmd = SearchRealmsCommand()
        searchRealmCmd.setFilter( Filter.equal( RealmDTO.NAME_ATTRIBUTE, "SystemDomain"))
        searchRealmCmd.execute()
        realms = searchRealmCmd.getRealms()
        if( len(realms) == 0 ):
            print "ERROR: Could not find realm SystemDomain"
            sys.exit( 2 )
 
        self.domain = realms[0].getTopLevelSecurityDomain()
        self.idSource = realms[0].getIdentitySources()[0]
 
 
    '''
     * Create an agent and set it to be restricted.
     *
     * @param: name the name of the agent to create
     * @param: addr the IP address for the agent
     * @param: alt array of alternate IP addresses
     * @return: the GUID of the agent just created
     * 
     * @throws CommandException if something goes wrong
    '''
    def createAgent(self, name, addr, alt):
        # need a HostDTO to be set
        host = HostDTO()
        host.setName(name)
        host.setPrimaryIpAddress(addr)
        host.setSecurityDomainGuid(self.domain.getGuid())
        host.setNotes("Created by AM Demo code")
 
        # the agent to be created
        agent = AgentDTO()
        agent.setName(name)
        agent.setHost(host)
        agent.setPrimaryAddress(addr)
        agent.setAlternateAddresses(alt)
        agent.setSecurityDomainId(self.domain.getGuid())
        agent.setAgentType(AgentConstants.STANDARD_AGENT)
        agent.setRestriction(1) # only allow activated groups
        agent.setEnabled(1)
        agent.setOfflineAuthDataRefreshRequired(0)
        agent.setNotes("Created by AM Demo code")
 
        cmd = AddAgentCommand(agent)
 
	try:        
	    cmd.execute()
        except DuplicateDataException:
            print "ERROR: Agent " + name + " already exists."
	    sys.exit(2)
 
        # return the created agents GUID for further linking
        return cmd.getAgentGuid()
 
 
    '''
     * Lookup an agent by name.
     *
     * @param: name the agent name to lookup
     * @return: the GUID of the agent
     * 
     * @throws CommandException if something goes wrong
    '''
    def lookupAgent(self, name):
        cmd = SearchAgentsCommand()
        cmd.setFilter(Filter.equal(AgentConstants.FILTER_HOSTNAME, name))
        cmd.setLimit(1)
        cmd.setSearchBase(self.domain.getGuid())
        # the scope flags are part of the SecurityDomainDTO
        cmd.setSearchScope(SecurityDomainDTO.SEARCH_SCOPE_ONE_LEVEL)
 
        cmd.execute()
 
	if (len(cmd.getAgents()) < 1):
            print "ERROR: Unable to find agent " + name + "."  
	    sys.exit(2)
 
        return cmd.getAgents()[0]
 
 
    '''
     * Update an agent, assumes a previous lookup done by lookupAgent.
     *
     * @param agent the result of a previous lookup
     *
     * @throws CommandException if something goes wrong
    '''
    def updateAgent(self, agent):
        cmd = UpdateAgentCommand()
 
        agentUpdate = AgentDTO()
        # copy the rowVersion to satisfy optimistic locking requirements
        BeanUtils.copyProperties(agent, agentUpdate)
 
        # ListAgentDTO does not include the SecurityDomainId
        # use the GUID of the security domain where agent was created
        agentUpdate.setSecurityDomainId(self.domain.getGuid())
 
        # clear the node secret flag and modify some others
        agentUpdate.setSentNodeSecret(0)
        agentUpdate.setOfflineAuthDataRefreshRequired(1)
        agentUpdate.setIpProtected(1)
        agentUpdate.setEnabled(1)
        agentUpdate.setNotes("Modified by AM Demo code")
 
        # set the requested updates in the command
        cmd.setAgentDTO(agentUpdate)
 
        # perform the update
        cmd.execute()
 
 
    '''
     * Delete an agent.
     *
     * @param: agentGuid the GUID of the agent to delete
     * 
     * @throws CommandException if something goes wrong
    '''
    def deleteAgent(self, agentGuid):
        cmd = DeleteAgentsCommand( [agentGuid] )
        cmd.execute()
 
 
    '''
     * Create an IMS user, needs to exist before an AM user can be
     * created.
     *
     * @param: userId the user's login UID
     * @param: password the user's password
     * @param: first the user's first name
     * @param: last the user's last name
     * 
     * @return: the GUID of the user just created
     * 
     * @throws CommandException if something goes wrong
    '''
    def createUser(self, userId, password, first, last):
        cal = Calendar.getInstance()
 
        # the start date
        now = cal.getTime()
# DrJ: add 50 years from now!    
        cal.add(Calendar.YEAR, 50)
 
        # the account end date
        expire = cal.getTime()
 
        principal = PrincipalDTO()
        principal.setUserID( userId )
        principal.setFirstName( first )
        principal.setLastName( last )
        #     principal.setPassword( password )
 
        principal.setEnabled(1)
        principal.setLockoutStatus(0)
        principal.setAccountStartDate(now)
        #principal.setAccountExpireDate(expire)
        #principal.setAccountExpireDate(0)
        principal.setAdminRole(0)
        principal.setCanBeImpersonated(0)
        principal.setTrustToImpersonate(0)
 
        principal.setSecurityDomainGuid( self.domain.getGuid() )
        principal.setIdentitySourceGuid( self.idSource.getGuid() )
        principal.setDescription("Created by DrJ utilities")
 
        cmd = AddPrincipalsCommand()
        cmd.setPrincipals( [principal] )
 
        try:
            cmd.execute()
	except DuplicateDataException:
            print "ERROR: User " + userId + " already exists."
	    sys.exit(2)
 
        # only one user was created, there should be one GUID result
        return cmd.getGuids()[0]
 
 
    '''
     * Lookup a user by login UID.
     * 
     * @param: userId the user login UID
     *
     * @return: the GUID of the user record.
    '''
    def lookupUser(self, userId):
        cmd = SearchPrincipalsCommand()
        cmd.setFilter(Filter.equal(PrincipalDTO.LOGINUID, userId))
        cmd.setSystemFilter(Filter.empty())
        cmd.setLimit(1)
        cmd.setIdentitySourceGuid(self.idSource.getGuid())
        cmd.setSecurityDomainGuid(self.domain.getGuid())
        cmd.setGroupGuid(None)
        cmd.setOnlyRegistered(1)
        cmd.setSearchSubDomains(0)
 
        cmd.execute()
 
	if (len(cmd.getPrincipals()) < 1):
            print "ERROR: Unable to find user " + userId + "."
	    sys.exit(2)
 
        return cmd.getPrincipals()[0]
 
 
    '''
     * Update the user definition.
     *
     * @param user the principal object from a previous lookup
    '''
    def updateUser(self, user):
        cmd = UpdatePrincipalCommand()
        cmd.setIdentitySourceGuid(user.getIdentitySourceGuid())
 
        updateDTO = UpdatePrincipalDTO()
        updateDTO.setGuid(user.getGuid())
        # copy the rowVersion to satisfy optimistic locking requirements
        updateDTO.setRowVersion(user.getRowVersion())
 
        # collect all modifications here
        mods = []
 
        # first change the email
        mod = ModificationDTO()
        mod.setOperation(ModificationDTO.REPLACE_ATTRIBUTE)
        mod.setName(PrincipalDTO.EMAIL)
        mod.setValues([ user.getUserID() + "@mycompany.com" ])
        mods.append(mod) # add it to the list
 
        # also change the password
        mod = ModificationDTO()
        mod.setOperation(ModificationDTO.REPLACE_ATTRIBUTE)
        mod.setName(PrincipalDTO.PASSWORD)
        mod.setValues([ "MyNewPAssW0rD1!" ])
        mods.append(mod) # add it to the list
 
        # change the middle name
        mod = ModificationDTO()
        mod.setOperation(ModificationDTO.REPLACE_ATTRIBUTE)
        mod.setName(PrincipalDTO.MIDDLE_NAME)
        mod.setValues([ "The Big Cahuna" ])
        mods.append(mod) # add it to the list
 
        # make a note of this update in the description
        mod = ModificationDTO()
        mod.setOperation(ModificationDTO.REPLACE_ATTRIBUTE)
        mod.setName(PrincipalDTO.DESCRIPTION)
        mod.setValues([ "Modified by AM Demo code" ])
        mods.append(mod) # add it to the list
 
        # set the requested updates into the UpdatePrincipalDTO
        updateDTO.setModifications(mods)
        cmd.setPrincipalModification(updateDTO)
 
        # perform the update
        cmd.execute()
 
 
    '''
     * Delete a user.
     *
     * @param: userGuid the GUID of the user to delete
     * 
     * @throws CommandException if something goes wrong
    '''
    def deleteUser(self, userGuid):
        cmd = DeletePrincipalsCommand()
        cmd.setGuids( array( [userGuid], String ) )
        cmd.setIdentitySourceGuid( self.idSource.getGuid() )
        cmd.execute()
 
 
    '''
     * Create an Authentication Manager user linked to the IMS user.
     * The user will have a limit of 3 bad passcodes, default shell
     * will be "/bin/sh", the static password will be "12345678" and
     * the Windows Password for offline authentication will be "Password123!".
     *
     * @param: guid the GUID of the IMS user
     * 
     * @throws CommandException if something goes wrong
    '''
    def createAMUser(self, guid):
        principal = AMPrincipalDTO()
        principal.setGuid(guid)
        principal.setBadPasscodes(3)
        principal.setDefaultShell("/bin/sh")
        principal.setDefaultUserIdShellAllowed(1)
        # these next three innocent-looking lines cost you a license! do not use them!! - DrJ 
        #principal.setStaticPassword("12345678")
        #principal.setStaticPasswordSet(1)
        #principal.setWindowsPassword("Password123!")
 
        cmd = AddAMPrincipalCommand(principal)
        cmd.execute()
 
 
    '''
     * Create a group to assign a user to.
     *
     * @param: name the name of the group to create
     * @return: the GUID of the group just created
     * 
     * @throws CommandException if something goes wrong
    '''
    def createGroup(self, name):
        group = GroupDTO()
        group.setName(name)
        group.setDescription("Created by AM Demo code")
        group.setSecurityDomainGuid(self.domain.getGuid())
        group.setIdentitySourceGuid(self.idSource.getGuid())
 
        cmd = AddGroupCommand()
        cmd.setGroup(group)
 
	try:
            cmd.execute()
	except DuplicateDataException:
            print "ERROR: Group " + name + " already exists."
	    sys.exit(2)
 
        return cmd.getGuid()
 
    '''
     * Lookup a group by name.
     *
     * @param: name the name of the group to lookup
     * @return: the GUID of the group
     * 
     * @throws CommandException if something goes wrong
    '''
    def lookupGroup(self, name):
        cmd = SearchGroupsCommand()
        cmd.setFilter(Filter.equal(GroupDTO.NAME, name))
        cmd.setSystemFilter(Filter.empty())
        cmd.setLimit(1)
        cmd.setIdentitySourceGuid(self.idSource.getGuid())
        cmd.setSecurityDomainGuid(self.domain.getGuid())
        cmd.setSearchSubDomains(0)
        cmd.setGroupGuid(None)
 
        cmd.execute()
 
	if (len(cmd.getGroups()) < 1):
            print "ERROR: Unable to find group " + name + "."
	    sys.exit(2)
 
        return cmd.getGroups()[0]
 
 
    '''
     * Update a group definition.
     *
     * @param group the current group object
    '''
    def updateGroup(self, group):
        cmd = UpdateGroupCommand()
        cmd.setIdentitySourceGuid(group.getIdentitySourceGuid())
 
        groupMod = UpdateGroupDTO()
        groupMod.setGuid(group.getGuid())
        # copy the rowVersion to satisfy optimistic locking requirements
        groupMod.setRowVersion(group.getRowVersion())
 
        # collect all modifications here
        mods = []
 
        mod = ModificationDTO()
        mod.setOperation(ModificationDTO.REPLACE_ATTRIBUTE)
        mod.setName(GroupDTO.DESCRIPTION)
        mod.setValues([ "Modified by AM Demo code" ])
        mods.append(mod)
 
        # set the requested updates into the UpdateGroupDTO
        groupMod.setModifications(mods)
        cmd.setGroupModification(groupMod)
 
        # perform the update
        cmd.execute()
 
 
    '''
     * Delete a group.
     *
     * @param: groupGuid the GUID of the group to delete
     * 
     * @throws CommandException if something goes wrong
    '''
    def deleteGroup(self, groupGuid):
        cmd = DeleteGroupCommand()
        cmd.setGuids( [groupGuid] )
        cmd.setIdentitySourceGuid( self.idSource.getGuid() )
        cmd.execute()
 
 
    '''
     * Assign the user to the specified group.
     *
     * @param: userGuid the GUID for the user to assign
     * @param: groupGuid the GUID for the group
     * 
     * @throws CommandException if something goes wrong
    '''
    def linkUserToGroup(self, userGuid, groupGuid):
        cmd = LinkGroupPrincipalsCommand()
        cmd.setGroupGuids( [groupGuid] )
        cmd.setPrincipalGuids( [userGuid] )
        cmd.setIdentitySourceGuid(self.idSource.getGuid())
 
        cmd.execute()
 
    '''
     * Assign the group to the restricted agent so users can authenticate.
     *
     * @param: agentGuid the GUID for the restricted agent
     * @param: groupGuid the GUID for the group to assign
     * 
     * @throws CommandException if something goes wrong
    '''
    def assignGroupToAgent(self, agentGuid, groupGuid):
        cmd = LinkAgentsToGroupsCommand()
        cmd.setGroupGuids( [groupGuid] )
        cmd.setAgentGuids( [agentGuid] )
        cmd.setIdentitySourceGuid(self.idSource.getGuid())
 
        cmd.execute()
 
    '''
     * Assign next available token to this user.
     *
     * @param: userGuid the GUID of the user to assign the token to
     * 
     * @throws CommandException if something goes wrong
    '''
    def assignNextAvailableTokenToUser(self, userGuid):
        cmd = GetNextAvailableTokenCommand()
        try:
            cmd.execute()
        except DataNotFoundException:
            print "ERROR: No tokens available"
        else:
            tokens = [cmd.getToken().getId()]
            cmd2 = LinkTokensWithPrincipalCommand(tokens, userGuid)
            cmd2.execute()
            print ("Assigned next available SecurID token to user jdoe")
 
    '''
     * Lookup an admin role and return the GUID.
     *
     * @param name the name of the role to lookup
     * @return the GUID for the required role
     *
     * @throws CommandException if something goes wrong
     '''
    def lookupAdminRole(self, name):
        cmd = SearchAdminRolesCommand()
 
        # set search filter to match the name
        cmd.setFilter(Filter.equal(AdminRoleDTOBase.NAME_ATTRIBUTE, name))
        # we only expect one anyway
        cmd.setLimit(1)
        # set the domain GUID
        cmd.setSecurityDomainGuid(self.domain.getGuid())
 
        cmd.execute()
	if (len(cmd.getAdminRoles()) < 1):
            print "ERROR: Unable to find admin role " + name + "."
	    sys.exit(2)
 
        return cmd.getAdminRoles()[0].getGuid()
 
    '''
     * Assign the given admin role to the principal provided.
     *
     * @param adminGuid the GUID for the administrator
     * @param roleGuid the GUID for the role to assign
     *
     * @throws CommandException if something goes wrong
     '''
    def assignAdminRole(self, adminGuid, roleGuid):
        cmd = LinkAdminRolesPrincipalsCommand()
        cmd.setIgnoreDuplicateLink(1)
        cmd.setPrincipalGuids( [ adminGuid ] )
        cmd.setAdminRoleGuids( [ roleGuid ] )
        cmd.execute()
        print ("Assigned SuperAdminRole to user jdoe")
 
    '''
     * Lookup a password policy by name and return the object.
     *
     * @param name the policy name
     * @return the object
     *
     * @throws CommandException if something goes wrong
     '''
    def lookupPasswordPolicy(self, name):
        cmd = SearchPasswordPoliciesCommand()
        cmd.setRealmGuid(self.domain.getGuid())
 
        # match the policy name
        cmd.setFilter(Filter.startsWith(PasswordPolicyDTO.NAME, name))
 
        cmd.execute()
 
	if (len(cmd.getPolicies()) < 1):
            print ("ERROR: Unable to find password policy with name starting with " + name + ".")
	    sys.exit(2)
 
        # we only expect one anyway
        return cmd.getPolicies()[0]
 
    '''
     * Update the given password policy, currently it just disables
     * password history.
     *
     * @param policy the policy to update
     *
     * @throws CommandException if something goes wrong
     '''
    def updatePasswordPolicy(self, policy):
        cmd = UpdatePasswordPolicyCommand()
 
        # disable password history
        policy.setHistorySize(0)
        cmd.setPasswordPolicy(policy)
 
        cmd.execute()
 
    '''
     * Create a collection of related entities, user, agent, group, token.
     *
     * @param admin the administrator user name
     * @param password the administrator password
     * 
     * @throws Exception if something goes wrong
    '''
    def doCreate(self):
 
        # Create a hypothetical agent with four alternate addresses
        addr = "1.2.3.4"
        alt = [ "2.2.2.2",  "3.3.3.3", "4.4.4.4", "5.5.5.5" ]
 
        # create a restricted agent
        agentGuid = self.createAgent("Demo Agent", addr, alt)
        print ("Created Demo Agent")
 
        # create a user group
        groupGuid = self.createGroup("Demo Agent Group")
        print ("Created Demo Agent Group")
 
        # assign the group to the restricted agent
        self.assignGroupToAgent(agentGuid, groupGuid)
        print ("Assigned Demo Agent Group to Demo Agent")
 
        # create a user and the AMPrincipal user record
        userGuid = self.createUser("jdoe", "Password123!", "John", "Doe")
        self.createAMUser(userGuid)
        print ("Created user jdoe")
 
        # link the user to the group
        self.linkUserToGroup(userGuid, groupGuid)
        print ("Added user jdoe to Demo Agent Group")
 
    '''
     * add user by DrJ
     *
     * @param admin the administrator user name
     * @param password the administrator password
     * 
     * @throws Exception if something goes wrong
    '''
    def doAdd(self):
        # create a user and the AMPrincipal user record
        # loop over all users listed in addusers.txt
        f = open('addusers.txt','r')
        str = f.readline()
        while str:
            strs = str.rstrip()
            cols = strs.split(",")
            userid = cols[0]
            fname = cols[1]
            lname = cols[2]
            print userid
# if user already exists we want to go continue with the list
	    try:
                userGuid = self.createUser(userid, "*LK*", fname, lname)
                self.createAMUser(userGuid)
                print "Created user userid,fname,lname: ", userid,",",lname,",",fname,"\n"
            except:
                print "exception for user ",userid,"\n"
            str = f.readline()
 
        f.close()
 
 
    '''
     * Assign the next available token to the user.
     *
     * @param admin the administrator user name
     * @param password the administrator password
     * 
     * @throws Exception if something goes wrong
    '''
    def doAssignNextToken(self):
 
        # lookup and then ...
        userGuid = self.lookupUser("jdoe").getGuid()
 
        # assign the next available token to this user
        self.assignNextAvailableTokenToUser(userGuid)
 
        # now that he has a token make him an admin
        roleGuid = self.lookupAdminRole("SuperAdminRole")
        self.assignAdminRole(userGuid, roleGuid)
 
    '''
     * Delete the entities created by the doCreate method.
     *
     * @param admin the administrator user name
     * @param password the administrator password
     * 
     * @throws Exception if something goes wrong
    '''
    def doDelete(self):
 
        # lookup and then ...
        # loop over all users listed in delusers.txt
        f = open('delusers.txt','r')
        str = f.readline()
        while str:
            # format: userid,fname,lname  . We just want the userid
            cols = str.split(",")
            userid = cols[0]
            print userid
# if user doesn't exist we want to go continue with the list
	    try:
                userGuid = self.lookupUser(userid).getGuid()
                # ... cleanup
                self.deleteUser(userGuid)
                print "Deleted user ",userid
            except:
                print "exception for user ",userid,"\n"
            str = f.readline()
 
        f.close()
 
    '''
     * Update the various entities created by the doCreate method.
     *
     * @throws Exception if something goes wrong
     '''
    def doUpdate(self):
        # lookup and then ...
        agent = self.lookupAgent("Demo Agent")
        group = self.lookupGroup("Demo Agent Group")
        user = self.lookupUser("jdoe")
 
        # ... update
        self.updateAgent(agent)
        print ("Updated Demo Agent")
        self.updateGroup(group)
        print ("Updated Demo Agent Group")
        self.updateUser(user)
        print ("Updated user jdoe")
 
    '''
     * Disable password history limit on default password policy so
     * we can issue multiple updates for the user password.
     *
     * @throws Exception if something goes wrong
     '''
    def doDisablePasswordHistory(self):
        # lookup and then ...
        policy = self.lookupPasswordPolicy("Initial")
 
        # ... update
        self.updatePasswordPolicy(policy)
        print ("Disabled password history")
 
# Globals here
'''
 * Show usage message and exit.
 * 
 * @param msg the error causing the exit
'''
def usage(msg):
    print ("ERROR: " + msg)
    print ("Usage: APIDemos <create|delete> <admin username> <admin password>")
    sys.exit(1)
 
'''
 * Use from command line with three arguments.
 * 
 * <p>
 * First argument:
 * create - to create the required entities
 * assign - to assign the next available token to the user
 * delete - to delete all created entities
 * </p>
 * <p>
 * Second argument is the administrator user name.
 * Third argument is the administrator password.
 * </p>
 * 
 * @param args the command line arguments
'''
 
if len(sys.argv) != 4:
    usage("Missing arguments")
 
# skip script name
args = sys.argv[1:]
 
# establish a connected session with given credentials
conn = ConnectionFactory.getConnection()
session = conn.connect(args[1], args[2])
 
# make all commands execute using this target automatically
CommandTargetPolicy.setDefaultCommandTarget(session)
 
 
try:
    # create instance
    api = AdminAPIDemos()
    # call delusers before addusers
    print "Deleting users...\n"
    api.doDelete()
    print "Adding users...\n"
    api.doAdd()
 
finally:
    # logout when done
    session.logout()

I of course worked from their demo file, AdminAPIDemos.py, and kept the name for simplicity. I added a a doAdd routine and modified their doDelete function.

These modified functions expect external files to exist, addusers.txt and delusers.txt. The syntax of addusers.txt is:

loginname1,first_name,last_name
loginname2,first_name,last_name
...

Delusers.txt has the same syntax.

The idea is that if you can create these files once per day with the new users/removed users from your corporate directory by some other means, then you have a way to use them as a basis for keeping your AM internal database in sync with your external enterprise directory, whatever it might be.

Other Notes
Initially I saw my users were set to expire after a year or so. The original code I borrwed from had lines like this:

        cal = Calendar.getInstance()
 
        # the start date
        now = cal.getTime()
 
        cal.add(Calendar.YEAR, 1)
 
        # the account end date
        expire = cal.getTime()

which caused this. I eventually found how to set a flag to create the account with unlimited validity.

I also introduced a very simple regex handling to break up the input lines. This caused the need for importing additional classes:

from java.util.regex import *
from java.lang import *

I could not get python regexes to work.

I also found these three innocent-looking lines were costing me a license unit for each added user:

        principal.setStaticPassword("12345678")
        principal.setStaticPasswordSet(1)
        principal.setWindowsPassword("Password123!")

So I commented them out as I did not need them.

That’s it!

Getting the SDK running cost me a few days but at least I’ve documented that as well in pretty good detail: Problems with Jython API for RSA Authentication Manager.

Conclusion
We’ve shared with the community an actual, working jython API for adding/removing users from an RSA Authentication Manager v 7.1 database.