Turning HP SiteScope into SiteScope Classic with Perl

Post author By john
Post date February 28, 2012
15 Comments on Turning HP SiteScope into SiteScope Classic with Perl

Intro
HP siteScope is a terrific web application tool and not too expensive for those who have any kind of a budget. The built-in monitor types are a bit limited, but since it allows calls to user-provided scripts your imagination is the only real limitation. For those with too many responsibilities and too little time on their hands it is a real productivity enhancer.

I’ve been using the product for 12 years now – since it was Freshwater SiteScope. I still have misgivings about the interface change introduced some years ago when it was part of Mercury. It went from simple and reliable to Java, complicated and flaky. To this day I have to re-start a SiteScope screen in my browser on a daily basis as the browser cannot recover from a server restart or who knows what other failures.

So I longed for the days of SiteScope Classic. We kept it running for as long as possible, years in fact. But at some point there were no more releases created for the classic view. So I investigated the feasibility of creating my own conversion tool. And…partially succeeded. Succeeded to the point where I can pull up the web page on my Blackberry and get the statuses and history. Think you can do that with regular HP SiteScope? I can’t. Maybe there’s an upgrade for it, but still. It’s nice to have the classic interface when you want to pull up the statuses as quickly as possible, regardless of the Blackberry display issue.

Looking back at my code, I obviously decided to try my hand at OO (object oriented) programming in Perl, with mixed results. Perl’s OO syntax isn’t the best, which addles comprehension. Without further ado, let’s jump into it.

The Details
It relies on something I noticed, that this URL on your HP SiteScope server, http://localhost:8080/SiteScope/services/APIConfigurationImpl?method=getConfigurationSnapshot, contains a tree of relationships of all the monitors. Cool, right? But it’s not a tree like you or I would design. Between parent and child is an intermediate layer. I suppose you need that because a group can contain monitors (my only focus in this exercise), but it can also contain alerts and maybe some other properties as well. So I guess the intermediate layer gives them the flexibility to represent all that, though it certainly added to my complication in parsing it. That’s why you’ll see the concern over “grandkids.” I developed a recursive, web-enabled Perl program to parse through this xml. That gives me the tools to build the nice hierarchical groupings. But it does not give me the statuses.

For the status of each monitor I wrote a separate scraper script that simply reads the entire daily SiteScope log every minute! Crude, but it works. I use it for an installation with hundreds of monitors and a log file that grows to 9 MB by the end of the day so I know it scales to that size. Beyond that it’s untested.

In addition to giving only the relationships, the xml also changes with every invocation. It attaches ID numbers to the monitors which initially you think is a nice unique identifier, but they change from invocation to invocation! So an additional challenge was to match up the names of the monitors in the xml output to the names as recorded in the SiteScope log. Also a bit tricky, but in general doable.

So without further ado, here’s the source code for the xml parser and main program which gets called from the web:

#!/usr/bin/perl
# Copyright work under the Artistic License, http://www.opensource.org/licenses/Artistic-2.0
# build v.simple SiteScope web GUI appropriate for smartphones
# 7/2010
#
# Id is our package which defines th Id class
use Id;
use CGI::Pretty;
my $cgi=new CGI;
$DEBUG = 0;
# GIF location on SiteScope classic
$ssgifs = "/artwork/";
$health{good} = qq(<img src="${ssgifs}okay.gif">);
$health{error} = qq(<img src="${ssgifs}error.gif">);
$health{warning} = qq(<img src="${ssgifs}warning.gif">);
# report CGI
$rprt = "/SS/rprt";
# the frustrating thing is that this xml output changes almost every time you call it
$url = 'http://localhost:8080/SiteScope/services/APIConfigurationImpl?method=getConfigurationSnapshot';
# get current health of all monitors - which is scraped from the log every minute by a hilgarj cron job
$monitorstats = "/tmp/monitorstats.txt";
print "Content-type: text/plain\n\n" if $DEBUG;
open(MONITORSTATS,"$monitorstats") || die "Cannot open monitor stats file $monitorstats!!";
while(<MONITORSTATS>) {
  chomp;
  ($monitor,$status,$value) = /([^\t]+)\t([^\t]+)\t([^\t]+)/;
  $monitors{"$monitor"} = $status;
  $monitorv{"$monitor"} = $value;
}
open(CURL,"curl $url 2>/dev/null|") || die "cannot open $url for reading!!\n";
my %myobjs = ();
# the xml is one long line!
@lines = <CURL>;
#print "xml line: $lines[0]\n" if $DEBUG;
@multiRefs = split "<multiRef",$lines[0];
#parse multiRefs
# create top-level object
my $id = Id->new (
      id => "id0");
# hash of this object with id as key
$myobjs{"id0"} = $id;
 
# first build our objects...
foreach $mref (@multiRefs) {
  next unless $mref =~ /\sid=/;
#  id="id0" ...
  ($parentid) =  $mref =~ /id=\"(id\d+)/;
  print "parentid: $parentid\n" if $DEBUG;
# watch out for <item><key xsi:type="soapenc:string">groupSnapshotChildren</key><value href="#id3 ...
# vs <item><key xsi:type="soapenc:string">Network</key><value href="#id40"/>
  print "mref: $mref\n" if $DEBUG;
  @ids = split /<item><key/, $mref;
# then loop over ids mentioned in this mref
  foreach $myid (@ids) {
    next unless $myid =~ /href="#(id\d+)/;
    next unless $myobjs{"$parentid"};
# types include group, monitor, alert
    ($typebyregex) = $myid =~ />snapshot_(\w+)SnapshotChildren</;
    $parenttype = $myobjs{"$parentid"}->type();
    $type = $typebyregex ? $typebyregex : $parenttype;
    print "type: $type\n" if $DEBUG;
# skip alert definitions
    next if $type eq "alert";
    print "myid: $myid\n" if $DEBUG;
    ($actualid) = $myid =~ /href="#(id\d+)/;
    print "actualid: $actualid\n" if $DEBUG;
# construct object
    my $id = Id->new (
      id => $actualid,
      type => $type,
      parentid => $parentid );
# build hash of these objects with actualid as key
    $myobjs{$actualid} = $id;
# addchild to parent. note that parent should already have been encountered
    $myobjs{"$parentid"}->addchild($actualid);
    if ($myid !~ /groupSnapshotChildren/) {
# interesting child - has name (every other generation has no name!)
      ($name) = $myid =~ /string\">(.+?)<\/key/;  # use non-greedy operator
      print "name: $name\n" if $DEBUG;
# some names are not of interest to us: alerts, which end in "error" or "good"
      if ($name !~ /(error|good)$/) {
# name may not be unique - get extended name which include all parents
        if (defined $myobjs{"$parentid"}->parentid()) {
          $gdparid = $myobjs{"$parentid"}->parentid();
          $gdparname = $myobjs{"$gdparid"}->extname();
# extname -> extended, or distinguished name.  Should be unique
          $extname = $gdparname. '/' . $name;
        } else {
# 1st generation
          print "1st generation\n" if $DEBUG;
          $extname = $name;
        }
        print "extname: $extname\n" if $DEBUG;
        $id->name($name);
        $id->extname($extname);
        $id->isanamedid(1);
        $myobjs{"$parentid"}->hasnamedkids(1); # want to mark its parent as "special"
# we also need our hash to reference objects by extended name since id changes with each extract and name
may not be unique
        $myobjs{"$extname"} = $id;
      } # end conditional over desirable name check
    } else {
      $id->isanamedid(0);
    }
  }
}
#
# now it's all parsed and our objects are alive. Let's build a web site!
#
# build a cookie containing path
my $pi = $ENV{PATH_INFO};
$script = $ENV{SCRIPT_NAME};
$ua = $ENV{HTTP_USER_AGENT};
# Blackberry browser test
$BB = $ua =~ /^BlackBerry/i ? 1 : 0;
$MSIE = $ua =~ /MSIE /;
# font-size depends on browser
$FS = "font-size: x-small;" if $MSIE;
$cookie = $cgi->cookie("pathinfo");
$uri = $script . $pi;
$cookie=$cgi->cookie(-name=>"pathinfo", -value=>"$uri");
print $cgi->header(-type=>"text/html",-cookie=>$cookie);
($url) = $pi =~ m#([^/]+)$#;
#  -title=>'SmartPhone View',
# this doesn't work, sigh...
#print $cgi->start_html(-head=>meta({-http_equiv=>'Refresh'}));
print qq( <HEAD>
<meta http-equiv="Expires" content="0">
<meta http-equiv="Pragma" content="no-cache">
<meta HTTP-EQUIV="Refresh" CONTENT="60; URL=$url">
<TITLE>SiteScope Classic $url Detail</TITLE>
<style type="text/css">
a.good {color: green; }
a.warning {color: green; }
a.error {color: red; }
td {font-family: Arial, Helvetica, sans-serif; $FS}
p.ss {font-family: Arial, Helvetica, sans-serif;}
</style>
<link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" />
<script type=text/javascript>
function changeme(elemid,longvalue)
{
document.getElementById(elemid).innerText=longvalue;
}
function restoreme(elemid,truncvalue)
{
document.getElementById(elemid).innerText=truncvalue;
}
</script>
</HEAD><body>
);
 
#print $cgi->h1("This is the heading");
# parse path
# top lvl name:2nd lvl name:3rd lvl name
$altpi = $cgi->path_info();
print $cgi->p("pi is $pi") if $DEBUG;
#print $cgi->p("altpi is $altpi");
# relative url
$rurl = $cgi->url(-relative=>1);
if ($pi eq "") {
# the top
# top id is id3
  print qq(<p class="ss">);
  $myid = "id3";
  foreach $kid ($myobjs{"$myid"}->get_children()) {
    my $kidname = $myobjs{"$kid"}->name();
# kids can be subgroups or standalone monitors
    my $health = recurse("/$kidname");
    print "$health{$health} <a href=\"$rurl/$kidname\">$kidname</a><br>\n";
    $prodtest = $kid if $kidname eq "Production";
  }
  print "</p>\n";
} else {
  $extname = $pi;
  print "pi,name,extname,script: $pi,$name,$extname,$script\n" if $DEBUG;
# print where we are
  $uriname = $pi;
  $uriname =~ s#^/##;
  #print $cgi->p("name is $name");
  #print $cgi->p("uriname is $uriname");
  $uricompositepart = "/";
  @uriparts = split('/',$uriname);
  $lastpart = pop @uriparts;
  print qq(<p class="ss"><a href="$script"><b>Sitescope</b></a><br>);
  print qq(<b>Monitors in: );
  foreach $uripart (@uriparts) {
    my $healthp = recurse("$uricompositepart$uripart");
# build valid link
    ##$link = qq(<a class="good" href="$script$uricompositepart$uripart">$uripart</a>: );
    $link = qq(<a class="$healthp" href="$script$uricompositepart$uripart">$uripart</a>: );
    $uricompositepart .= "$uripart/";
    print $link;
  }
  my $healthp = recurse("$uricompositepart$lastpart");
  $color = $healthp eq "error" ? "red" : "green";
  print qq(<font color="$color">$lastpart</font></b></p>\n);
  print qq(<table border="1" cellspacing="0">);
  #print qq(<table>);
  %hashtrs = ();
  foreach $kid ($myobjs{"$extname"}->get_children()) {
    print "kid id: " . $myobjs{"$kid"}->id() . "\n" if $DEBUG;
    next unless $myobjs{"$kid"}->hasnamedkids();
    foreach $gdkid ($myobjs{"$kid"}->get_children()) {
      print "gdkid id: " . $myobjs{"$gdkid"}->id() . "\n" if $DEBUG;
      $gdkidname = $myobjs{"$gdkid"}->name();
      $gdkidextname = $myobjs{"$gdkid"}->extname();
      my $health = recurse("$gdkidextname");
      my $type = $myobjs{"$gdkid"}->type();
# dig deeper to learn health of the grankid's grandkids
      $objct = $healthct{good} = $healthct{error} = $healthct{warning} = 0;
      foreach $ggkid ($myobjs{"$gdkidextname"}->get_children()) {
        print "ggkid id: " . $myobjs{"$ggkid"}->id() . "\n" if $DEBUG;
        next unless $myobjs{"$ggkid"}->hasnamedkids();
        foreach $gggdkid ($myobjs{"$ggkid"}->get_children()) {
          print "gggdkid id: " . $myobjs{"$gggdkid"}->id() . "\n" if $DEBUG;
          $gggdkidname = $myobjs{"$gggdkid"}->name();
          $gggdkidextname = $myobjs{"$gggdkid"}->extname();
          my $health = recurse("$gggdkidextname");
          $objct++;
          $healthct{$health}++;
        }
      }
      $elemct++;
      $elemid = "elemid" . $elemct;
# groups should have distinctive cell background color to set them apart from monitors
      if ($type eq "group") {
        $bgcolor = "#F0F0F0";
        $celllink = "$lastpart/$gdkidname";
        $truncvalue = qq(<font color="red">$healthct{error}</font>/$objct);
        $tdval = $truncvalue;
      } else {
        $bgcolor = "#FFFFFF";
        $celllink = "$rprt?$gdkidname";
# truncate monitor value to save display space
        $longvalue = $monitorv{"$gdkidname"};
        (my $truncvalue) = $monitorv{"$gdkidname"} =~ /^(.{7,9})/;
        $truncvalue = $truncvalue? $truncvalue : "&nbsp;";
        $tdval = qq(<span id="$elemid" onmouseover="changeme('$elemid','$longvalue')" onmouseout="restorem
e('$elemid','$truncvalue')">$truncvalue</span>);
      }
      $hashtrs{"$gdkidname"} = qq(<tr><td bgcolor="#000000">$health{$health} </td><td>$tdval</td><td bgcol
or="$bgcolor"><a href="$celllink">$gdkidname</a></td></tr>\n);
# for health we're going to have to recurse
    }
  }
# print out in alphabetical order
  foreach $key (sort(keys %hashtrs)) {
    print $hashtrs{"$key"};
  }
  print "</table>";
}
print $cgi->end_html();
#######################################
sub recurse {
# to get the union of health of all ancestors
my $moniext = shift;
my ($moni) = $moniext =~ m#/([^/]+)$#;
# don't bother recursing and all that unless we have to...
return $myobjs{"$moniext"}->health() if defined $myobjs{"$moniext"}->health();
print "moni,moniext: $moni, $moniext\n" if $DEBUG;
my ($kid,$gdkidextname,$health,$cumhealth);
$cumhealth = $health = $monitors{"$moni"} ? $monitors{"$moni"} : "good";
foreach $kid ($myobjs{"$moniext"}->get_children()) {
    if ($myobjs{"$kid"}->hasnamedkids()) {
      foreach $gdkid ($myobjs{"$kid"}->get_children()) {
        $gdkidextname = $myobjs{"$gdkid"}->extname();
# for health we're going to have to recurse
        $health = recurse("$gdkidextname");
        if ($health eq "error" || $cumhealth eq "error") {
          $cumhealth = "error";
        } elsif ($health eq "warning" || $cumhealth eq "warning") {
          $cumhealth = "warning";
        }
      }
    } else {
# this kid is end of line
      $health = $monitors{"$kid"} ? $monitors{"$kid"} : "good";
        if ($health eq "error" || $cumhealth eq "error") {
          $cumhealth = "error";
        } elsif ($health eq "warning" || $cumhealth eq "warning") {
          $cumhealth = "warning";
        }
    }
}
$myobjs{"$moniext"}->health("$cumhealth");
return $cumhealth;
} # end sub recurse

#!/usr/bin/perl # Copyright work under the Artistic License, http://www.opensource.org/licenses/Artistic-2.0 # build v.simple SiteScope web GUI appropriate for smartphones # 7/2010 # # Id is our package which defines th Id class use Id; use CGI::Pretty; my $cgi=new CGI; $DEBUG = 0; # GIF location on SiteScope classic $ssgifs = "/artwork/"; $health{good} = qq(<img src="${ssgifs}okay.gif">); $health{error} = qq(<img src="${ssgifs}error.gif">); $health{warning} = qq(<img src="${ssgifs}warning.gif">); # report CGI $rprt = "/SS/rprt"; # the frustrating thing is that this xml output changes almost every time you call it $url = 'http://localhost:8080/SiteScope/services/APIConfigurationImpl?method=getConfigurationSnapshot'; # get current health of all monitors - which is scraped from the log every minute by a hilgarj cron job $monitorstats = "/tmp/monitorstats.txt"; print "Content-type: text/plain\n\n" if $DEBUG; open(MONITORSTATS,"$monitorstats") || die "Cannot open monitor stats file $monitorstats!!"; while(<MONITORSTATS>) { chomp; ($monitor,$status,$value) = /([^\t]+)\t([^\t]+)\t([^\t]+)/; $monitors{"$monitor"} = $status; $monitorv{"$monitor"} = $value; } open(CURL,"curl $url 2>/dev/null|") || die "cannot open $url for reading!!\n"; my %myobjs = (); # the xml is one long line! @lines = <CURL>; #print "xml line: $lines[0]\n" if $DEBUG; @multiRefs = split "<multiRef",$lines[0]; #parse multiRefs # create top-level object my $id = Id->new ( id => "id0"); # hash of this object with id as key $myobjs{"id0"} = $id; # first build our objects... foreach $mref (@multiRefs) { next unless $mref =~ /\sid=/; # id="id0" ... ($parentid) = $mref =~ /id=\"(id\d+)/; print "parentid: $parentid\n" if $DEBUG; # watch out for <item><key xsi:type="soapenc:string">groupSnapshotChildren</key><value href="#id3 ... # vs <item><key xsi:type="soapenc:string">Network</key><value href="#id40"/> print "mref: $mref\n" if $DEBUG; @ids = split /<item><key/, $mref; # then loop over ids mentioned in this mref foreach $myid (@ids) { next unless $myid =~ /href="#(id\d+)/; next unless $myobjs{"$parentid"}; # types include group, monitor, alert ($typebyregex) = $myid =~ />snapshot_(\w+)SnapshotChildren</; $parenttype = $myobjs{"$parentid"}->type(); $type = $typebyregex ? $typebyregex : $parenttype; print "type: $type\n" if $DEBUG; # skip alert definitions next if $type eq "alert"; print "myid: $myid\n" if $DEBUG; ($actualid) = $myid =~ /href="#(id\d+)/; print "actualid: $actualid\n" if $DEBUG; # construct object my $id = Id->new ( id => $actualid, type => $type, parentid => $parentid ); # build hash of these objects with actualid as key $myobjs{$actualid} = $id; # addchild to parent. note that parent should already have been encountered $myobjs{"$parentid"}->addchild($actualid); if ($myid !~ /groupSnapshotChildren/) { # interesting child - has name (every other generation has no name!) ($name) = $myid =~ /string\">(.+?)<\/key/; # use non-greedy operator print "name: $name\n" if $DEBUG; # some names are not of interest to us: alerts, which end in "error" or "good" if ($name !~ /(error|good)$/) { # name may not be unique - get extended name which include all parents if (defined $myobjs{"$parentid"}->parentid()) { $gdparid = $myobjs{"$parentid"}->parentid(); $gdparname = $myobjs{"$gdparid"}->extname(); # extname -> extended, or distinguished name. Should be unique $extname = $gdparname. '/' . $name; } else { # 1st generation print "1st generation\n" if $DEBUG; $extname = $name; } print "extname: $extname\n" if $DEBUG; $id->name($name); $id->extname($extname); $id->isanamedid(1); $myobjs{"$parentid"}->hasnamedkids(1); # want to mark its parent as "special" # we also need our hash to reference objects by extended name since id changes with each extract and name may not be unique $myobjs{"$extname"} = $id; } # end conditional over desirable name check } else { $id->isanamedid(0); } } } # # now it's all parsed and our objects are alive. Let's build a web site! # # build a cookie containing path my $pi = $ENV{PATH_INFO}; $script = $ENV{SCRIPT_NAME}; $ua = $ENV{HTTP_USER_AGENT}; # Blackberry browser test $BB = $ua =~ /^BlackBerry/i ? 1 : 0; $MSIE = $ua =~ /MSIE /; # font-size depends on browser $FS = "font-size: x-small;" if $MSIE; $cookie = $cgi->cookie("pathinfo"); $uri = $script . $pi; $cookie=$cgi->cookie(-name=>"pathinfo", -value=>"$uri"); print $cgi->header(-type=>"text/html",-cookie=>$cookie); ($url) = $pi =~ m#([^/]+)$#; # -title=>'SmartPhone View', # this doesn't work, sigh... #print $cgi->start_html(-head=>meta({-http_equiv=>'Refresh'})); print qq( <HEAD> <meta http-equiv="Expires" content="0"> <meta http-equiv="Pragma" content="no-cache"> <meta HTTP-EQUIV="Refresh" CONTENT="60; URL=$url"> <TITLE>SiteScope Classic $url Detail</TITLE> <style type="text/css"> a.good {color: green; } a.warning {color: green; } a.error {color: red; } td {font-family: Arial, Helvetica, sans-serif; $FS} p.ss {font-family: Arial, Helvetica, sans-serif;} </style> <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" /> <script type=text/javascript> function changeme(elemid,longvalue) { document.getElementById(elemid).innerText=longvalue; } function restoreme(elemid,truncvalue) { document.getElementById(elemid).innerText=truncvalue; } </script> </HEAD><body> ); #print $cgi->h1("This is the heading"); # parse path # top lvl name:2nd lvl name:3rd lvl name $altpi = $cgi->path_info(); print $cgi->p("pi is $pi") if $DEBUG; #print $cgi->p("altpi is $altpi"); # relative url $rurl = $cgi->url(-relative=>1); if ($pi eq "") { # the top # top id is id3 print qq(<p class="ss">); $myid = "id3"; foreach $kid ($myobjs{"$myid"}->get_children()) { my $kidname = $myobjs{"$kid"}->name(); # kids can be subgroups or standalone monitors my $health = recurse("/$kidname"); print "$health{$health} <a href=\"$rurl/$kidname\">$kidname</a><br>\n"; $prodtest = $kid if $kidname eq "Production"; } print "</p>\n"; } else { $extname = $pi; print "pi,name,extname,script: $pi,$name,$extname,$script\n" if $DEBUG; # print where we are $uriname = $pi; $uriname =~ s#^/##; #print $cgi->p("name is $name"); #print $cgi->p("uriname is $uriname"); $uricompositepart = "/"; @uriparts = split('/',$uriname); $lastpart = pop @uriparts; print qq(<p class="ss"><a href="$script"><b>Sitescope</b></a><br>); print qq(<b>Monitors in: ); foreach $uripart (@uriparts) { my $healthp = recurse("$uricompositepart$uripart"); # build valid link ##$link = qq(<a class="good" href="$script$uricompositepart$uripart">$uripart</a>: ); $link = qq(<a class="$healthp" href="$script$uricompositepart$uripart">$uripart</a>: ); $uricompositepart .= "$uripart/"; print $link; } my $healthp = recurse("$uricompositepart$lastpart"); $color = $healthp eq "error" ? "red" : "green"; print qq(<font color="$color">$lastpart</font></b></p>\n); print qq(<table border="1" cellspacing="0">); #print qq(<table>); %hashtrs = (); foreach $kid ($myobjs{"$extname"}->get_children()) { print "kid id: " . $myobjs{"$kid"}->id() . "\n" if $DEBUG; next unless $myobjs{"$kid"}->hasnamedkids(); foreach $gdkid ($myobjs{"$kid"}->get_children()) { print "gdkid id: " . $myobjs{"$gdkid"}->id() . "\n" if $DEBUG; $gdkidname = $myobjs{"$gdkid"}->name(); $gdkidextname = $myobjs{"$gdkid"}->extname(); my $health = recurse("$gdkidextname"); my $type = $myobjs{"$gdkid"}->type(); # dig deeper to learn health of the grankid's grandkids $objct = $healthct{good} = $healthct{error} = $healthct{warning} = 0; foreach $ggkid ($myobjs{"$gdkidextname"}->get_children()) { print "ggkid id: " . $myobjs{"$ggkid"}->id() . "\n" if $DEBUG; next unless $myobjs{"$ggkid"}->hasnamedkids(); foreach $gggdkid ($myobjs{"$ggkid"}->get_children()) { print "gggdkid id: " . $myobjs{"$gggdkid"}->id() . "\n" if $DEBUG; $gggdkidname = $myobjs{"$gggdkid"}->name(); $gggdkidextname = $myobjs{"$gggdkid"}->extname(); my $health = recurse("$gggdkidextname"); $objct++; $healthct{$health}++; } } $elemct++; $elemid = "elemid" . $elemct; # groups should have distinctive cell background color to set them apart from monitors if ($type eq "group") { $bgcolor = "#F0F0F0"; $celllink = "$lastpart/$gdkidname"; $truncvalue = qq(<font color="red">$healthct{error}</font>/$objct); $tdval = $truncvalue; } else { $bgcolor = "#FFFFFF"; $celllink = "$rprt?$gdkidname"; # truncate monitor value to save display space $longvalue = $monitorv{"$gdkidname"}; (my $truncvalue) = $monitorv{"$gdkidname"} =~ /^(.{7,9})/; $truncvalue = $truncvalue? $truncvalue : " "; $tdval = qq(<span id="$elemid" onmouseover="changeme('$elemid','$longvalue')" onmouseout="restorem e('$elemid','$truncvalue')">$truncvalue</span>); } $hashtrs{"$gdkidname"} = qq(<tr><td bgcolor="#000000">$health{$health} </td><td>$tdval</td><td bgcol or="$bgcolor"><a href="$celllink">$gdkidname</a></td></tr>\n); # for health we're going to have to recurse } } # print out in alphabetical order foreach $key (sort(keys %hashtrs)) { print $hashtrs{"$key"}; } print "</table>"; } print $cgi->end_html(); ####################################### sub recurse { # to get the union of health of all ancestors my $moniext = shift; my ($moni) = $moniext =~ m#/([^/]+)$#; # don't bother recursing and all that unless we have to... return $myobjs{"$moniext"}->health() if defined $myobjs{"$moniext"}->health(); print "moni,moniext: $moni, $moniext\n" if $DEBUG; my ($kid,$gdkidextname,$health,$cumhealth); $cumhealth = $health = $monitors{"$moni"} ? $monitors{"$moni"} : "good"; foreach $kid ($myobjs{"$moniext"}->get_children()) { if ($myobjs{"$kid"}->hasnamedkids()) { foreach $gdkid ($myobjs{"$kid"}->get_children()) { $gdkidextname = $myobjs{"$gdkid"}->extname(); # for health we're going to have to recurse $health = recurse("$gdkidextname"); if ($health eq "error" || $cumhealth eq "error") { $cumhealth = "error"; } elsif ($health eq "warning" || $cumhealth eq "warning") { $cumhealth = "warning"; } } } else { # this kid is end of line $health = $monitors{"$kid"} ? $monitors{"$kid"} : "good"; if ($health eq "error" || $cumhealth eq "error") { $cumhealth = "error"; } elsif ($health eq "warning" || $cumhealth eq "warning") { $cumhealth = "warning"; } } } $myobjs{"$moniext"}->health("$cumhealth"); return $cumhealth; } # end sub recurse

I call it simply “ss” to minimize the typing required. You see it uses a package called Id.pm which I wrote to encapsulate the class and methods. Here is Id.pm:

package Id;
# Copyright work under the Artistic License, http://www.opensource.org/licenses/Artistic-2.0
# class for storing data about an id
# URL (not currently protected): http://localhost:8080/SiteScope/services/APIConfigurationImpl?method=getC
onfigurationSnapshot
# class for storing data about a group
use warnings;
use strict;
use Carp;
#group methods
# constructor
# get_members
# get_name
# get_id
# addmember
#
# member methods
# constructor
# get_id
# get_name
# get_type
# get_gp
# set_gp
 
sub new {
  my $class = shift;
  my $self = {@_};
  bless($self, "Id");
  return $self;
}
# get-set methods, p. 355
sub parentid { $_[0]->{parentid}=$_[1] if defined $_[1]; $_[0]->{parentid} }
sub isanamedid { $_[0]->{isanamedid}=$_[1] if defined $_[1]; $_[0]->{isanamedid} }
sub id { $_[0]->{id}=$_[1] if defined $_[1]; $_[0]->{id} }
sub name { $_[0]->{name}=$_[1] if defined $_[1]; $_[0]->{name} }
sub extname { $_[0]->{extname}=$_[1] if defined $_[1]; $_[0]->{extname} }
sub type { $_[0]->{type}=$_[1] if defined $_[1]; $_[0]->{type} }
sub health { $_[0]->{health}=$_[1] if defined $_[1]; $_[0]->{health} }
sub hasnamedkids { $_[0]->{hasnamedkids}=$_[1] if defined $_[1]; $_[0]->{hasnamedkids} }
 
# get children - use anonymous array, book p. 221-222
sub get_children {
# return empty array if arrary hasn't been defined...
  defined @{$_[0]->{children}} ? @{$_[0]->{children}} : ();
}
# adding children
sub addchild {
  $_[0]->{children} = [] unless defined  $_[0]->{children};
  push @{$_[0]->{children}},$_[1];
}
 
1;

ss also assumes the existence of just a few of the images from SiteScope classic – the green circle for good, red diamond for error and yellow warning, etc.. I borrowed them SiteScope classic.

Here is the code for the log scraper:

#!/usr/bin/perl
# analyze SiteScope log file
# Copyright work under the Artistic License, http://www.opensource.org/licenses/Artistic-2.0
# 8/2010
$DEBUG = 0;
$logdir = "/opt/SiteScope/logs";
$monitorstats = "/tmp/monitorstats.txt";
$monitorstatshis = "/tmp/monitorstats-his.txt";
$date = `date +%Y_%m_%d`;
chomp($date);
$file = "$logdir/SiteScope$date.log";
open(LOG,"$file") || die "Cannot open SiteScope log file: $file!!\n";
# example lines:
# 16:51:07 08/02/2010     good    LDAPServers     LDAP SSL test : ldapsrv.drj.com exit: 0, 0.502 sec    1:
3481  0       502
#16:51:22 08/02/2010     good    Network DNS: (AMEAST) ns2  0.033 sec   2:3459      200     33      ok
#16:51:49 08/02/2010     good    Proxy   proxy.pac script on iwww    0.055 sec   2:12467 200     55   ok
     4288    1280782309      0    0  55      0       0      200  0
#16:52:04 08/02/2010     good    Proxy   Disk Space: earth /logs   66% full, 13862MB free, 41921MB total
 3:3598      66      139862
#16:52:09 08/02/2010     good    DrjExtranet  URL: wwwsecure.drj.com     0.364 sec    1:3604      200
364  ok 26125   1280782328     0    0   358     4       2       200  0
while(<LOG>) {
  ($time,$date,$status,$group,$monitor,$value) = /(\S+)\s(\S+)\t(\S+)\t(\S+)\t([^\t]+)\t([^\t]+)/;
  print '$time,$date,$status,$group,$monitor,$value' . "$time,$date,$status,$group,$monitor,$value\n" if $DEBUG;
  next if $group =~ /__health__/; # don't care about these lines
  $mons{"$monitor"} = 1;
  push @{$mont{"$monitor"}} , $time;
  push @{$mond{"$monitor"}} , $date;
  push @{$monh{"$monitor"}} , $status;
  push @{$monv{"$monitor"}} , $value;
}
# open output at last moment to minimize chances of reading while locked for writing
open(MONITORSTATS,">$monitorstats") || die "Cannot open monitor stats file $monitorstats!!\n";
open(MONITORSTATSHIS,">$monitorstatshis") || die "Cannot open monitor stats file $monitorstatshis!!\n";
# write it all out - will always print the latest values
foreach $monitor (keys %mons) {
# dereference our anonymous arrays
  @times = @{$mont{"$monitor"}};
  @dates = @{$mond{"$monitor"}};
  @status = @{$monh{"$monitor"}};
  @value = @{$monv{"$monitor"}};
# last element is the latest measured status and value
  print MONITORSTATS "$monitor\t$status[-1]\t$value[-1]\n";
  print MONITORSTATSHIS "$monitor\n";
  #for ($i=-11;$i<0;$i++) {
# put latest measure on top
  for ($i=-1;$i>-13;$i--) {
    $time = defined $times[$i] ? $times[$i] : "NA";
    $date = defined $dates[$i] ? $dates[$i] : "NA";
    $stat = defined $status[$i] ? $status[$i] : "NA";
    $val = defined $value[$i] ? $value[$i] : "NA";
    print MONITORSTATSHIS "\t$time\t$date\t$stat\t$val\n";
  }
}

#!/usr/bin/perl # analyze SiteScope log file # Copyright work under the Artistic License, http://www.opensource.org/licenses/Artistic-2.0 # 8/2010 $DEBUG = 0; $logdir = "/opt/SiteScope/logs"; $monitorstats = "/tmp/monitorstats.txt"; $monitorstatshis = "/tmp/monitorstats-his.txt"; $date = `date +%Y_%m_%d`; chomp($date); $file = "$logdir/SiteScope$date.log"; open(LOG,"$file") || die "Cannot open SiteScope log file: $file!!\n"; # example lines: # 16:51:07 08/02/2010 good LDAPServers LDAP SSL test : ldapsrv.drj.com exit: 0, 0.502 sec 1: 3481 0 502 #16:51:22 08/02/2010 good Network DNS: (AMEAST) ns2 0.033 sec 2:3459 200 33 ok #16:51:49 08/02/2010 good Proxy proxy.pac script on iwww 0.055 sec 2:12467 200 55 ok 4288 1280782309 0 0 55 0 0 200 0 #16:52:04 08/02/2010 good Proxy Disk Space: earth /logs 66% full, 13862MB free, 41921MB total 3:3598 66 139862 #16:52:09 08/02/2010 good DrjExtranet URL: wwwsecure.drj.com 0.364 sec 1:3604 200 364 ok 26125 1280782328 0 0 358 4 2 200 0 while(<LOG>) { ($time,$date,$status,$group,$monitor,$value) = /(\S+)\s(\S+)\t(\S+)\t(\S+)\t([^\t]+)\t([^\t]+)/; print '$time,$date,$status,$group,$monitor,$value' . "$time,$date,$status,$group,$monitor,$value\n" if $DEBUG; next if $group =~ /__health__/; # don't care about these lines $mons{"$monitor"} = 1; push @{$mont{"$monitor"}} , $time; push @{$mond{"$monitor"}} , $date; push @{$monh{"$monitor"}} , $status; push @{$monv{"$monitor"}} , $value; } # open output at last moment to minimize chances of reading while locked for writing open(MONITORSTATS,">$monitorstats") || die "Cannot open monitor stats file $monitorstats!!\n"; open(MONITORSTATSHIS,">$monitorstatshis") || die "Cannot open monitor stats file $monitorstatshis!!\n"; # write it all out - will always print the latest values foreach $monitor (keys %mons) { # dereference our anonymous arrays @times = @{$mont{"$monitor"}}; @dates = @{$mond{"$monitor"}}; @status = @{$monh{"$monitor"}}; @value = @{$monv{"$monitor"}}; # last element is the latest measured status and value print MONITORSTATS "$monitor\t$status[-1]\t$value[-1]\n"; print MONITORSTATSHIS "$monitor\n"; #for ($i=-11;$i<0;$i++) { # put latest measure on top for ($i=-1;$i>-13;$i--) { $time = defined $times[$i] ? $times[$i] : "NA"; $date = defined $dates[$i] ? $dates[$i] : "NA"; $stat = defined $status[$i] ? $status[$i] : "NA"; $val = defined $value[$i] ? $value[$i] : "NA"; print MONITORSTATSHIS "\t$time\t$date\t$stat\t$val\n"; } }

As I said it gets called every minute by cron.

That’s it! I enter the url sitescope.drj.com/SS/ss to access the main program which gets executed because I made /SS a CGI-BIN directory.

This gives you a read-only, Java-free view into your SiteScope status and hierarchy which beckons back to the good old days of Freshwater SiteScope.

Know your limits
What it does not do, unfortunately, is allow you to run a monitor – that seems like the next most simple thing which I should have been able to do but couldn’t figure out – much less define new monitors (never going to happen) or alerts.

I use this successfully against my HP SiteScope instance of roughly 400 monitors which itself is on a VM and there is no apparent strain. At some point this simple-minded script would no longer scale to suit the task at hand, but it might be good for up to a few thousand monitors.

And now a word about open source alternatives
Since I was so enamored with SiteScope Classic there seemed to be no compelling reason to shell out the dough for HP SiteScope with its unwanted interface, so I briefly looked around at free alternatives. Free sounds good, right? Not so much in practice. Out there in Cyberspace there is an enthusiast for a product called Zabbix. I just want to go on the record that Zabbix is the most confused piece of junk I have run across. You are getting less than what you paid for ($0) because you will be wasting a lot of time with it, and in the end it isn’t all that capable. Nagios also had its limits – I can’t remember the exact reason I didn’t go down that route, but there were definite reasons.

HP SiteScope is no panacea. “HP” and “stifling bureaucracy” need to be mentioned in the same sentence. Every time we renew support it is the most confusing mess of line items. Every time there’s a new cast of characters over at HP who nothing about the account’s history. You practically have to beg them to accept your money for a low-budget item like SiteScope because they really don’t pursue it in any way. Then their SAID and contract numbers stuff is confusing if you only see it once every few years.

Conclusion
A conversion program does exist for turning the finicky HP SiteScope Java-encumbered view into pure SiteScope Classic because I wrote it! But it’s a limited read-only view. Still, it’s helpful in a pinch and can even be viewed on the Blackberry’s browser.

Another problem is that HP has threatened to completely change the API so this tool, which is designed for HP SiteScope v 10.12, will probably completely break for newer versions. Oh, well.

References
This post shows some silly mistakes to avoid when doing a minor upgrade in version 11.

Tags HP SiteScope, monitoring, nagios, SiteScope Classic, zabbix

15 replies on “Turning HP SiteScope into SiteScope Classic with Perl”

[…] from 10 years ago began to wane. One of my example Perl scripts is provided in my post about turning HP SiteScope into SiteScope Classic. After deciding I needed a Python project only a few days passed before I came across what I […]

Thanks for this writeup. This was helpful in deciding how to handle a workaround for loosing the .mg files in SiteScope 11. Have you done anything like this with version 11/the updated API?

No, I haven’t yet upgraded to the updated API and so I haven’t seen it.

[…] let’s call it Scattering, was just hard-to-explain slow. Web site accesses as measured in HP SiteScope was slower than at another site, lets call it Cooper, and the variation seemed larger. You could […]

[…] monitoring I like HP SiteScope because it’s more economical than hard-core systems. I wrote a little about it in a previous post. Nagios is also commonly used and it’s open source, meaning free. Avoid Zabbix at all […]

[…] human nature to know that it is easy to ignore the first alert. So I’ve actually programmed HP SiteScope alerts to send additional ones out after four hours of continuous errors. When I got the second one […]

I also love the Classic Sitescope when it was still Freshwater. Do you happen to have a copy of any version before 8.0?!?

I was one of the first SiteScope customers back in the late ’90s during the Freshwater era. The development team as simply awesome and really listened to their customers.

While I haven’t used SiteScope in years, I just installed version 11 and was utterly shocked by the crappy java browser interface.

It’s so sad to see great software die a slow death, especially in a “stifling bureaucracy”.

I know. I remember looking at the Freshwater web site one day. I think I went to the About us page. There was a picture of what must have been their office in Boulder. An aquarium zig-zagged back and forth across the whole open office space, so that each and every developer had a view of fish in front of them all day long! I just laughed and laughed. It was the funniest thing I ever saw. I tried to rib one of them about it later, but they said they liked it. That was Freshwater SiteScope in its essence: likeably quirky and doing great work, before they were corporatized and homogenized by first Mercury Interactive, and then by the behemoth HP. I’m willing to bet that aquarium isn’t around any longer even if they retained some of the original developers from Boulder.
You kept seeing the fish marginalized in the logo until it disappeared altogether, like a dementor was sucking the life spirit out of the company.
But as a shareholder I also see the other side. Someone’s gotta feed the fish. Could Freshwater SiteScope have survived as an independent, with the likes of Nagios and Big Brother eating its lunch? Doubtful. Say what you will about HP, and they do make an easy target, they have been making money, and SiteScope survives…

Hi John,

Really enjoyed the write up! I’m relatively new to SiteScope, I’ve only been using it since early 2011, version 10.10 I think? I guess in a way I’m glad I did not see the web view that came before the Java implementation. It truly is a fussy application, and gives me many headaches. I believe HP will be releasing a web view soon though, so perhaps they are listening to customers and providing this improvement over the heavy Java interface.

Anyway, I have been thinking about building a web interface for some time, just like you have done, and largely in the same way using the XML config dump. But I found the same as you, that much parsing would be necessary, and so have just been dealing with it.

Thanks for your contribution!

~Eric

I have been looking for something like Freshwater Sitescope for years. It was simple to use and it just worked! Why a company needs to buy them out and make the product complicated and slow I don’t understand.

I would settle for a bunch of PHP scripts that did pretty much the same thing.

Hey John,
Great write up! I’m curious if you’ve taken a look at the API at all. It looks like 2 years ago you had not updated to the newest version of the API. Just wondering if you’ve recently done anything with it.
I’m going to use the API to pull information from the SitescopeDaily log, or use something more along the lines of the script above.

I’m now running SS v 11.23, but have had no time to look into the new API..

Thanks for letting me know.
If I use it, I’ll let you know how it goes. I’m starting to think it might be easier to just do something like your solution instead of having to deal with the compressed data.

Hi John and Jim – Just curious if either of you have had the time and/or any luck with this? We would be interested as well but so far haven’t had much much in the way of time or success. Love the idea! And we definitely miss Freshwater and those Think Tanks.

15 replies on “Turning HP SiteScope into SiteScope Classic with Perl”

Leave a Reply Cancel reply