Category: JavaScript

Categories
Admin JavaScript Network Technologies

Practical Zabbix examples

Intro
I share some Zabbix items I’ve had to create which I find useful.

Convert DateAndTime SNMP output to human-readable format
Of course this is not very Zabbix-specific, as long as you realize that Zabbix produces the outer skin of the function:

function (value) {
// DrJ 2020-05-04
// see https://support.zabbix.com/browse/ZBXNEXT-3899 for SNMP DateAndTime format
'use strict';
//var str = "07 E4 05 04 0C 32 0F 00 2B 00 00";
var str = value;
// alert("str: " + str);
// read values are hex
var y256 = str.slice(0,2); var y = str.slice(3,5); var m = str.slice(6,8); 
var d = str.slice(9,11); var h = str.slice(12,14); var min = str.slice(15,17);
// convert to decimal
var y256Base10 = +("0x" + y256);
// convert to decimal
var yBase10 = +("0x" + y);
var Year = 256*y256Base10 + yBase10;
//  alert("Year: " + Year);
var mBase10 = +("0x" + m);
var dBase10 = +("0x" + d);
var hBase10 = +("0x" + h);
var minBase10 = +("0x" + min);
var YR = String(Year); var MM = String(mBase10); var DD = String(dBase10);
var HH = String(hBase10);
var MIN = String(minBase10);
// padding
if (mBase10 < 10)  MM = "0" + MM; if (dBase10 < 10) DD = "0" + DD;
if (hBase10 < 10) HH = "0" + HH; if (minBase10 < 10) MIN = "0" + MIN;
var Date = YR + "-" + MM + "-" + DD + " " + HH + ":" + MIN;
return Date;

I put that javascript into the preprocessing step of a dependent item, of course.

All my real-life examples do not fill in the last two fields: +/-, UTC offset. So in my case the times must be local times. But consequently I have no idea how a + or – would be represented in HEX! So I just ignored those last fields in the SNNMP DateAndTime which otherwise might have been useful.

Here’s an alternative version which calculates how long its been in hours since the last AV signature update.

// DrJ 2020-05-05
// see https://support.zabbix.com/browse/ZBXNEXT-3899 for SNMP DateAndTime format
'use strict';
//var str = "07 E4 05 04 0C 32 0F 00 2B 00 00";
var Start = new Date();
var str = value;
// alert("str: " + str);
// read values are hex
var y256 = str.slice(0,2); var y = str.slice(3,5); var m = str.slice(6,8); var d = str.slice(9,11); var h = str.slice(12,14); var min = str.slice(15,17);
// convert to decimal
var y256Base10 = +("0x" + y256);
// convert to decimal
var yBase10 = +("0x" + y);
var Year = 256*y256Base10 + yBase10;
//  alert("Year: " + Year);
var mBase10 = +("0x" + m);
var dBase10 = +("0x" + d);
var hBase10 = +("0x" + h);
var minBase10 = +("0x" + min);
var YR = String(Year); var MM = String(mBase10); var DD = String(dBase10);
var HH = String(hBase10);
var MIN = String(minBase10);
var Sigdate = new Date(Year, mBase10 - 1, dBase10,hBase10,minBase10);
//difference in hours
var difference = Math.trunc((Start - Sigdate)/1000/3600);
return difference;

Calculated bandwidth from an interface that only provides byte count
Again in this example the assumption is you have an item, probably from SNMP, that lists the total inbound/outbound byte count of a network interface – hopefully stored as a 64-bit number to avoid frequent rollovers. But the quantity that really excites you is bandwidth, such as megabits per second.

Use a calculated item as in this example for Bluecoat ProxySG:

change(sgProxyInBytesCount)*8/1000000/300

Give it type numeric, Units of mbps. sgProxyInBytesCount is the key for an SNMP monitor that uses OID

IF-MIB::ifHCInOctets.{$INTERFACE_TO_MEASURE}

where {$INTERFACE_TO_MEASURE} is a macro set for each proxy with the SNMP-reported interface number that we want to pull the statistics for.

The 300 in the denominator of the calculated item is required for me because my item is run every five minutes.

Alternative
No one really cares about the actual total value of byte count, right? So just re-purpose the In Bytes Count item a bit as follows:

  • add preprocessing step: Change per second
  • add second preprocessing step, Custom multiplier 8e-6

The first step gives you units of bytes/second which is less interesting than mbps, which is given by the second step. So the final units are mbps.

Be sure to put the units as !mbps into the Zabbix item, or else you may wind up with funny things like Kmbps in your graphs!

Creating a baseline

Even as of Zabbix v 5, there is no built-in baseline item type, which kind of sucks. Baseline can mean many different things to many people – it really depends on the data. In the corporate world, where I’m looking at bandwidth, my data has these distinct characteristics:

  • varies by hour-of-day, e.g., mornings see heavier usage than afternoons
  • there is the “Friday effect” where somewhat less usage is seen on Fridays, and extremely less usage occurs on weekends, hence variability by day-of-week
  • probably varies by day of month, e.g., month-end closings

So for this type of data (except the last criterion) I have created an appropriate baseline. Note I would do something different if I were graphing something like the solar generation from my solar panels, where the day-of-week variability does not exist.

Getting to the point, I have created a rolling lookback item. This needs to be created as a Zabbix Item of type Calculated. The formula is as follows:

(last(sgProxyInBytesCount,#1,1w)+
last(sgProxyInBytesCount,#1,2w)+
last(sgProxyInBytesCount,#1,3w)+
last(sgProxyInBytesCount,#1,4w)+
last(sgProxyInBytesCount,#1,5w)+
last(sgProxyInBytesCount,#1,6w))/6

In this example sgProxyInBytesCount is my key from the reference item. Breaking it down, it does a rolling lookback of the last six measurements taken at this time of day on this day of the week over the last six weeks and averages them. Voila, baseline! The more weeks you include the more likely you are to include data you’d rather not like holidays, days when things were busted, etc. I’d like to have a baseline that is from a fixed time, like “all of last year.” I have no idea how. I actually don’t think it’s possible.

But, anyway, the baseline approach above should generally work for any numeric item.

Refinement

The above approach only gives you six measurements, hence 1/sqrt(6) ~ 40% standard deviation by the law of large numbers, which is still pretty jittery as it turns out. So I came up with this refined approach which includes 72 measurements, hence 1/sqrt(72) ~ 12% st dev. I find that to be closer to what you intuitively expect in a baseline – a smooth approximation of the past. Here is the refined function:

(avg(sgProxyInBytesCount,1h,1w)+
avg(sgProxyInBytesCount,1h,2w)+
avg(sgProxyInBytesCount,1h,3w)+
avg(sgProxyInBytesCount,1h,4w)+
avg(sgProxyInBytesCount,1h,5w)+
avg(sgProxyInBytesCount,1h,6w))/6

I would have preferred a one-hour interval centered around one week ago, etc., e.g., something like 1w+30m, but such date arithmetic does not seem possible in Zabbix functions. And, yeah, I could put 84600s (i.e., 86400 – 1800), but that is much less meaingful and so harder to maintain. Here is a three-hour graph whose first half still reflects the original (jittery) baseline, and last half the refined function.

Latter part has smoothed baseline in light green

What I do not have mastered is whether we can easily use a proper smoothing function. It does not seem to be a built-in offering of Zabbix. Perhaps it could be faked by a combination of pre-processing and Javascript? I simply don’t know, and it’s more than I wish to tackle for the moment.

Data gap between mulitple item measurements looks terrible in Dashboard graph – solution

In a Dashboard if you are graphing items which were not all measured at the same time, the results can be frustrating. For instance, an item and its baseline as calculated above. The central part of the graph will look fine, but at either end giant sections will be missing when the timescale of display is 30 minutes or 60 minutes for items measured every five minutes or so. Here’s an example before I got it totally fixed.

Zabbix item timing mismatch

See the left side – how it’s broken up? I had beguin my fix so the right side is OK.

The data gap solution

Use Scheduling Intervals in defining the items. Say you want a measurement every five minutes. Then make your scheduling interval m/5 in all the items you are putting on the same graph. For good measure, make the regular interval value infrequent. I use a macro {$UPDATE_LONG}. What this does is force Zabbix to measure all the items at the same time, in this case every five minutes on minutes divisible by five. Once I did that my incoming bandwith item and its corresponding baseline item aligned nicely.

Low-level Discovery

I cottoned on to the utility of this part of Zabbix a little late. Hey, slow learner, but I eventually got there. What I found in my F5 devices is that using SNMP to monitor the /var filesystem was a snap: it was always device 32 (final OID digit). But /var/log monitoring? Not so much. Every device seemed different, with no obvious pattern. Active and standby units – identical hardware – and some would be 53, the partner 55. Then I rebooted a device and its number changed! So, clearly, dynamically assigned and no way was I going to keep up with it. I had learned the numbers by doing an snmpwalk. The solution to this dynamically changing OID number is to use low-level discovery.

Conclusion
A couple of really useful but poorly documented items are shared. Perhaps more will be added in the future.


References and related
https://support.zabbix.com/browse/ZBXNEXT-3899 for SNMP DateAndTime format

My first Zabbix post was mostly documentation of a series of disasters and unfinished business.

Blog post about calculated items by a true expert: https://blog.zabbix.com/zabbix-monitoring-with-calculated-items-explained/9950/

Low-level Discovery write-up: https://blog.zabbix.com/how-to-use-zabbix-low-level-discovery/9993/

Categories
JavaScript Linux Perl

A simple Perl script to build JavaScript folder objects

Part 2
Intro
This is the 2nd part in a two-part blog where I present a simple example of a JavaScript folder browser. In Part 1 I provided all the JavaScript required. By itself it may have seemed an academic exercise, but once you appreciate that it isn’t hard to write a program which creates the JavaScript objects from your server’s directory structure, well, now you have something that’s pretty powerful and useful.

The details
I considered writing this in Python, which seems to be a more current language, but old habits die hard as they say. I just know Perl too well to suffer the pain of learning all those neat tricks all over again in another language. Maybe someday I’ll re-write it in Python.
Notice the recursion through the directories? I first used that 17 years ago! Why throw out good code?

Here is the code, which I named scan.pl:

#!/usr/bin/perl
# DrJ - 7/2012
# scan picture-containing directories using recursion and build javascript objects from them
use Getopt::Std;
getopts('d:j:');
$homedir = $opt_d;
$jsfile = $opt_j;
usage() if ! $opt_d || ! $opt_j;
$DEBUG = 0;
print "Homedir: $homedir, jsfile: $jsfile\n";
open(JS,">$jsfile") || die "Cannot open JavaScript file: $jsfile!!\n";
$date = `date +%D`;
($homedirnoslash) = $homedir =~ /^\/(.+)/; # assumes leading "/"
 
# print opening of function
print JS qq#function init() {
// Generated data from scan.pl - DrJ $date
folder['browse'] = {path:'',depth:0,kids:['$homedirnoslash']};
#;
 
# get things going with our recursive function
traverse($homedir,0);
 
# closing statement
print JS qq(}\n);  # close of init JavaScript function
close(JS);
 
sub traverse {
my ($dir,$depth) = @_;
my @kids = ();
print "Traverse. dir: $dir\n" if $DEBUG;
opendir(DIR, $dir) || die "Cannot open dir $dir!!\n";
foreach (readdir(DIR)) {
  next if $_ eq '.' || $_ eq '..';
  print "Traverse. file: $_\n" if $DEBUG;
  $path = "$dir/$_";
  if (-d $path) {         # a directory
# we want only the last part of the path
    (my $lastpath) = $path =~ /([^\/]+)$/;
    push(@kids,$lastpath);
    traverse($path,$depth + 1); # recurse!
  } elsif ($_=~/$filespec/) {        #
  }
} # end loop over files in this directory
# write out the JS objects
print JS qq(folder["$dir"] = {path:"$dir",depth:$depth,kids:[);
my $i = 0;
# kids are in jumbled order.  Do regular sort on them.
foreach (sort @kids) {
  $comma = $i++ > 0 ? "," : "";
  print JS qq($comma"$_");
}
# end of object. close it out.
print JS qq(]};\n);
} # end sub traverse
#
sub usage {
  print "usage: $0 -d root_directory -j JavaScript_output_file\n";
  exit(1);
}

It’s pretty self-explanatory. Call it like this example:

> ./scan.pl -d /homepic -j init.js

and it produces an init.js file filled with an init() function and all the necessary folder objects, assuming the top-level folder to browse is /homepic.

My init.js looks like this:

function init() {
// Generated data from scan.pl - DrJ 07/27/12
 
folder['browse'] = {path:'',depth:0,kids:['homepic']};
folder["/homepic/pictures_chronological/2011_06"] = {path:"/homepic/pictures_chronological/2011_06",depth:2,kids:[]};
// lots more lines like this omitted
folder["/homepic"] = {path:"/homepic",depth:0,kids:["kodak_pictures","pictures_chronological"]};
}

And I tested it in browse13.html, which looks just like browse12.html, except I got rid of the init() function and added an include line at the top:

<html>
<head>
<script type="text/javascript" src="init.js"></script>
...

I am a little concerned about performance. This clearly isn’t designed to scale to tens of thousands of directories, but will it be sufficiently fast for my purposes? My init.js is 213 lines and about 25 KB in size. browse13.html which calls it loads fast and runs fast. So, yes, success!

Part 1, A Simple Javascript Folder Browser

Conclusion
We have created a fairly powerful and general-purpose folder browser out of fairly simple usage of JavaScript and Perl. It makes an ideal base upon which to build further.

Categories
Apache JavaScript Perl

A Simple Javascript Folder Browser

Part 1

Intro
I haven’t posted much lately. I’ve been tied up creating this folder browser using client-side JavaScript. I probably made every mistake in the book, but I worked through them all and the outcome is pretty cool, if I say so myself! It works in IE, FireFox and even Blackberry!

The details
I broke down the development of this browse app into 12 stages. Given time, I might show the progression of my thinking as each version becomes closer and closer to fulfilling all initial objectives. But who has time? I’ll show the source for browse3.html, warts and all, and then skip many iterations and jump to showing the final source, browse12.html.

Browse3.html
It ain’t pretty. It isn’t even correct. But it “does stuff.” It assumes Apache web server is running and “borrows” the closed and open folder icons from apache’s /icons directory. In my case I have a top-level directory called /homepic with folders under that and sub-folders under those folders that I want the ability to browse and , ultimately, take some action such as displaying all the folder’s images in the image viewer I wrote earlier.

<!DOCTYPE html>
<html>
<head>
<script type="text/javascript">
// global object
var folder = new Object;
function displayDate()
{
document.getElementById("demo").innerHTML=Date();
}
function init() {
// big initialization - generated code from perl perusal of directories
// see http://www.quirksmode.org/js/associative.html
//  var folder = new Object;
// we need this empty assignment to extend object with subproperties later on
  folder['homepic'] = '/homepic';
  //folder['/homepic'].path = '/homepic';
  folder.homepic.state = 'closed';
  folder[0] = '/homepic/kodak_pictures';
  folder[1] = '/homepic/pictures_chronological';
  folder['/homepic/kodak_pictures'] = '/homepic/kodak_pictures';
  folder['/homepic/kodak_pictures'].state = 'closed';
  //folder['/homepic/kodak_pictures'].path = '/homepic/kodak_pictures';
  folder['/homepic/pictures_chronological'] = '/homepic/pictures_chronological';
  folder['/homepic/pictures_chronological'].state = 'closed';
  //folder['/homepic/pictures_chronological'].path = '/homepic/pictures_chronological';
  var child = 'homepic';
  var cstate = folder['homepic'].state;
  var cpath = folder[0];
}
function browse(f) {
document.getElementById("demo").innerHTML="folder: "+f+" ";
var icon;
var imgid;
var fname;
if (f == "browse") {
// see http://www.quirksmode.org/dom/intro.html#create
// one-time initialization
  init();
  document.getElementById("browse").innerHTML='';
  icon = document.createElement('IMG');
  icon.src = "/icons/folder.gif";
  icon.id = "icon-/homepic";
  icon.onclick = Function('browse("/homepic")');
  document.getElementById("browse").appendChild(icon);
  var divfolder = document.createElement('DIV');
  divfolder.id = "/homepic";
  document.getElementById("browse").appendChild(divfolder);
  var x = document.createTextNode('homepic');
  document.getElementById('/homepic').appendChild(x);
  //document.getElementById("browse").innerHTML='<img id="homepic" onclick="browse(\'homepic\')" src="/icons/folder.gif">ho
mepic<br>';
} else {
  imgid = 'icon-' + f;
// change img to one of open folder
  document.getElementById(imgid).src = "/icons/folder.open.gif";
  // nope? imgid.src  = "/icons/folder.open.gif";
  for (i=0;i<3;i++) {
    icon = document.createElement('IMG');
    icon.src = "/icons/folder.gif";
    fname = folder[f].children[i];
    //fname = folder['homepic'].children[i];
    icon.id = "icon-" + fname;
    icon.onclick = Function('browse('+fname+')');
    document.getElementById(f).appendChild(icon);
    var divfolder = document.createElement('DIV');
    divfolder.id = fname;
    document.getElementById(f).appendChild(divfolder);
    var x = document.createTextNode(fname);
    document.getElementById(f).appendChild(x);
  } // end loop over children
}
}
// note variable # or arguments are being passed
function  openfolder()
{
var f = arguments[0];
var dir = "/icons/";
var fopen = "folder.open.gif";
var fclosed = "folder.gif";
var folder = document.getElementById(f);
var ftype = folder.src;
// src includes http... Get rid of stuff in front
var patt = /.*(folder.+)/;
var ftypebare = ftype.replace(patt,"$1");
// for debugging
document.getElementById("demo").innerHTML="folder: "+f+" "+ftypebare;
if (ftypebare == fopen) {
// close folder and remove sub-folders
  folder.src = dir+fclosed;
  document.getElementById("homepic/pictures_chronological").innerHTML='';
} else {
// open up folder and reveal sub-folders
  folder.src = dir+fopen;
  for (var i = 0; i < arguments.length; i++) {
    var placeid = arguments[i];
    var srcid = '/' + placeid;
    document.getElementById(placeid).innerHTML='&nbsp;&nbsp;&nbsp;<img id="' + srcid + '" onclick="openfolder(\'pictures_ch
ronological\')" src="/icons/folder.gif"/> pictures_chronological<br>';
  }
}
 
}
</script>
</head>
<body>
 
<h1>Folder Browser</h1>
<p id="demo">Debug Aid.</p>
<p id="browse"><a href="#" onclick='browse("browse")'>Folder Browser</a></p>
 
<img id="homepic" onclick="openfolder('homepic','homepic/pictures_chronological','homepic/canon_pictures')" src="/icons/fol
der.gif"/> homepic<br>
<div id="homepic/pictures_chronological"></div>
<div id="homepic/canon_pictures"></div>
<img id="cfolder2" onclick="openfolder(2)" src="/icons/folder.gif"/><br>
<img id="cfolder3" onclick="openfolder(3)" src="/icons/folder.gif"/><br>
<img id="cfolder4" onclick="openfolder(4)" src="/icons/folder.gif"/><br>
 
</body>
</html>

So you see in browse3 I’m wrestling with how to work with JavaScript Objects (which I didn’t really know existed at that time). I badly wanted to give an associative array properties, as in the initialization line

folder['/homepic/kodak_pictures'].state = 'closed';

but I couldn’t find any examples on the Internet. I eventually learned that wasn’t a correct assignment.

So one of my biggest and most worthwhile lessons was to gain a decent understanding of JavaScript objects, which hold multiple values, and object properties.

I tried to use Firebug for Firefox, with very limited success, but at least I could step through the Javascript code and see which branch in a conditional was being executed compared to what I thought should be executed, which tipped me off to one vexing problem concerning opening and closing folders multiple times. Also just looking up the error in Internet Explorer by double-=clicking the warning sign in the corner was tremendously helpful.

So…skipping for now versions 4 – 11, we arrive at browse12.html:

Browse12.html

<!DOCTYPE html>
<html>
<head>
<script type="text/javascript">
// global object
var folder = new Object;
function displayDate()
{
document.getElementById("demo").innerHTML=Date();
}
function init() {
// big initialization - generated code from perl perusal of directories
// I think my big breakthrough was to learn about Javascript objects and their properties
// from the book Javascript The Definitive Guide, 5th edition by D.Flanagan
// Some additional inspiration came from http://www.quirksmode.org/dom/intro.html#create
// folder is an associative array with properties path, state,depth and kids (which is itself an array)
// first entry is initial top-level to get things started
  folder['browse'] = {path:'',depth:0,kids:['homepic']};
// regular entries:
  folder['/homepic'] = {path:'/homepic',depth:1,kids:['kodak_pictures','pictures_chronological']};
// values for sub-folders
// /homepic/kodak_pictures
  folder['/homepic/kodak_pictures'] = {path:'/homepic/kodak_pictures',depth:2,kids:['2002_05','2002_06']};
// /homepic/pictures_chronological
  folder['/homepic/pictures_chronological'] = {path:'/homepic/pictures_chronological',depth:2,kids:['2011_11','2011_12']};
}
function browse(f) {
document.getElementById("demo").innerHTML="folder: "+f+" ";
var icon;
var imgid;
var fname;
var table;
if (f == "browse") {
// one-time initialization
  init();
}
 
imgid = 'icon-' + f;
if (!folder[f]) {
// folder not defined: must be a terminal folder. Do something here eventually
// but for now do nothing whatsoever
} else {
  if (folder[f].state === undefined || folder[f].state == 'closed') {
  // change img to one of open folder
    if (f == 'browse') {
      document.getElementById(f).innerHTML='';
    } else {
      document.getElementById(imgid).src = "/icons/folder.open.gif";
    }
    var kids = folder[f].kids;
    for(var i=0;  i < kids.length; i++) {
      table = document.createElement("table");
      table.border = 0;
      var row = document.createElement("tr");
      var td1 = document.createElement("td");
      var td2 = document.createElement("td");
      var td3 = document.createElement("td");
      icon = document.createElement("img");
      icon.src = "/icons/folder.gif";
      if (f == "browse") {
        fname = "/" + kids[i];
      } else {
        fname = f + "/" + kids[i];
      }
      icon.id = "icon-" + fname;
      if (folder[fname])
        folder[fname].state = 'closed';
      icon.onclick = Function('browse("'+fname+'")');
      td1.width = 27 + folder[folder].depth*27;
      td1.align = "right";
      td1.appendChild(icon);
      var node = document.createTextNode(kids[i]);
      td3.appendChild(node);
      row.appendChild(td1); row.appendChild(td2); row.appendChild(td3);
      table.appendChild(row);
      document.getElementById(f).appendChild(table);
      var divfolder = document.createElement("div");
      divfolder.id = fname;
      document.getElementById(f).appendChild(divfolder);
      folder[f].state = 'open';
    } // end loop over children
  } else {
  // set folder to closed state
  // this innerHTML nullification is kind of a kludge, but it works
    document.getElementById(f).innerHTML='';
    document.getElementById(imgid).src = "/icons/folder.gif";
    folder[f].state = 'closed';
  } // end conditional over folder state
} // end condition over whether folder is defined or not
} // end function browse
 
</script>
</head>
<body>
 
<h1>Folder Browser</h1>
<p id="demo">Debug Aid.</p>
<p id="browse"><a href="#" onclick='browse("browse")'>Folder Browser</a></p>
</body>
</html>

That’s it! Not bad, huh? I pan to generate the init() function periodically and automatically from a Perl script which peruses my directories.

I could have used Ajax and generated the subfolder information on the fly as it is needed – not that I know how, I just know enough to know it is possible and therefore I could do it – but I thought this method of pre-loading all the information might be a little more efficient. If this were a folder and file browser it would be different, but for now it is just a folder browser.

So the main revelation is that I had to set my associative array members to be objects during initialization, as in

folder['/homepic'] = {path:'/homepic',depth:1,kids:['kodak_pictures','pictures_chronological']};

and one of the object values is itself an anonymous array that holds the sub-folders.

Buying an actual book was probably a good move. I went with JavaScript, The Definitive Guide, 5th edition. Note that this is not the latest edition – the 6th – but this way I could buy the book used for a lot less and not get myself further confused by HTML5, which I am not ready to tackle and which many browsers do not yet fully support. The book is pretty heavy going and the discussion of DOM was particularly difficult and the examples too few and too removed from the real world. But the Core Javascript discussion made a lot of sense to me so was by itself worth the purchase price.

In my next post I’ve posted the Perl script which can generate the folder object initialization.

Part 2, A simple Perl script to build JavaScript folder objects

Conclusion
In this part one of a two-part post I’ve provided the JavaScript that implements a very compact folder browser. It has been tested on both IE and Firefox. The 2nd part of this series will provide the Perl Jvascript code generator for automation of the object creation.