Categories
CentOS Debian Linux Raspberry Pi

drjohnstechtalk now runs on a modern OS

Intro

I’m thrilled to announce that the long-running blog drjohnstechtalk.com has now been migrated to a modern back-end operating system. drjohnetchtalk.com is, a far as I know, the only quality-written technical resource on the Internet which is not supported by ads. Instead it runs on a pay-it-forward approach, embracing the spirit of the old Internet before it was ruined by big money.

drjohnstechtalk.com has been providing solutions to obscure tech questions since 2011.

The details

I like to run my own server which I can use for other purposes as well. I think that approach used to be more common. Now it’s harder to find others using it. Anyway, my old hosting environment is a CentOS server. I had hoped it would last me up to 10 years! 10 years is about the duration of long-term support for Redhat linux. It’s a real pain to migrate a WordPress blog with lots of history where it is important to preserve the articles and the permalinks. This article documents the nightmare I put myself through to get that up and running. Before that there was a CentOS 6 server. Then in 2022 – only about two years later – I learned that CentOS was dead! IBM had killed it. I’m over-simplifying here somewhat, but not by much.

So my blog sort of limped on on this unsupported system, getting riskier by the day to run as I was missing out on security patches. Then my companyt accidentally included one of my blogs in a security scan and I saw I had some vulnerabilities. So I upgraded WordPress versions and plugin versions. So with up to date software, the stage was set to migrate to a newer OS. Further motivation was provided by the fact that after the WP upgrade, the pages loaded more slowly. And sometimes the site just collapsed and crashed.

I have come to love Debian linux due to my positive experience with running it on Raspberry Pis and a few other places. It tends to run more recent versions of open source software, for instance. So I chose a Debian linux server. Then I forget where I learned this. Perhaps I asked someone at work which web server to use, but the advice was to use nginx, not apache! This was very new to me as I had never run nginx, not that I was in love with apache.

So, anyway, here I am writing this on my shiny new Debian 12 bookworm server which is running an nginx web server! And wow my site loads so much faster now. It’s really striking…

Running WordPress in a subdirectory with nginx

There always has to be a hard part, right? This was really, really hard. I run WP in the subdirectory blog as you can see from any of my URLs. I must have scoured a dozen sites on how to do it, none of which completely worked for me. So I had to do at least some of the heavy lifting and work out a working config on my own.

Here it is:

# mostly taken from https://www.nginx.com/resources/wiki/start/topics/recipes/wordpress/
# but with some important mods
upstream php {
    server unix:/var/run/php/php8.2-fpm.sock;
}

server {
  listen 443 ssl;

    include snippets/self-signed.conf;


    server_name drjohnstechtalk.com www.drjohnstechtalk.com;

    root /web/drjohns;
    index index.php index.html;


    access_log /var/log/nginx/drjohns.access.log;
    error_log /var/log/nginx/drjohns.error.log;

    client_max_body_size 100M;

# the following section prevents wp-admin from infintely redirecting to itself!
    location /blog/wp-admin {
            root /web/drjohns;
            try_files $uri $uri/ /blog/wp-admin/index.php?$args;
    }

    location /blog {
            root /web/drjohns/blog;
            try_files $uri $uri/ /blog/index.php?$args;
    }
    location ~ \.php$ {
#NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
         include fastcgi_params;
         fastcgi_intercept_errors on;
         fastcgi_pass php;
         fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    }
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg) {
            expires max;
            log_not_found off;
    }
}

I had to add ths svg file type to ignore, the location directive that matches /blog/wp-admin/. I had to define the upstream label as php and refer to that label in fastcgi_pass. I had to figure out my correct version of fastcgi. I tossed out some location directives which weren’t too important to me.

I disabled the wp-hide-login plugin while I grappled with why I was getting first a 404 not found for /blog/wp-admin/, then later, the too many redirects error. But I still had the issue with it disabled. Once I resolved the problem by adding the /blog/wp-admin location directive – I seem to be the only one on the Internet offering this solution and no other solution worked for me! – then I re-enabled the hide login plugin. The other plugins are working I would say.

Firewall?

I gather the current approach to host-based firewall on Debian 12 is to run ufw. A really good article on setting it up is here: https://www.cyberciti.biz/faq/set-up-a-firewall-with-ufw-on-debian-12-linux/

I’m on the fence about it, fearing it might slow my speedy server. But it looks pretty good. So for now I am relying on AWS Network Security Group rules. Did you know you can ask them to increase your max rule quota frmo 20 to 40? Yes, you can. I did and got approved overnight. I have added the Cloudflare ranges.

Cloudflare

I continue to use Cloudflare as reverse proxy, certificate issuer, DNS provider and light security screening. The change to the new server did not alter that. But I needed a new config file to properly report the origin IP address in my access files. The following file does the trick for me. It is up to date as of February 2024, can be placed in your /etc/nginx/conf.d directory and called, e.g., cloudlfare.conf.

# up to date as of 2/2024
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 104.16.0.0/13;
set_real_ip_from 104.24.0.0/14;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 131.0.72.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 2400:cb00::/32;
set_real_ip_from 2606:4700::/32;
set_real_ip_from 2803:f800::/32;
set_real_ip_from 2405:b500::/32;
set_real_ip_from 2405:8100::/32;
set_real_ip_from 2c0f:f248::/32;
set_real_ip_from 2a06:98c0::/29;

real_ip_header CF-Connecting-IP;

The idea is that if the source IP of the HTTP connection to nginx is from the Cloudflare range of IPs, then this must represent a request proxied through Cloudflare and the original IP of the client is in the HTTP header CF-Connecting-IP, which nginx can report on. If not, just use the normal IP from the TCP connection.

Swap space

On CentOS I had to provide some swap space because otherwise apache + mariaDB + WordPress would easily send its cpu soaring. So far I have not had to do that with my new Debian 12! That is great… So I have a t2.small instance with 25 GB of gp2 storage (100 iops). The server is basically running with a 0.00 load average now. I don’t get a lot of traffic so I hope that infrastructure will suffice.

Set the timezone

My Debian system started out in the UTC timezone. This command confirms that:

sudo timedatectl

This command brings up a menu and i can change the timezone to US Eastern:

sudo dpkg-reconfigure tzdata

Automate patching

It hasn’t run yet, but I’m hoping this root crontab entry will automate the system updates:

59 2 * * 0 (date && apt-get update && apt-get upgrade -y) >> /home/admin/hosting/update.log 2>&1

Debian 12 lifecycle

There should be three years of full support plus two more years of long term support for a stable Debian release, if I’ve undrstood it correctly. So I believe I may hope to get five years out of my Bookworm version, give or take. Debian — Debian Releases

Fixing the vi editor

I’ve never really had a problem with vi until this server. I show how I fixed it in this blog post.

Status after a few days – not all positive news

Well after a few days I feel the server response has noticeably slowed. I could not run top because I messed up the terminal with my fix to vi! So in a panic I restarted mariadb which seemed to help performance a lot. I will have to figure out how to monitor for this problem and how best to address it. I’m sure it will return. Here is my monitor.sh script:

#!/bin/bash
# restart mariaDB if home page response becomes greater than one second
curl -m1 -o /dev/null -ksH 'Host:drjohnstechtalk.com' https://localhost/blog/
# if curl didn't have enough time (one sec), its exit status is 28
[ $? -eq 28 ] && (systemctl stop mariadb; sleep 3; systemctl start mariadb; echo mariadb restart at $(date))

I invoke it from root’s crontab every three minutes:

# check that our load time is within reason or else restart mariadb -DrJ 2/24
*/3 * * * * sleep 25;cd /home/admin/hosting; ./monitor.sh >> monitor.log 2>&1

I do love my kludges. I will be on the lookout for a better long-term solution.

Conclusion

The technical blogging web site drjohnstechtalk.com now runs on new infrastructure: Debian 12 running nginx. It is muich faster than before. The migration was moderately painful! I have shared the technical details on how I managed to do it. I hope that, unlike my previous platform of CentOS 8, this platform lasts me for the next 10 years!

References and related

My second article!

nginx’s own advice about how to configure it to run WordPress

Trying to upgrade WordPress brings a thicket of problems

One of many RPi projects of mine: Raspberry Pi light sensor project

ufw firewall for Debian 12

Debian — Debian Releases

Cloudflare, an added layer of security for your web site

IP Ranges | Cloudflare

What happened to insert mode on the latest version of vi?

Categories
Linux Raspberry Pi

Raspberry Pi light sensor project

Intro

I ultimately want to turn off the connected display when it is nighttime and the lights have been turned off. And I want it to turn itself back on during daylight. The reason is because my RPi-driven slideshow is running and somoeone may be sleeping in that room.

This is till a work in progress. Pardon our dust and gibberish. What I’ve already found out is too important to delay publication.

Equipment and skills
How to turn off and on an HDMI port with a Raspberry Pi 4

Honestly this is the most significant thing I have found in this investigation. The methods used for older RPis do not work! In other words you can run vcgencmd display_power 0 and clever variations of that command until you’re blue in the face and the thnig stubbornly won’t turn off. tvservice -o ? Nope. That’ll suggest that command was deprecated and use kmsprint instead.

But I can say as of this writing (Jan ’24) the kms* commands are not mature and do not permit you to turn off the display. kmsprint tells you some stuff, but it does not allow you to set things. Remember RPi is for the education and hobbyist crowd so we have to give them the slack to experiment and try new things, even when they aren’t fully formed.

Instead they give you a way to restore the old commands. Edit the file /boot/config.txt.

Make it look like this:

#Enable DRM VC4 V3D driver
#use the fake kms driver in place of the native kms driver so we can  control hdmi power -DrJ 1/24
##dtoverlay=vc4-kms-v3d
dtoverlay=vc4-fkms-v3d

Reboot. Congrats you are now using the fake kms driver (fkms) and now have compatibility with the old commands. But instead of using tvservice, for my purposes, I think vcgencmd is better because the frame buffer state is not lost.

So now this command will indeed turn off the display:

vcgencmd display_power 0 # turn hdmi display off

vegcmd display_power 1 # turns the HDMI display back on

vcgencmd display_power 0 # turn hdmi display off

Prepare for our light sensor

I really don’t know if I’ll ever get the light sensor to work or not. Anticipating that it will, I have created this GPIO callback routine in python which in my dreams will turn off the HDMI display when the room is dark and turn it back on when the ambient light crosses a threshold. Who knows… But the code is pretty cool because it permits you to play with it as well using software command you send to a GPIO pin.

I call it gpio_basic.py:

import RPi.GPIO as GPIO
import time
import os
import datetime

GPIO.setmode(GPIO.BCM)
channel=4
GPIO.setup(channel, GPIO.IN)
reading = GPIO.input(channel)
print('Initial Reading',reading,flush=True)
old_reading = reading
while True: # infinite loop
    time.sleep(4)
# rising means ambient light went from light to dark
    reading = GPIO.input(channel) # 1 => dark, D0 LED turns off; 0 if light
    if reading == old_reading: continue

# else section where the state has changed
    print('This is a change in state on channel',channel,'at ',datetime.datetime.now(),flush=True)
    print('Reading',reading)
    if reading == 1:
        print('Turning off the HDMI display...',flush=True)
        os.system("vcgencmd display_power 0")
    else:
        time.sleep(6)
        print('Turn on HDMI display...',flush=True)
        os.system("vcgencmd display_power 1")
    old_reading = reading

Now to play with it, in another window get into python and run these commands:

import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)
channel=4
GPIO.setup(channel, GPIO.IN)
GPIO.input(channel)
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_UP)
#or
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

Those commented out lines at the end are the key ones. By executing them you should see the display turn off or on, and your other program should output some verbiage. Now hopefully the sensor will work something like that!

Wiring

Don’t know if this will pan out. I envision wiring (RPi pin #’s on the left):

  • 1 – Vcc (+ 3.3 V voltage)
  • 6 – Gnd (ground)
  • 7 – D0 (digital out)

At this point I don’t see the point of wiring the fourth lead on the diode sensor (which is A0, analog out – the RPi cannot read analog out), but having only three of them wired doesn’t quite seem right either. My electronics knowledge is weak!

So maybe I’ve wasted my money on these sensors, or I need to read them with a microcontroller and use i2c to talk to the RPi (too much effort and too much expense for my taste), but I hope not. Will know soon…

My first sensor, the photodiode, works! But its threshold is near one end of the control, which isn’t so great. But it’s fun to play with. In other words, its low light sensitivity may not be adequate for my purposes where I need to distinguish low ambient light (in a room with only a glowing TV screen, for instance) from even lower ambient light (lights off). Turns out our human eyes are the best measuring devices! Actually this photo diode, properly tuned, is quite good! I believe there is some jitter in the measurements however. So it can jump around during low light a bit. I have to consider how much of a problem that is.

The photodiode has a power led and a light sensor led. They are both way too bright. I suspect they could even create a feedback loop. I covered both with masking tape, leaving room for the adjustment screw.

But much, much worse than the the photodiode is the photoresistor. That at best distinguishes between a decent amount of light, and quite dark. But the transition between the two is sticky (I believe that would be called hysteresis). It will not work at all for my purposes based on my initial testing. It cannot distinguish between low light and very low light no matter where the dial is set.

Since this is all working, we just need to make it permanent by starting at boot time. So in my crontab file I added this line:

# Turn monitor off and on depending on ambien light! - DrJ 1/9/24
@reboot sleep 42; python3 gpio_basic.py > gpio_basic.log 2>&1
RPi4 shown with the slick metal case and the photodiode during daytime
Measure temperature of the CPU and why it matters

vcgencmd measure_temp

With my new fancy aluminum, heat-dissipating case I get around 40.4° C. On my RPi 3, air-cooled it is around 49° C. Why the sudden concern around heat? I’m just beginning to suspect that you know those times when you use the command line and things just seem to freeze? I always just assumed it was a glitch in the WiFi. But maybe it was actually the cpu getting too hot and having to pause itself to avoid burning up. I would see this when transferring files on my RPi 4. so the RPi 4 probably really does run hot unless you take steps to cool it, and the aluminum case in the equipment list above is really cool, ha , ha.

Tips

The command pinout is very useful. Pinout docs here: https://www.raspberrypi.com/documentation/computers/raspberry-pi.html

References and related

This guide talks about explicit support for generic digital input devices, not just a button: https://randomnerdtutorials.com/raspberry-pi-digital-inputs-python/

This talks about built-in pull-up pull-down resistors configurable via software! Who knew?

This might be useful for simple GPIO stuff: https://tutorials-raspberrypi.com/raspberry-pi-gpio-explanation-for-beginners-programming-part-2/

Gory details with gory circuit diagram: https://www.circuits.dk/everything-about-raspberry-gpio/

Running multiple RPi slideshows

How to deal with th GPIO pins using python: https://sourceforge.net/p/raspberry-gpio-python/wiki/Inputs

This project used the RPi GPIO OUTPUT pins to control a power relay device

Appendix A

I started with this code, gpio_callback.py, but the one condition kept getting called when the GPIO pin was reading 1. So I wrtoe gpio_basic.py and use that instead.

import RPi.GPIO as GPIO
import time
import os

GPIO.setmode(GPIO.BCM)
channel=4
GPIO.setup(channel, GPIO.IN)
reading = GPIO.input(channel)
print('Initial Reading',reading)
def my_callback(channel):
# rising means ambient light went from light to dark
    print('This is a edge event callback function!')
    print('Edge detected on channel %s'%channel)
    print('This is run in a different thread to your main program')
    print('Gonna stop that slideshow now...')
    reading = GPIO.input(channel) # 1 => dark, D0 LED turns off; 0 if light
    print('Reading',reading)
    if reading == 1:
        time.sleep(1)
        print('Turning off the HDMI display...')
        os.system("vcgencmd display_power 0")
    else:
        time.sleep(1)
        print('Turn on HDMI display...')
        os.system("vcgencmd display_power 1")

# RISING,FALLING or BOTH. https://sourceforge.net/p/raspberry-gpio-python/wiki/Inputs/
GPIO.add_event_detect(channel, GPIO.BOTH, callback=my_callback)  # add edge detection on a channel
# for testing, use:
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_UP)
# or
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)
time.sleep(31415927) # one year

Categories
Debian Linux Raspberry Pi

My favorite bash scripting tips

Intro

The linux bash shell is great and very flexible. I love to use it and have even installed WSL 2 on my PCs so I can use it as much as possible. When it comes to scripting it’s not exactly my favorite. there is so much history it has absorbed that there are multiple ways to do everything: the really old way, the new way, the alternate way, etc. And your version of bash can also determine what features you can use. nevertheless, I guess if you stick to the basics it makes sense to use bash for simple scripting tasks.

So just like I’ve compiled all the python tips I need for writing my simple python scripts in one convenient, searchable page, I will now do the same for bash. No one but me uses it, but that’s fine.

Iterate (loop) over a range of numbers

END=255 # for instance to loop over an ocetet of an IP address
for i in $(seq 1 $END); do
  echo $i
done
# But if it's OK to just hard-wire start and end, then it's simpler to use:
for i in {1..255}; do echo $i; done

Infinite loop
while /bin/true; do...done

You can always exit to stop it.

Sort IPs in a sensible order

$ sort -n -t . -k1,1 -k2,2 -k 3,3 -k4,4 tmp

What directory is this script in?

DIR=$(cd $(dirname $0);pwd);echo$DIR

Guarantee this script is interpreted (run) by bash and not good ‘ole shell (sh)!
if [ ! "$BASH_VERSION" ] ; then
  exec /bin/bash "$0" "$@"
  exit
fi
Count total occurrences of the word print in a bunch of files which may or may not be compressed, storing the output in a file

print=0
zgrep -c print tst*|cut -d: -f2|while read pline; do prints=$((prints + pline));echo $prints>prints; done

Note that much of the awkwardness of the above line is to get around issues I had with variable scope.

Permitted characters in variable names

Don’t use _ as you might in python! Stick to alphanumeric, but also do not begin with a number!

Execute a command

I used to use back ticks ` in the old days. parentheses is more visually appealing:

print1=$(cat prints)

Variable type

No, variables are not typed. Everything is treated as a string.

Function definition

Put function definitions before they are invoked in the script. Invocation is by plain name. function syntax is as in the example.

sendsummary() {
# function execution statements go here, then close it out
} # optionally with a comment like end function sendsummary
sendsummary # invoke our sendsummary function
Indentation

Unlike python, line indentation does not matter. I recommend to indent blocks of code two spaces, for example, for readability.

Booleans and order of execution
[[ "$DEBUG" -eq "1" ]] && echo subject, $subject, intro, "$intro"

The second statement only gets executed if the first one evaluated as true. Now a more complex example.

[[ $day -eq $DAY ]] || [[ -n “$anomalies” ]] && { statements…}

The second expressions get evaluated if the first one is false. If either the first or second expressions are true, then the last expression — a series of statements in what is essentially an unnamed function, hence the enclosing braces — gets executed. The -n is a test to see of length of a string is non-zero. See man test.

Conditionals

Note that clever use of && and || can in many cases obviate the need for a class if…then structure. But you can use if thens. An if block is terminated by a fi. There is an else statement as well as an elif (else if) statement.

grep conditionals
ping -c1 8.8.8.8|grep -iq '1 received'
[ $? -eq 0 ] && echo this host is alive

So the $? variable after grep is run contains 0 if there was a match and 1 if there was no match. -q argument puts grep in “quiet” mode (no output).

More sophisticated example testing exit status and executing multiple commands

#!/bin/bash
# restart mariaDB if home page response becomes greater than one second
curl -m1 -ksH 'Host:drjohnstechtalk.com' https://localhost/blog/ > /dev/null
# if curl didn't have enough time (one sec), its exit status is 28
[ $? -eq 28 ] && (systemctl stop mariadb; sleep 3; systemctl start mariadb; echo mariadb restart at $(date))

Note that I had to group the commands after the conditional test with surrounding parentheses (). That creates a code block. Without those the semicolon ; would have indicated the end of the block! A semicolon ; separates commands. Further note that I nested parentheses and that seems to work as you would hope. also note that STDOUT has been redirected by the greater than sign > to /dev/null in order to silently discard all STDOUT output. /dev/null is linux-specific. The windows equivalent, apparently, is nul. Use curl -so nul suppress output on a Windows system.

Reading in parameters from a config file

Lots of techniques demoed in this example!

# read in params from file QC.conf
IFS=$'\n'
echo Parameters from file
for line in $(<QC.conf); do
  [[ "$line" =~ ^# ]] || {
  pval=$(echo "$line"|sed 's/ //g')
  lhs=$(echo "$pval"|cut -d= -f1)
  rhs=$(echo "$pval"|cut -d= -f2)
  declare -g $lhs="$rhs"
  echo $lhs is ${!lhs}
  }
done

Note the use of declare with the -g (global) switch to assign a variable to a variable-defined variable name! Note the use of < to avoid creation of a subshell. Note the use of -P argument in grep so that it uses perl-style regex! Note the way to get the value of a variable whose name itself is represented by a variable var is ${!var}.

This script parses a config file with values like a = a_val, where spaces may or may not be present.

One square bracket or two?

I have no idea and I use whatever I get to work. All my samples work and I don’t have time to test all variations.

Variable scope

I really struggled with this so I may come back to this topic!

Variable interpolation

$variable will suffice for simple, i.e., one-word content. But if the variable contains anything a bit complex such as words separated by spaces, or containing unusual characters, better go with double quotes around it, “$variable”. And sometimes syntactically throw in curly braces to separate it from other elements, “${variable}”

Eval
eval="ls -l"
$eval # executes ls -l
Shell expansion
mv Pictures{,.old} # renames directory Pictures to Pictures.old
Poor man’s launch at boot time

Use crontab’s @reboot feature!

@reboot sleep 25; ./recordswitch.sh > recordswitch.log 2>&1

The above expression also shows how to redirect standard error to standard out and have both go into a file.

Use extended regular expressions, retrieving a positional field using awk, and how to subtract (or add) two numbers
t1=`echo -n $line|awk '{print $1}'` 
t2=`echo -n $line|awk '{print $4}'` 
# test for integer inputs 
[[ "$t1" =~ ^[0-9]+$ ]] && [[ "$t2" =~ ^[0-9]+$ ]] && downtime=$(($t1-$t2))

Oops, I used the backticks there! I never claim that my way is the best way, just the way that I know to work! I know of a zillion options to add or subtract numbers…

Get last field using awk
echo hi.there.111|awk -F\. '{print $NF}' # returns 111
Print all but the first field using awk

awk ‘{$1=””; print substr($0,2)}’

Why do assignments have no extra spaces?

It simply doesn’t work if you try to put in spacing around the assignment operator =.

Divert stdout and stderr to a file from within the script
log=/tmp/my-log.log
exec 1>$log 
exec 2>&1
Lists, arrays amd dictionary variables

I don’t think bash is for you if you need these types of variables.

Formatted date

date +%F

produces yyyy-mm-dd, i.e., 2024-01-25

date +%Y%m%d -> 20240417

Poor man’s source code versioning

The old EDT/TPU editor on VAX used to do this automatically. Now I want to save a version of whatever little script I’m currently working on in the ~/tmpFRI (if it’s Friday) directory to sort of spread out my work by day of the week. I call this script cpj so it’s easy to type:

#!/bin/bash
# save file using sequential versioning to tmp area named after this day - DrJ
DIR='~'/tmp$(date +%a|tr '[a-z]' '[A-Z]') # ~/tmp + day of the week, e.g., FRI
DIRREAL=$(eval "echo $DIR") # the real diretory we need
mkdir -p $DIRREAL
for file in $*; do
  res=$(ls $DIRREAL|egrep "$file"'\.[0-9]{1,}$') # look for saved version numbers of this filename
  if test -n "$res"; then # we have seen this file...
    suffix=$(echo $res|awk -F\. '{print $NF}')  # pull out just the number at the end
    nxt=$(($suffix+1)) # add one to the version number
    saveFile="${file}"."${nxt}"
  else # new file to archive or no versioned number exists yet
    [[ -f $DIRREAL/$file ]] && saveFile="$file".1
    [[ -f $DIRREAL/$file ]] || saveFile=""
  fi
  cp "$file" $DIRREAL/"$saveFile"
  [[ -n $saveFile ]] && target=$DIR/"$saveFile"
  [[ -n $saveFile ]] || target="$DIR"
  echo copying "$file" to "$target"
done

It is a true mis-mash of programming styles, but it gets the job done. Note the use of eval. I’m still wrapping my head around that. Also note the technique used to upper case a string using tr. Note the use of extended regular expressions and egrep. Note the use of tilde ~ expansion. I insist on showing the target directory as ~/tmpSAT or whatever because that is what my brain is looking for. Note the use of nested $‘s.

Now that cpj is in place I occasionally know I want to make that versioned copy before I launch the vi editor, so I created a vij in my bash alias file thusly:

vij () { cpj "$@";sleep 1;vi "$@"; }

Another example

I wrote this to retain one backup per month plus the last 28 days.

#!/bin/bash
# do some date arithmetic to preserve backup from first Monday in the month
#[[ $(date +%a) == "Wed" ]] && { echo hi; }
DEBUG=0
DRYRUN=''
[[ $DEBUG -eq 1 ]] && DRYRUN='--dry-run'
if [[ $(date +%a) == "Mon" ]] && [[ $(date +%-d) -lt 8 ]]; then
# preserve one month ago's backup!
  echo "On this first Monday of the month we are keeping the Monday backup from four weeks ago"
else
  d4wksAgo=$(date +%Y%m%d -d'-4 weeks') # four weeks ago
  oldBackup=zones-${d4wksAgo}.tar.gz
  git rm $DRYRUN backups/$oldBackup
fi
today=$(date +%Y%m%d)
todaysBackup=zones-${today}.tar.gz
git add $DRYRUN backups/$todaysBackup

It incorpoates a lot of the tricks I’ve accumulated over the years, too numerous to recount. But it’s a good example to study.

Calculate last weekday

today=$(date -u +%Y%m%d) # UTC date
# last weekday calculation
delta="-1"
[[ $(date -u +%a) != "Mon" ]] || delta="-3"
lastday=$(date -u +%Y%m%d -d"${delta} days")

Output the tab character in an echo statement

Just use the -e switch as in this example:

echo -e “$subnet\t$SSID”

Get top output in a non-interactive (batch) shell

top -b -n 1

Prompting for user input

echo -n “Give your input: “

read userInput

Print first 120 characters of each line in a text file

cat file | cut -c -120

Reverse the lines in a file

tac file > file-reversed # tac is cat in reverse!

Send email when there is no mailx, mail or postifx setup

Use curl!

curl –url smtp://mail-relay.com –mail-from $sender –mail-rcpt $recipient -T <(echo -e “$msg”)

Format json into something readable

curl json_api|python3 -m json.tool

Merge every other line in a file

sed ‘N;s/\n/ /’ file

Ending script on compound conditional can be a bad idea

I ended my script with this statement:

# send alerts if needed
[[ $notify -gt 0 ]] && alerting

Problem was, this last statement has normal value of 1 (first condition is false so second expression not evaluated) so whole script exits with value 1 and my ADO pipeline felt that was an error! Guess I’ll add an exit 0 at the end…

Editing file in place with sed

Thge -i switch to sed is designed to do your substitutions right in the file. Here’s an actual crontab entry where I used that switch:

35 22 * * * sed -i s'/enabled=0/enabled=1/' /etc/yum.repos.d/thousandeyes.repo > /dev/null 2>&1
Conclusion

I have documented here most of the tecniques I use from bash to achieve simple yet powerful scripts. My style is not always top form, but as I learn better ways I will adopt and improve.

Categories
Linux Raspberry Pi

Multiple Raspberry Pi photo frames

Intro

I have previously shared my work on displaying pictures in a nice slideshow from a Google Drive to an HDMI monitor. All these years later, it is still working and every day we see a new slideshow or randomly yet thoughtfully chosen pictures.

Building on this, today I extended this solution to display these pictures on a second monitor in a different room.

Equipment
  • RPi 4 (what I happened to have around. RPi 3 would also be fine)
  • HDMI monitor
  • Raspberry Pi Lite OS
The details

I needed to install fbi (sudo apt-get install fbi).

I needed to copy over m4.pl from the primary display, and for good measure black.jpg.

The general idea is to copy the pics over to the second display once per day.

I call this program copyslideshow.sh.

#!/bin/bash
# copy pictures from primary source
rm -rf Pictures.old
mv Pictures{,.old}
sshpass -p raspberry scp -r [email protected]:mediashow .
sshpass -p raspberry scp -r [email protected]:Pictures .
./m4.pl  $HOME/mediashow >> m4.log 2>&1

Yeah so I hard-coded the RPi password which is still set to the default. I’m willing to take the risk

Then in crontab I added this line:

# get yesterday's pictures!
1 5 * * * killall m4.pl; ./copyslideshow.sh > copyslideshow.log 2>&1

Note that these displays are not synced. That would be a whole ordeal. In fact we thought it would be cool to display different pictures. So the second monitor will be showing yesterday’s slideshow from the main monitor.

Automating turn-on, turn-off of the HDMI display based on the ambient room light

Since this second slideshow is in a bedroom, I wanted to have it turn off when the lights were out, and turn back on again during daylight. This was a really interesting challenge for me as I got to use an inexpensive external sensor with my RPi. And I got it to work, and it works quite well if I say so myself. That’s all written up in this post.

Reference and related

I have written many variations on this same topic. I guess this is a good one.

How I automatically turn on the HDMI display in the morning and turn it off again at night.

Categories
Network Technologies Raspberry Pi

Trying to improve my home WiFi with a range extender

Intro

My Teams meetings in the mornings had poor audio quality and sometimes I could not share my screen. My suspicions focused on my home WiFi Router, which is many years old. I decided to make an experiment and get a range extender. The results are, well, mixed at best.

Windows command

netsh wlan show interface

There is 1 interface on the system:
Name : Wi-Fi 
Description : Intel(R) Dual Band Wireless-AC 3168 
GUID : f1c094c0-fcb7-4e47-86ba-51df737e58c8 
Physical address : 28:c6:3f:8f:3a:27 
State : connected 
SSID : DrJohn 
BSSID : ec:c3:02:eb:2d:7c 
Network type : Infrastructure 
Radio type : 802.11ac 
Authentication : WPA2-Personal 
Cipher : CCMP 
Connection mode : Auto Connect 
Channel : 153 
Receive rate (Mbps) : 292.5 
Transmit rate (Mbps) : 292.5 
Signal : 99% 
Profile : DrJohn

802.11ac is WiFi 5. 802.11n is WiFi 2, to be clear about it.

What’s going on

My work laptop starts out using WiFi 5 (803.11ac). The signal is around 60% or so. So I guess not super great. Then after an hour or so it switches to WiFi 2 (802.11n)! Audio in my meetings gets disturbed during this time.

My WiFi Extender did not really change this behavior to my surprise! But maybe the quality is better.

One morning I started out on WiFi 4, the signal quality varied between 94% down to 61%, all while nothing was being moved, and within a matter of minutes! The lower Signal values are associated with slower transmit and receive rates, naturally. But at least with the extender WiFi 4 seems OK. It’s useable for my interactive meetings. In my experience, once you are on WiFi 4 you are very unlikely to automagically get switched back to WiFi 5. But the reverse is not true. So there’s a lot of variability in the signal over the course of minutes. But I stayed on WiFi 4 for over three hours without its changing. I connected to a differ SSID, then connected back to my _EXT SSID and, bam, WiFi 5, but only at 52% signal strength.

The way I know this behavior in detail is that I happen to have a ThousandEyes endpoint agent installed and I have access to this history of the connection quality, signal strength, thoughput, etc. ThousandEyes is pretty cool.

Further experimentation

The last couple days I’ve been getting WiFi 5 and it’s been sticking. What’s the difference? This sounds incredibly banal, but I stood the darn extender upright! That’s right, during those days when I was mostly getting WiFi 4 the Extender had all its antennae sticking out, but it was flat on a table. I am in a room across the hallway. Then I managed to stand it upright – a little tricky since it is pluued into an extension cord. I’m still across the hallway. But things have been behaving better ever since.

Does a WiFi extender create a new SSID?

Yes! It creates an SSID named after your SSID with an _EXT appended to that name. However, it is very important to note that it is a bridged network so it means your _EXT-connected devices see all your devices not on _EXT, and that makes it very convenient. The subnet used is your primary router’s subnet, in other words.

This TP-Link (see references) seems to have lots of nice features. MIMO, AP mode, mesh mode, etc. You may or may not need them right away. For instance, the device has several status LEDs which get kind of bright for a bedroom at nighttime. Originally we covered it with a dark T-Shirt. Then I looked at it and saw it has an LED switch! That’s right. Just press that LED switch and those way-too-bright LEDs stop illuminating, while the device keeps on working. A very small but thoughtful feature which you would never even think to look for but turns out to be important. It might have overheated had we kept it covered with that T-Shirt.

Raspberry Pi

A good command is:

sudo iwconfig wlan0

wlan0 IEEE 802.11 ESSID:"Music_EXT"
Mode:Managed Frequency:5.765 GHz Access Point: 9C:53:22:02:6B:59
Bit Rate=433.3 Mb/s Tx-Power=31 dBm
Retry short limit:7 RTS thr:off Fragment thr:off
Encryption key:off
Power Management:on
Link Quality=62/70 Signal level=-48 dBm
Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0

To be continued…

References and related

TPLink AC1900 WiFi Range Extender at Amazon (Costs about $69. I do not get promotional credits!)

Categories
Linux Python Raspberry Pi

vlc command-line tips

Intro

I’m looking to test my old Raspberry Pi model 3 to see if it can play mp4 videos I recorded on my Samsung Galaxy A51 smartphone. I had assumed it would get overwhelmed and give up, but I haven’t tried in many years, so… The first couple videos did play, sort of. I was using vlc. Now if you’ve seen any of my posts you know I’ve written a zillion posts on running a dynamic slideshow based on RPi. Though the most important of these posts was written years ago, it honestly still runs and runs well to this day, amazingly enough. Usually technology changes or hardware breaks. But that didn’t happen. Every day I enjoy a brand new slideshow in my kitchen.

In most of my posts I use the old stalwart program fbi. In fact I don’t even have XWindows installed – it’s not a requirement if you know what you’re doing. But as far as I can see, good ‘ole fbi doesn’t do streaming media such as videos in mp4 format. As far as I know, vlc is more modern and most importantly, better supported. So after a FAIL trying with mplayer (still haven’t diagnose that one), I switched to trials with vlc.

I haven’t gotten very far, and that’s why I wanted to share my learnings. There’s just so much you can do with vlc, that even what you may think are the most common things anyone would want are very hard to find working examples for. So that’s where I plan to contribute to the community. As I figure out an ‘easy” thing, I will add it here. And if I’m the only one who ever refers to this post, so be it. I love my own My favorite python tips, post, for instance. it has everything I use on a regular basis. So I’m thinking this will be similar.

References and related

My RPi slideshow blog post

My favorite python tips – everything I need!

Categories
Linux Raspberry Pi

Interpreting speech with a Raspberry Pi

Or the beginning or creating your own smart speaker

Intro

Imagine you could use a low-cost device to interpret speech without the aid of the big cloud services and their complexity and security and big-brotherly-ness. Well if you have a DIY mindset, you can!

I wanted to control the raspberry pi-based slideshow I have written about many times in the past with voice commands. The question became How could I do it and is it even possible at all? And would I need to master the complex apis provided by either Amazon or Google cloud services? Well, it turns out that it is possible to do passable speech to text without any external cloud provider; and I am very excited to share what I’ve learned so far.

Equipment

raspberry pi 4 (even my old RPI 3 seems to work)

USB microphone

Raspberry Pi OS

Skills

basic linux and python skills are required

vosk – your main tool

I’m going to cut to the chase and just tell you that the vosk api is how I got this all working, but not before I drove into several dead-ends.

Here are the vosk installation instructions, which do work on RPi:

Vosk Installation (alphacephei.com)

It will be helpful to install and test the examples:

git clone https://github.com/alphacep/vosk-api
cd vosk-api/python/example
python3 ./test_simple.py test.wav

On my RPi 4 it took 36 s the first time, and 6.6 s the second time to run this test.wav. So I got worried and fully expected it would be just too slow on these underpowed RPi systems.

But I forged ahead and looked for an example which could do real-time speech-to-text. They provide a microphone example. It requires some additional packages. But even after installing them it still produced a nasty segmentation fault. So I gave up on that. Then I noticed an ffmpeg-based example. Well, turns out I have lots of prior ffmpeg experience as I also post about live recording of audio with the raspberry pi.

It turns out their example was simply to use ffmpeg to interpret a file, but I didn’t know that to begin with. But I know my way around ffmpeg that I could use it for processing a lvie stream. So I made those changes, and voila. I’m glad to say I was dead wrong about the processing speed. On the RPi 4 it can keep up with its text-to-speech task in real time!

Basic program to examine your speech in real time

I developed the following python script based off one of the python examples from the api. I call it drjtst4.py, just to give it a name:

#!/usr/bin/env python3

import subprocess
import re
from modules import aux_modules

from vosk import Model, KaldiRecognizer, SetLogLevel

SAMPLE_RATE = 16000

SetLogLevel(0)

model = Model(lang="en-us")
rec = KaldiRecognizer(model, SAMPLE_RATE)
start,start_a = 0,0
input_device = 'plughw:1,0'
phrase = ''
accumulating = False
# wake word hey photo is often confused with a photo by vosk...
wake_word_re = '^(hey|a) photo'

with subprocess.Popen(["ffmpeg","-loglevel", "quiet","-f","alsa","-i",
                            input_device,
                            "-ar", str(SAMPLE_RATE) , "-ac", "1", "-f", "s16le", "-"],
                            stdout=subprocess.PIPE) as process:

    while True:
        data = process.stdout.read(4000)
        if len(data) == 0:
            break
        if rec.AcceptWaveform(data):
            print('in first part')
            print(rec.Result())
            text = rec.PartialResult()
# text is a "string" which is basically a dict
            start,start_a,accumulating,phrase = aux_modules.process_text(wake_word_re,text,start,start_a,accumulating,phrase)
        else:
# this part always seems to be executed for whatever reason
            print('in else part')
            text = rec.PartialResult()
            start,start_a,accumulating,phrase = aux_modules.process_text(wake_word_re,text,start,start_a,accumulating,phrase)
            print(rec.PartialResult())

# we never seem to get here
    print(rec.FinalResult())
    print('In final part')
    text = rec.FinalResult()

I created a modules directory and in it a file called aux_modules.py. It look like this:

import re,time,json

def process_text(wake_word_re,text_s,start,start_a,accumulating,phrase):
    max = 5.5 # seconds
    inactivity = 10 # seconds
    short_max = 1.5 # seconds
    elapsed = 0
    if time.time() - start_a < inactivity:
# Allow some time to elapse since we just took an action
        return start,start_a,accumulating,phrase
# convert text to real text. Real text is in 'partial'
    text_d = json.loads(text_s)
    text = ''
    if 'partial' in text_d:
        text = text_d['partial']
    if 'text' in text_d:
        text = text_d['text']
    if not text == '': phrase = text
    if re.search(wake_word_re,text):
        if not accumulating:
            start = time.time()
            accumulating = True
            print('Wake word detected. Now accumulating text.')
    l = len(re.split(r'\s',text))
    print('text, word ct',text,l)
    if accumulating:
        elapsed = time.time() - start
        print('Elapsed time:',elapsed)
        if l > 1:
           phrase = text
    if elapsed > max or (elapsed > short_max and l == 1):
# we're at a natural ending here...
        print('This is the total text',phrase)
# do some action
# reset everything
        accumulating = False
        phrase = ''
        start_a = time.time()
    return start,start_a,accumulating,phrase

And you just invoke it with python3 drjtst4.py.

Sample session output
in else part
text, word ct 1
{
"partial" : ""
}
in else part
text, word ct hey 1
{
"partial" : "hey"
}
in else part
text, word ct hey 1
{
"partial" : "hey"
}
in else part
text, word ct hey 1
{
"partial" : "hey"
}
in else part
Wake word detected. Now accumulating text.
text, word ct hey photo 2
Elapsed time: 0.0004639625549316406
{
"partial" : "hey photo"
}
in else part
text, word ct hey photo 2
Elapsed time: 0.003415822982788086
{
"partial" : "hey photo"
}
in else part
text, word ct hey photo 2
Elapsed time: 0.034906625747680664
{
"partial" : "hey photo"
}
in else part
text, word ct hey photo 2
Elapsed time: 0.09063172340393066
{
"partial" : "hey photo"
}
in else part
text, word ct hey photo 2
Elapsed time: 0.2488384246826172
{
"partial" : "hey photo"
}
in else part
text, word ct hey photo 2
Elapsed time: 0.33771753311157227
{
"partial" : "hey photo"
}
in else part
text, word ct hey photo place 3
Elapsed time: 0.7102789878845215
{
"partial" : "hey photo place"
}
in else part
text, word ct hey photo place 3
Elapsed time: 0.7134637832641602
{
"partial" : "hey photo place"
}
in else part
text, word ct hey photo player 3
Elapsed time: 0.8728365898132324
{
"partial" : "hey photo player"
}
in else part
text, word ct hey photo player 3
Elapsed time: 0.8759913444519043
{
"partial" : "hey photo player"
}
in else part
text, word ct hey photo play slideshow 4
Elapsed time: 1.0684640407562256
{
"partial" : "hey photo play slideshow"
}
in else part
text, word ct hey photo play slideshow 4
Elapsed time: 1.0879075527191162
{
"partial" : "hey photo play slideshow"
}
in else part
text, word ct hey photo play slideshow 4
Elapsed time: 1.3674390316009521
{
"partial" : "hey photo play slideshow"
}
in else part
text, word ct hey photo play slideshow 4
Elapsed time: 1.3706269264221191
{
"partial" : "hey photo play slideshow"
}
in else part
text, word ct hey photo play slideshow 4
Elapsed time: 1.5532972812652588
{
"partial" : "hey photo play slideshow"
}
in else part
text, word ct hey photo play slideshow 4
Elapsed time: 1.5963218212127686
{
"partial" : "hey photo play slideshow"
}
in else part
text, word ct hey photo play slideshow 4
Elapsed time: 1.74298095703125
{
"partial" : "hey photo play slideshow"
}
in else part
text, word ct hey photo play slideshow 4
Elapsed time: 1.842745065689087
{
"partial" : "hey photo play slideshow"
}
in else part
text, word ct hey photo play slideshow 4
Elapsed time: 1.9888567924499512
{
"partial" : "hey photo play slideshow"
}
in else part
text, word ct hey photo play slideshow 4
Elapsed time: 2.0897343158721924
{
"partial" : "hey photo play slideshow"
}
in first part
{
"text" : "hey photo play slideshow"
}
text, word ct 1
Elapsed time: 2.3853299617767334
This is the total text hey photo play slideshow
in else part
{
"partial" : ""
}
in else part
{
"partial" : ""
}
A word on accuracy

It isn’t Alexa or Google. No one expected it would be, right? But if you’re a native English speaker it isn’t too bad. You can see it trying to correct itself.

The desire to choose an uncommon wake word of three syllables is at direct odds with how neural networks are trained! So… although I desired my wake word to be “hey photo,” I also allow “a photo.” A photo was probably in their training set whereas Hey photo certainly was not. Hence the bias against recognizing a unique wake word. And no way will I re-train their model – way too much effort. But to lower false positives this phrase has to occur at the beginning of a spoken phrase.

Turning this into a smart speaker

You can see I’ve got all the pieces set up. At least I think I do! I’ve got my wake word. I don’t have natural language processing but I think I can forgo that. I have a place in the code where I print out the “final text.” That’s where the spoken command is perceived to have been uttered and and a potential action could be exectured at that point.

Dead ends

To be fleshed out later as time permits.

Conclusion

I have demonstrated how speech-to-text without use of complex cloud apis such as those provided by Amazon and Google can be easily achieved on an inexpensive raspberry pi.

I will be building on this facility in subsequent posts as I turn my RPi-powered slideshow into a slideshow which reacts to voice commands!

References and related

Vosk Installation (alphacephei.com)

Raspberry Pi slideshow

This conference USB mic works really well for me.

Categories
Raspberry Pi

Raspberry Pi Display blanks out after a few seconds

Intro

If you’ve seen any of my many Raspberry Pi photo frame posts, you’ve seen me mention the Pi Display. I recently began having problems with my setup, after many months. I didn’t know if it was the RPi or the display. My display was cracked from abuse anyway so I took the plunge and got a new one. And I re-imaged my micro SD card. I’ll be danged if my brand new display wasn’t blanking out, even as the RPi was still booting! That was frustrating. Here’s how I resolved that. I felt I should publish this article because it’s a little hard to find this exact problem with solution described on the Internet.

The details

The good news is that the Pi Display is still available, incredibly. It comes with absolutely zero documentation. It refers you to element14 or newark web sites, I forget. But those sites just display very generic pages and I couldn’t manage to find any real documentation. I assume the product is considered too old or unsupported or whatever. But since I had bought it already and it seems exactly like the one I was replacing, I went ahead and connected it as before.

Note that the display did not blank out during the installation of Raspbian Lite. That’s a strong hint as to where the problem lies. But after installing that and rebooting, the Pi Display shows bland and white horizontal lines, becomes fainter, and then just blanks out altogether. Rather distressing. I don’t recall this problem from the older versions.

Equipment

I’m still using an old RPi model 3 I have laying around. I guess the new RPi 4 is OK, too. I believe it draws more power though so I actually kind of prefer the old RPi 3’s. You can manage to power both RPi 3 + the PiDisplay from a single power plug if you do it right.

The OS image is Raspberry Pi OS Lite, “bullseye.”

7″ Pi Display – see references.

Random power plug from Amazon. 1.8 amps.

The solution

First, note that I had installed Raspbian bullseye – the latest image as of January 2022. Also note that when connected to an HDMI display, this screen blank-out never occurs! What I needed to do was to connect the RPi to an HDMI display (my TV in my case) and do the sudo apt-get update and sudo apt-get upgrade commands. That recommendation you will find everywhere. In addition, I needed to run raspi-config. In there you navigate to do this navigation:

Display Options > Screen blanking > Would you like to enable screen blanking?

Choose No

I held my breath as I rebooted, and it seemed to want to blank, but then came back a half second later and the display has been illuminated ever since. Now it seems every time I reboot it blanks out as before, during boot, but then “remembers” it’s not supposed to do that and recovers within a second!

How to power both RPi and Pi Display from a single power source

You can connect a short micro USB cable from the USB output of the display to the power input of the RPi – that’s what I do. You could also use the included leads, but I haven’t messed with it.

However, note that – as I found out the hard way – not just any power adapter will do! I started with an Amazon adapter rated for 1 amp. Everything fired up fine, and even ran the slideshow for a bit, but then I guess the power demands were too great and the RPi spontaneously rebooted. So then I switched the system to a larger adapter, also from Amazon – 1.8 amps. So far so good with that one. If that failed, I was prepared with a 2 amp adapter – from a Samsung Galaxy phone. But it seems that will not be necessary.

And this is kind of the beauty of the RPi 3 – you can get away with stuff like that. Your RPi 4 could never do that. It’s a power hog.

Conclusion

The Pi Display is still viable! The price has gone up, but it works. As far as I know in fact it is the “official” display for an RPi. The RPi has a special tab to receive a ribbon cable that connects the two together.

The Pi Display will blank out after a few seconds with newer Raspberry Pi OS images, it seems, even though when paired with an HDMI display this never happens. But it’s easy enough to fix this frustrating problem. Just set screen blanking to No in raspi-config. This ought to better documented on the Internet.

References and related

The Pi Display is a 7″ touch-sensitive LCD display. Its cost is now up to $90. I don’t recommend using its touch capability, but then I don’t like that feature on any display, come to think of it: Amazon.com: Raspberry Pi 7″ Touch Screen Display : Electronics But this latest Pi Display is brighter than my older one. That makes it more visible and enjoyable to watch.

One of my very many posts on creating a photo frame with an RPi and the PiDisplay (but also works fine on HDMI TV’s – I tested it!): https://drjohnstechtalk.com/blog/2021/01/raspberry-pi-photo-frame-using-the-pictures-on-your-google-drive-ii/

Categories
Python Raspberry Pi

Solution to NPR’s puzzle using python

Intro

Of course I’m using my Raspberry Pi. I installed a free dictionary (sudo apt-get install wamerican).

I wanted to practice python, but to not go crazy, so I cheated with some grooming with simple shell commands, e.g.,

$ egrep ‘^[a-z]{5}$’ /usr/share/dict/words > words

That plucks out all the five-letter words and stores them in a local file called words. The original dictionary had about 100,000 “words.” There are about 4000 five-letter words left.

I called the python program v.py. Here it is

import sys,os,re
input = sys.argv[1]
output = []
for character in input:
    number = ord(character) - 96
    output.append(number)
#print(output)
sum=0
for character in input:
    number = ord(character) - 96
    sum += number
#print(sum)
newsum = 0
ends = ''
if sum == 51:
    #print(input)
    for i in input[3:]:
        number = ord(i) - 96
        newsum += number
        ends = ends + i
        #print(i,number,newsum,ends)
    if newsum < 27:
        newl = chr(newsum+96)
        newword = re.sub(ends,newl,input)
        print(input,newword)

I ran it like this:

$ cat words|while read line; do python3 v.py $line >> all-possibilities; done

This plucks out all the five-letter words whose characters add up to 51, and adds the value of the last two letters and creates a new word from them. replacing the last two letters with the new letter.

Drumroll

The results are in.

$ cat all-possibilities

allay allz
avoid avom
bergs berz
beset besy
blocs blov
bombs bomu
broke brop
bused busi
comas comt
condo cons
cribs criu
crude crui
cured curi
dines dinx
elite eliy
erect erew
fates fatx
files filx
flies flix
fluff flul
...
thick thin

Now, go write your own program. I will share the answer when it's too late to submit - it comes towards the end of the list. It sticks out like a sore thumb - nothing else comes close. So if you just persist you'll see it.

Conclusion

I learned a teensy bit of python, my motivation being to solve the current npr puzzle. And it worked! But my program was surprisingly slow. I guess I wrote it in an inefficient manner.

Categories
Admin Linux Raspberry Pi

Scripts checker

Intro

Imagine an infrastructure team empowered to create its own scripts to do such things as regularly update external dynamic lists (EDLs) or interact with APIs in an automated fashion. At some point they will want to have a meta script in place to check the output of the all the automation scripts. This is something I developed to meet that need.

I am getting tired of perl, and I still don’t know python, so I decided to enhance my bash scripting for this script. I learned some valuable things along the way.

checklogs.sh

I call the script checklogs.sh Here it is.

#!/bin/bash
# DrJ 2021/12/17, updated 2023/7/26
# it is desired to run this using the logrotate mechanism
#
# logrotate invokes with /bin/sh so we have to do this trick...
if [ ! "$BASH_VERSION" ] ; then
  exec /bin/bash "$0" "$@"
  exit
fi
DIR=$(cd $(dirname $0);pwd)
INI=$DIR/log.ini
DAY=2 # Day of week to analyze full week of logs. Monday is 1, Tuesday 2, etc
DEBUG=0
maxdiff=10
maxerrors=10
minstarts=10
TMPDIR=/var/tmp
cd $TMPDIR
recipients="[email protected]"
#
checklog2() {
  starts=0;ends=0;errors=0
  [[ "$DEBUG" -eq "1" ]] && echo ID, $ID, LPATH, $LPATH, START, $START, ERROR, $ERROR, END, $END
  LPATH="${LPATH}${wildcard}"
# the Ec switches mean (E) extnded regular expressions, (c) count of matching lines
  zgrep -Ec "$START" ${LPATH}|cut -d: -f2|while read sline; do starts=$((starts + sline));echo $starts>starts; done
  zgrep -Ec "$END" ${LPATH}|cut -d: -f2|while read sline; do ends=$((ends + sline));echo $ends>ends; done
# Outlook likes to remove our newline characters - double up on them with this sed trick!
  zgrep -Ec "$ERROR" ${LPATH}|cut -d: -f2|sed 'a\\'|while read sline; do errors=$((errors + sline));echo $errors>errors; done
  exampleerrors=$(zgrep -E "$ERROR" ${LPATH}|head -10)
  starts=$(cat starts)
  ends=$(cat ends)
  errors=$(cat errors)
  info="${info}===========================================
$ID SUMMARY
  Total starts: $starts
  Total finishes: $ends
  Total errors: $errors
  Most recent errors: "
  info="${info}${exampleerrors}

"
  unset NEW
# get cumulative totals
  starttot=$((starttot + starts))
  endtot=$((endtot + ends))
  errortot=$((errortot + errors))
  [[ "$DEBUG" -eq "1" ]] && echo starttot, $starttot, endtot, $endtot, errortot, $errortot
  [[ "$DEBUG" -eq "1" ]] || rm starts ends errors
} # end of checklog2 function

checklog() {
# clear out stats and some variables
starttot=0;endtot=0;errortot=0;info=""
#this IFS and following line is trick to preserve those darn backslash charactes in the input file
IFS=$'\n'
for line in $(<$INI); do
  [[ "$line" =~ ^# ]] || {
  pval=$(echo "$line"|sed s'/: */:/')
  lhs=$(echo $pval|cut -d: -f1)
  rhs=$(echo "$pval"|cut -d: -f2-)
  lhs=$(echo $lhs|tr [:upper:] [:lower:])
  [[ "$DEBUG" -eq "1" ]] && echo line is "$line", pval is $pval, lhs is $lhs, rhs is "$rhs"
  if [ "$lhs" = "identifier" ]; then
    [[ "$DEBUG" -eq "1" ]] && echo matched lhs = identifer section
    [[ -n "$NEW" ]] && checklog2
    ID="$rhs"
  fi
  [[ "$lhs" = "path" ]] && LPATH="$rhs" && NEW=false
  [[ "$lhs" = "error" ]] && ERROR="$rhs"
  [[ "$lhs" = "start" ]] && START="$rhs"
  [[ "$lhs" = "end" ]] && END="$rhs"
  }
done
# call one last time at the end
checklog2
} # end of checklog function

anomalydetection() {
# a few tests - you can always come up with more...
  diff=$((starttot - endtot))
  [[ $diff -gt $maxdiff ]] || [[ $starttot -lt $minstarts ]] || [[ $errortot -gt $maxerrors ]] && {
    ANOMALIES=1
    [[ "$DEBUG" -eq "1" ]] && echo ANOMALIES, $ANOMALIES, starttot, $starttot, endtot, $endtot, errortot, $errortot
  }
} # end function anomalydetection

sendsummary() {
  subject="Weekly summary of sesamstrasse automation scripts - please review"
  [[ -n "$ANOMALIES" ]] && subject="${subject} - ANOMALIES DETECTED PLEASE REVIEW CAREFULLY!!"

  intro="This summarizes the results from the past week of running automation scripts on sesamstrasse.
Please check that values seem reasonable. If things are out of range, check with Heiko or look at
sesamstrasse yourself.

"

  [[ "$DEBUG" -eq "1" ]] && echo subject, $subject, intro, "$intro", info, "$info"
  [[ "$DEBUG" -eq "1" ]] && args="-v"
  echo "${intro}${info}"|mail "$args" -s "$subject" "$recipients"
} # end function sendsummary

# MAIN PROGRAM
# always check the latest log
checklog
anomalydetection

# only check all logs if it is certain day of the week. Monday = 1, etc
day=$(date +%u)
[[ "$DEBUG" -eq "1" ]] && echo day, $day
[[ $day -eq $DAY ]] || [[ -n "$ANOMALIES" ]] && {
  [[ "$DEBUG" -eq "1" ]] && echo calling checklog with wildcard set
  wildcard='*'
  checklog
  sendsummary
}

[[ "$DEBUG" -eq "1" ]] && echo message so far is "$info"

log.ini

# The suggestion: To have a configuration file with log identifiers
#(e.g. “anydesk-edl”) and per identifier: log file path (“/var/log/anydesk-edl.log”),
# error pattern (“.+\[Error\].+”), start pattern (“.+\[Notice\] Starting$”) end pattern (“.+\[Notice\] Done$”).
#Then just count number of executions (based on start/end) and number of errors.

# the start/end/error values are interpreted as extended regular expressions - see regex(7) man page
identifier: anydesk-edl
path: /var/log/anydesk-edl.log
error: .+\[Error\].+
start: .+\[Notice\] Starting$
end: .+\[Notice\] Done$

identifier: firewall-requester-to-edl
path: /var/log/firewall-requester-to-edl.log
error: .+\[Error\].+
start: .+\[Notice\] Starting$
end: .+\[Notice\] Done$

identifier: sase-ips-to-bigip
path: /var/log/sase-ips-to-bigip.log
error: .+\[Error\].+
start: .+\[Notice\] Starting$
end: .+\[Notice\] Done$

What this script does

So when the guy writes an automation script, he is so meticulous that he follows the same convention and hooks it into the syslogger to create uniquely named log files for it. He writes out a [Notice] Starting when his script starts, and a [Notice] Done when it ends. And errors are reported with an [Error] details. Some of the scripts are called hourly. So we agreed to have a script that checks all the other scripts once a week and send a summary email of the results. I look to see that the count of starts and ends is roughly the same, and I report back the ten most recent errors from a given script. I also look for other basic things. That’s the purpose of the function anomalydetection in my script. It’s just basic tests. I didn’t want to go wild.

But what if there was a problem with one of the scripts, wouldn’t we want to know sooner than possibly six days later? So I decided to have my script run every day, but only send email on the off days if an anomaly was detected. This made the logic a tad more complex, but nothing bash and I couldn’t handle. It fits the need of an overworked operational staff.

Techniques I learned and re-learned from developing this script

cron scheduling – more to it than you thought

I used to naively think that it suffices to look into the crontab files of all users to discover all the scheduled processes. What I missed is thinking about how log rotate works. How does it work? Turns out there is another section of cron for jobs run daily, weekly and monthly. logrotate is called from cron.daily.

logrotate – potential to do more

The person who wrote the automation scripts is a much better scripter than I am. I didn’t want to disappoint so I put in the extra effort to discover the best way to call my script. I reasoned that logrotate would offer the opportunity to run side scripts, and I was absolutely right about that! You can run a script just before the logs rotate, or just after. I chose the just before timing – prerotate. In actual fact logrotate calls the prerotate script with all the log files to be rotate as arguments, which you notice we don’t take advantage of, because at the time we were unsure how we were going to interface. But I figure let’s just leave it now. man logrotate to learn more.

By the way although I developed on a generic Debian system, it should work on a Raspberry Pi as well since it is Debian based.

BASH – the potential to do more, at a price

You’ll note that I use some bash-specific extensions in my script. I figure bash is near universal, so why not? The downside is that when logrotate invokes an external script, it calls is using old-fashioned shell. And my script does not work. Except I learned this useful trick:

if [ ! "$BASH_VERSION" ]; then
  exec /bin/bash "$0" "$@"
  exit
fi

Note this is legit syntax in SHELL and a legit conditional operator expression. So it means if you – and by you I mean the script talking about itself – are invoked via SHELL, then invoke yourself via BASH and exit the parent afterwards. And this actually does work (To do: have to check which occurs first, the syntax checking or the command invocation).

More

Speaking of that conditional, if you want to know all the major comparison tests, do a man test. I have around to use the double bracket expressions [[ more and more, though they are BASH specific I believe. The double bracket can be followed by a && and then an open curly brace { which can introduce a block of code delimited of course by a close curly brace }. So for me this is an attractive alternative to SHELL’s if conditional then code block fi syntax, and probably just slightly more compact. Replace && with || to execute the code block when the condition does not evaluate to be true.

zgrep is grep for compressed files, but we knew that right? But it’s agnostic – it works like grep on both compressed and uncompressed files. That’s important because with rotated logs you usually have a combination of both.

Now the expert suggested a certain regular expression for the search string. It wasn’t working in my first pass. I reasoned that zgrep may have a special mode to act more like egrep which supports extended regular expressions (EREs). EREs aren’t really the same as perl-compatible regular expressions (PCREs) but for this kind of simple stuff we want, they’re close enough. And sure enough zgrep has the -E option to force it to interpret the expression as an ERE. Great.

RegEx

So in the log.ini file the regular expression has a \[…\] syntax. The backslash is actually required because otherwise the […] syntax is interpreted as a character class, where all the characters between the brackets get tried to match a single character in the string to be matched. That’s a very different match!

My big thing was – will I have to further escape those lines read in from log.ini, perhaps to replace a \[ with a \\[? Stuff like that happens. I found as long as I used those double quotes around the variables (see below) I did not need to further escape them. Similarly, I found that the EREs in log.ini did not need to be placed between quotes though the guy initially proposed that. It looks cleaner without them.

Variable scope

I wasted a lot of time on a problem which I thought may be due to some weird variable scoping. I’ve memorized this syntax cat file|while read line; do etc, etc so I use it a lot in my tiny scripts. It’s amazing I got away with it as much as I have because it has one huge flaw. if you start using variables within the loop you can’t really suck them out, unless you write them to a file. So while at first I thought it was a problem of variable scoping – why do my loop variables have no values when the code comes out of the loop? – it really isn’t that issue. It’s that the pipe, |, created a forked process which has its own variables. So to avoid that I switched to this weird syntax for line in $(<$INI); do etc. So it does the line-by-line file reading as before but without the pipe and hence without the “variable scope” problem.

But in another place in the script – where I add up numbers – I felt I could not avoid the pipe. So there I do write the value to a file.

The conclusion is that with the caveat that if you know what you’re doing, all variables have global scope, and that’s just as it should be. Hey, I’m from the old Fortran 66/77 school where we were writing Monte Carlos with thousands of lines of code and dozens of variables in a COMMON block (global scope), and dozens of contributors. It worked just fine thank you very much. Because we knew what we were doing.

Adding numbers in bash

Speaking of adding, I can never remember how to add numbers (integers). In bash you can do starts=$((starts + sline)) , where starts and sline are integers. At least this worked in Debian linux Stretch. I did not really get the same to work so well in SLES Linux – at least not inside a loop where I most needed it.

When you look up how to add numbers in bash there are about a zillion different ways to do it. I’m trying to stick to the built-in way.

Sending mail in Debian linux

You probably need to configure a smarthost if you haven’t used your server to send emails up until now. You have to reconfigure of the exim4 package:

dpkg-reconfigure exim4-config

This also can be done on a RPi if you ever find you need for it to send out emails.

Variables

If a variables includes linebreaks and you want to see that, put it between double-quotes, e.g., echo “$myVariableWithLineBreaks”. If you don’t do that it seems to remove the linebreaks. Use of the double quotes also seems to help avoid mangling variables that contain meta characters found in regular expressions such as .+ or \[.

Result of executing the commands

I grew up using the backtick metacharacter, `, to indicate that the enclosed command should be executed. E.g., old way:

DIR=`dirname $0`

But when you think about it, that metacharacter is small, and often you are unlucky and it sits right alongside a double quote or a single quote, making for a visual trainwreck. So this year I’ve come to love the use of $(command to be executed) syntax instead. It offers much improved readability. But then the question became, could I nest a command within a command, e.g., for my DIR assignment? I tried it. Now this kind of runs counter to my philosophy of being able to examine every single step as it executes because now I’m executing two steps at once, but since it’s pretty straightforward, I went for it. And it does work. Hence the DIR variable is assigned with the compound command:

DIR=$(cd $(dirname $0);pwd)

So now I wonder if you can go more than two levels deep? Each level is an incrementally bad idea – just begging for undetectable mistakes, so I didn’t experiment with that!

By the way the reason I needed to do that is that the script jumps around to another directory to create temporary files, and I wanted it to be able to reference the full path to its original directory, so a simpler DIR=$(dirname $0) wasn’t going to cut it if it’s called with a relative path such as ./checklogs.sh

Debugging

I make mistakes left and right. But I know what results I expect. So I generously insert statements as variables get assigned to double check them, prefacing them with a conditional [[ $DEBUG -eq 1 ]] && print out these values. As I develop DEBUG is set to 1. When it’s finally working, I usually set it to 0, though in some script I never quite reach that point. It looks like a lot of typing, but it’s really just cut and paste and not over-thinking it for the variable dump, so it’s very quick to type.

Another thing I do when I’m stuck is to watch as the script executes in great detail by appending -xv to the first line, e.g., #!/bin/bash -xv. But the output is always confusing. Sometimes it helps though.

Compensating for Outlook’s newline handling

Outlook is too clever for its own good and “helpfully” removes what it considers extra linefeeds. Thanks Microsoft. Really helpful. So if you add extra linefeeds you can kind of get around that, but then you go from 0 linefeeds in the displayed output to two. Again, thanks Microsoft.

Anyway, I disocvered sed ‘a/\\/’ is a way to add an extra linefeed to my error lines, where the problem was especially acute and noticeable.

Techniques I’d like to use in the future

You can assign a function to a variable and then call that variable. I know that will have lots of uses but I’m not used to the construct. So maybe for my next program.

Conclusion

This fairly simple yet still powerful script has forced me to become a better BASH shell scripter. In this post I review some of the basics that make for successful scripting using the BASH shell. I feel the time invested will pay off as there are many opportunities to write such utility scripts. I actually prefer bash to perl or python for these tasks as it is conceptually simpler, less ambitious, less pretentious, yes, far less capable, but adequate for my tasks. A few rules of the road and you’re off and running! bash lends itself to very quick testing cycles. Different versions of bash introduced additional features, and that gets trying. I hope I have found and utilized some of the basic stuff that will be available on just about any bash implementation you are likely to run across.

References and related

The nitty gritty details about BASH shell can be gleaned by doing a man bash. It seems daunting at first but it’s really not too bad once you learn how to skim through it.

This post shows how to properly use the syslog package within python to create these log files that I parse.