Categories
CentOS Debian Linux

What happened to insert mode on the latest version of vi?

Intro

As a creature of habit, I fall for an editor and can never imagine using something else. In the VAX days there was EDT, which if memory serves was replaced by the even better VPU. On Ultrix we ran a pretty nice editor simply called e from Rand Corporation. Then there was the love affair with emacs, and finally for the last 30 years vi.

Well with my latest server, a Debian 12 machine, I was having trouble with insert mode, specifically, inserting text from my Windows clipboard. Never had problems before….

Well for some reason, if you want to insert text from the clipboard, you now use <CTRL.>-<SHIFT>V in command mode. Well, at least on Windows 11 running WSL 2 that seems to work. I now realize that doesn’t work from Windows 10 with WSL. I had better figure this out soon…

Windows 10 running WSL

The terminal type (check the TERM environment variable) is set toxterm-color256. I tried pasting any and all registers which is the standard thing you would do if you use the standard Internet advice. None of it worked for me. I finally realized on my own that – and this harkens back to my old days with the beloved VAX 780 – that if I set the terminal type to vt100 all was good! Seriously. Back in the day we had physical VT100 terminals. Well, before that I think there was a VT52? Then maybe a VT102. VT202 was a big upgrade. Anyway, initially I added the following line to my .bashrc file:

export TERM=vt100

and now I can insert clipboard text the way I always have (mouse right-click) in vi insert mode. This kludge was how we fixed a lot of terminal display issues in the old days. But now I see display from top is messed up! Probably other curses-based apps as well. So two steps forward, one step back. So now what I’ve done is removed that line from .bashrc and put the following lines in my .bash_aliases file:

# DrJ kludge to get vi to work and keep top working
alias top='export TERM=xterm-256color;\top'
alias vi='export TERM=vt100;\vi'

That \top harkens back to an old linux convention where a command preceded by \ invokes a program but ignores defined aliases for that program.

Conclusion

I have offered one possible solution to the can’t insert text from the clipboard problem into my vi: set the TERM environment variable to the old-fashioned vt100. Now I can once again right-click while in insert mode to paste in clipboard text.

This was a very vexing issue for a creature of habit such as me!

References and related

This whole issue came up only when I switched from CentOS 8 to Debian 12 as my back-end server. Believe me, Debian 12 is so superior in so many ways this little setback would never make a material impact in that decision. Here’s the write-up of my upgrade.

Categories
CentOS Debian Linux Raspberry Pi

drjohnstechtalk now runs on a modern OS

Intro

I’m thrilled to announce that the long-running blog drjohnstechtalk.com has now been migrated to a modern back-end operating system. drjohnetchtalk.com is, a far as I know, the only quality-written technical resource on the Internet which is not supported by ads. Instead it runs on a pay-it-forward approach, embracing the spirit of the old Internet before it was ruined by big money.

drjohnstechtalk.com has been providing solutions to obscure tech questions since 2011.

The details

I like to run my own server which I can use for other purposes as well. I think that approach used to be more common. Now it’s harder to find others using it. Anyway, my old hosting environment is a CentOS server. I had hoped it would last me up to 10 years! 10 years is about the duration of long-term support for Redhat linux. It’s a real pain to migrate a WordPress blog with lots of history where it is important to preserve the articles and the permalinks. This article documents the nightmare I put myself through to get that up and running. Before that there was a CentOS 6 server. Then in 2022 – only about two years later – I learned that CentOS was dead! IBM had killed it. I’m over-simplifying here somewhat, but not by much.

So my blog sort of limped on on this unsupported system, getting riskier by the day to run as I was missing out on security patches. Then my companyt accidentally included one of my blogs in a security scan and I saw I had some vulnerabilities. So I upgraded WordPress versions and plugin versions. So with up to date software, the stage was set to migrate to a newer OS. Further motivation was provided by the fact that after the WP upgrade, the pages loaded more slowly. And sometimes the site just collapsed and crashed.

I have come to love Debian linux due to my positive experience with running it on Raspberry Pis and a few other places. It tends to run more recent versions of open source software, for instance. So I chose a Debian linux server. Then I forget where I learned this. Perhaps I asked someone at work which web server to use, but the advice was to use nginx, not apache! This was very new to me as I had never run nginx, not that I was in love with apache.

So, anyway, here I am writing this on my shiny new Debian 12 bookworm server which is running an nginx web server! And wow my site loads so much faster now. It’s really striking…

Running WordPress in a subdirectory with nginx

There always has to be a hard part, right? This was really, really hard. I run WP in the subdirectory blog as you can see from any of my URLs. I must have scoured a dozen sites on how to do it, none of which completely worked for me. So I had to do at least some of the heavy lifting and work out a working config on my own.

Here it is:

# mostly taken from https://www.nginx.com/resources/wiki/start/topics/recipes/wordpress/
# but with some important mods
upstream php {
    server unix:/var/run/php/php8.2-fpm.sock;
}

server {
  listen 443 ssl;

    include snippets/self-signed.conf;


    server_name drjohnstechtalk.com www.drjohnstechtalk.com;

    root /web/drjohns;
    index index.php index.html;


    access_log /var/log/nginx/drjohns.access.log;
    error_log /var/log/nginx/drjohns.error.log;

    client_max_body_size 100M;

# the following section prevents wp-admin from infintely redirecting to itself!
    location /blog/wp-admin {
            root /web/drjohns;
            try_files $uri $uri/ /blog/wp-admin/index.php?$args;
    }

    location /blog {
            root /web/drjohns/blog;
            try_files $uri $uri/ /blog/index.php?$args;
    }
    location ~ \.php$ {
#NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
         include fastcgi_params;
         fastcgi_intercept_errors on;
         fastcgi_pass php;
         fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    }
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg) {
            expires max;
            log_not_found off;
    }
}

I had to add ths svg file type to ignore, the location directive that matches /blog/wp-admin/. I had to define the upstream label as php and refer to that label in fastcgi_pass. I had to figure out my correct version of fastcgi. I tossed out some location directives which weren’t too important to me.

I disabled the wp-hide-login plugin while I grappled with why I was getting first a 404 not found for /blog/wp-admin/, then later, the too many redirects error. But I still had the issue with it disabled. Once I resolved the problem by adding the /blog/wp-admin location directive – I seem to be the only one on the Internet offering this solution and no other solution worked for me! – then I re-enabled the hide login plugin. The other plugins are working I would say.

Firewall?

I gather the current approach to host-based firewall on Debian 12 is to run ufw. A really good article on setting it up is here: https://www.cyberciti.biz/faq/set-up-a-firewall-with-ufw-on-debian-12-linux/

I’m on the fence about it, fearing it might slow my speedy server. But it looks pretty good. So for now I am relying on AWS Network Security Group rules. Did you know you can ask them to increase your max rule quota frmo 20 to 40? Yes, you can. I did and got approved overnight. I have added the Cloudflare ranges.

Cloudflare

I continue to use Cloudflare as reverse proxy, certificate issuer, DNS provider and light security screening. The change to the new server did not alter that. But I needed a new config file to properly report the origin IP address in my access files. The following file does the trick for me. It is up to date as of February 2024, can be placed in your /etc/nginx/conf.d directory and called, e.g., cloudlfare.conf.

# up to date as of 2/2024
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 104.16.0.0/13;
set_real_ip_from 104.24.0.0/14;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 131.0.72.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 2400:cb00::/32;
set_real_ip_from 2606:4700::/32;
set_real_ip_from 2803:f800::/32;
set_real_ip_from 2405:b500::/32;
set_real_ip_from 2405:8100::/32;
set_real_ip_from 2c0f:f248::/32;
set_real_ip_from 2a06:98c0::/29;

real_ip_header CF-Connecting-IP;

The idea is that if the source IP of the HTTP connection to nginx is from the Cloudflare range of IPs, then this must represent a request proxied through Cloudflare and the original IP of the client is in the HTTP header CF-Connecting-IP, which nginx can report on. If not, just use the normal IP from the TCP connection.

Swap space

On CentOS I had to provide some swap space because otherwise apache + mariaDB + WordPress would easily send its cpu soaring. So far I have not had to do that with my new Debian 12! That is great… So I have a t2.small instance with 25 GB of gp2 storage (100 iops). The server is basically running with a 0.00 load average now. I don’t get a lot of traffic so I hope that infrastructure will suffice.

Set the timezone

My Debian system started out in the UTC timezone. This command confirms that:

sudo timedatectl

This command brings up a menu and i can change the timezone to US Eastern:

sudo dpkg-reconfigure tzdata

Automate patching

It hasn’t run yet, but I’m hoping this root crontab entry will automate the system updates:

59 2 * * 0 (date && apt-get update && apt-get upgrade -y) >> /home/admin/hosting/update.log 2>&1

Debian 12 lifecycle

There should be three years of full support plus two more years of long term support for a stable Debian release, if I’ve undrstood it correctly. So I believe I may hope to get five years out of my Bookworm version, give or take. Debian — Debian Releases

Fixing the vi editor

I’ve never really had a problem with vi until this server. I show how I fixed it in this blog post.

Status after a few days – not all positive news

Well after a few days I feel the server response has noticeably slowed. I could not run top because I messed up the terminal with my fix to vi! So in a panic I restarted mariadb which seemed to help performance a lot. I will have to figure out how to monitor for this problem and how best to address it. I’m sure it will return. Here is my monitor.sh script:

#!/bin/bash
# restart mariaDB if home page response becomes greater than one second
curl -m1 -o /dev/null -ksH 'Host:drjohnstechtalk.com' https://localhost/blog/
# if curl didn't have enough time (one sec), its exit status is 28
[ $? -eq 28 ] && (systemctl stop mariadb; sleep 3; systemctl start mariadb; echo mariadb restart at $(date))

I invoke it from root’s crontab every three minutes:

# check that our load time is within reason or else restart mariadb -DrJ 2/24
*/3 * * * * sleep 25;cd /home/admin/hosting; ./monitor.sh >> monitor.log 2>&1

I do love my kludges. I will be on the lookout for a better long-term solution.

Conclusion

The technical blogging web site drjohnstechtalk.com now runs on new infrastructure: Debian 12 running nginx. It is muich faster than before. The migration was moderately painful! I have shared the technical details on how I managed to do it. I hope that, unlike my previous platform of CentOS 8, this platform lasts me for the next 10 years!

References and related

My second article!

nginx’s own advice about how to configure it to run WordPress

Trying to upgrade WordPress brings a thicket of problems

One of many RPi projects of mine: Raspberry Pi light sensor project

ufw firewall for Debian 12

Debian — Debian Releases

Cloudflare, an added layer of security for your web site

IP Ranges | Cloudflare

What happened to insert mode on the latest version of vi?

Categories
Linux

How an ADO pipeline can modify its own repo

Intro

I wanted to run a job on an Azure DevOps pipeline which did a backup of DNS zones on Cloudflare and write the results, in the form of a compressed tar file, into the ADO repository since everyone on the team has access to it and knows how to make a clone of the repo.

My first attempts produced some stunningly bad results. I was wiping out recently created files in the repo and such. That is very undesirable.

The solution

By “stealing with pride” from colleagues and such, I arrived at this AFAIK working solution. Here is the yaml file.

trigger: none

pool:
  name: backup_agents

steps:
# next two lines needed so we can modify the git repo and add our backups
- checkout: self
  clean: true
  persistCredentials: true
  fetchDepth: 1

- script: pip3 install -vvv --timeout 60 -r Cloudflare-backup/requirements.txt
  displayName: 'Install requirements'

- script: python3 backup-all-zones.py
  displayName: 'Run script'
  workingDirectory: $(System.DefaultWorkingDirectory)/Cloudflare-backup
  env:
    CLOUDFLARE_API_TOKEN: $(cloudflare_api_token)
    PYTHONPATH: $(System.DefaultWorkingDirectory)/Cloudflare-backup:$(System.DefaultWorkingDirectory)
- script: |
    git config --global http.sslVerify false
    git config --global user.email "[email protected]"
    git config --global user.name "pipeline"
    cd Cloudflare-backup
    pwd
    ls
    git add backups/zones-*
    git commit -m "adding todays backup files"
    git push origin HEAD:refs/heads/main

schedules:
- cron: "47 23 * * *"
  displayName: Run the script at 23:47 UTC
  branches:
    include:
    - main

I’m not exactly where all the magic happens. I think the section at the top that does the self checkout must be important. Then, obviously, there are the git add/git commit -m/git push commands. I do not claim to understand the origin HEAD:refs/heads/main argument to git push. I just copied it from a working example.

And branches: include -main. I’m not sure what this does either.

I need a few more days of testing, to be really certain, but I no longer am reverting my repo to an old state as I was with my initial attempts which involved doing a git fetch and probably missed the self checkout step as well.

Conclusion

One day I hope to understand git. But that today is not today! nevertheless I got my ADO pipeline to add backup files to its own ADO repository! So that’s cool.

References and related

My own git cheatsheet

Cloudflare python api examples

Categories
Linux Raspberry Pi

Raspberry Pi light sensor project

Intro

I ultimately want to turn off the connected display when it is nighttime and the lights have been turned off. And I want it to turn itself back on during daylight. The reason is because my RPi-driven slideshow is running and somoeone may be sleeping in that room.

This is till a work in progress. Pardon our dust and gibberish. What I’ve already found out is too important to delay publication.

Equipment and skills
How to turn off and on an HDMI port with a Raspberry Pi 4

Honestly this is the most significant thing I have found in this investigation. The methods used for older RPis do not work! In other words you can run vcgencmd display_power 0 and clever variations of that command until you’re blue in the face and the thnig stubbornly won’t turn off. tvservice -o ? Nope. That’ll suggest that command was deprecated and use kmsprint instead.

But I can say as of this writing (Jan ’24) the kms* commands are not mature and do not permit you to turn off the display. kmsprint tells you some stuff, but it does not allow you to set things. Remember RPi is for the education and hobbyist crowd so we have to give them the slack to experiment and try new things, even when they aren’t fully formed.

Instead they give you a way to restore the old commands. Edit the file /boot/config.txt.

Make it look like this:

#Enable DRM VC4 V3D driver
#use the fake kms driver in place of the native kms driver so we can  control hdmi power -DrJ 1/24
##dtoverlay=vc4-kms-v3d
dtoverlay=vc4-fkms-v3d

Reboot. Congrats you are now using the fake kms driver (fkms) and now have compatibility with the old commands. But instead of using tvservice, for my purposes, I think vcgencmd is better because the frame buffer state is not lost.

So now this command will indeed turn off the display:

vcgencmd display_power 0 # turn hdmi display off

vegcmd display_power 1 # turns the HDMI display back on

vcgencmd display_power 0 # turn hdmi display off

Prepare for our light sensor

I really don’t know if I’ll ever get the light sensor to work or not. Anticipating that it will, I have created this GPIO callback routine in python which in my dreams will turn off the HDMI display when the room is dark and turn it back on when the ambient light crosses a threshold. Who knows… But the code is pretty cool because it permits you to play with it as well using software command you send to a GPIO pin.

I call it gpio_basic.py:

import RPi.GPIO as GPIO
import time
import os
import datetime

GPIO.setmode(GPIO.BCM)
channel=4
GPIO.setup(channel, GPIO.IN)
reading = GPIO.input(channel)
print('Initial Reading',reading,flush=True)
old_reading = reading
while True: # infinite loop
    time.sleep(4)
# rising means ambient light went from light to dark
    reading = GPIO.input(channel) # 1 => dark, D0 LED turns off; 0 if light
    if reading == old_reading: continue

# else section where the state has changed
    print('This is a change in state on channel',channel,'at ',datetime.datetime.now(),flush=True)
    print('Reading',reading)
    if reading == 1:
        print('Turning off the HDMI display...',flush=True)
        os.system("vcgencmd display_power 0")
    else:
        time.sleep(6)
        print('Turn on HDMI display...',flush=True)
        os.system("vcgencmd display_power 1")
    old_reading = reading

Now to play with it, in another window get into python and run these commands:

import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)
channel=4
GPIO.setup(channel, GPIO.IN)
GPIO.input(channel)
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_UP)
#or
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

Those commented out lines at the end are the key ones. By executing them you should see the display turn off or on, and your other program should output some verbiage. Now hopefully the sensor will work something like that!

Wiring

Don’t know if this will pan out. I envision wiring (RPi pin #’s on the left):

  • 1 – Vcc (+ 3.3 V voltage)
  • 6 – Gnd (ground)
  • 7 – D0 (digital out)

At this point I don’t see the point of wiring the fourth lead on the diode sensor (which is A0, analog out – the RPi cannot read analog out), but having only three of them wired doesn’t quite seem right either. My electronics knowledge is weak!

So maybe I’ve wasted my money on these sensors, or I need to read them with a microcontroller and use i2c to talk to the RPi (too much effort and too much expense for my taste), but I hope not. Will know soon…

My first sensor, the photodiode, works! But its threshold is near one end of the control, which isn’t so great. But it’s fun to play with. In other words, its low light sensitivity may not be adequate for my purposes where I need to distinguish low ambient light (in a room with only a glowing TV screen, for instance) from even lower ambient light (lights off). Turns out our human eyes are the best measuring devices! Actually this photo diode, properly tuned, is quite good! I believe there is some jitter in the measurements however. So it can jump around during low light a bit. I have to consider how much of a problem that is.

The photodiode has a power led and a light sensor led. They are both way too bright. I suspect they could even create a feedback loop. I covered both with masking tape, leaving room for the adjustment screw.

But much, much worse than the the photodiode is the photoresistor. That at best distinguishes between a decent amount of light, and quite dark. But the transition between the two is sticky (I believe that would be called hysteresis). It will not work at all for my purposes based on my initial testing. It cannot distinguish between low light and very low light no matter where the dial is set.

Since this is all working, we just need to make it permanent by starting at boot time. So in my crontab file I added this line:

# Turn monitor off and on depending on ambien light! - DrJ 1/9/24
@reboot sleep 42; python3 gpio_basic.py > gpio_basic.log 2>&1
RPi4 shown with the slick metal case and the photodiode during daytime
Measure temperature of the CPU and why it matters

vcgencmd measure_temp

With my new fancy aluminum, heat-dissipating case I get around 40.4° C. On my RPi 3, air-cooled it is around 49° C. Why the sudden concern around heat? I’m just beginning to suspect that you know those times when you use the command line and things just seem to freeze? I always just assumed it was a glitch in the WiFi. But maybe it was actually the cpu getting too hot and having to pause itself to avoid burning up. I would see this when transferring files on my RPi 4. so the RPi 4 probably really does run hot unless you take steps to cool it, and the aluminum case in the equipment list above is really cool, ha , ha.

Tips

The command pinout is very useful. Pinout docs here: https://www.raspberrypi.com/documentation/computers/raspberry-pi.html

References and related

This guide talks about explicit support for generic digital input devices, not just a button: https://randomnerdtutorials.com/raspberry-pi-digital-inputs-python/

This talks about built-in pull-up pull-down resistors configurable via software! Who knew?

This might be useful for simple GPIO stuff: https://tutorials-raspberrypi.com/raspberry-pi-gpio-explanation-for-beginners-programming-part-2/

Gory details with gory circuit diagram: https://www.circuits.dk/everything-about-raspberry-gpio/

Running multiple RPi slideshows

How to deal with th GPIO pins using python: https://sourceforge.net/p/raspberry-gpio-python/wiki/Inputs

This project used the RPi GPIO OUTPUT pins to control a power relay device

Appendix A

I started with this code, gpio_callback.py, but the one condition kept getting called when the GPIO pin was reading 1. So I wrtoe gpio_basic.py and use that instead.

import RPi.GPIO as GPIO
import time
import os

GPIO.setmode(GPIO.BCM)
channel=4
GPIO.setup(channel, GPIO.IN)
reading = GPIO.input(channel)
print('Initial Reading',reading)
def my_callback(channel):
# rising means ambient light went from light to dark
    print('This is a edge event callback function!')
    print('Edge detected on channel %s'%channel)
    print('This is run in a different thread to your main program')
    print('Gonna stop that slideshow now...')
    reading = GPIO.input(channel) # 1 => dark, D0 LED turns off; 0 if light
    print('Reading',reading)
    if reading == 1:
        time.sleep(1)
        print('Turning off the HDMI display...')
        os.system("vcgencmd display_power 0")
    else:
        time.sleep(1)
        print('Turn on HDMI display...')
        os.system("vcgencmd display_power 1")

# RISING,FALLING or BOTH. https://sourceforge.net/p/raspberry-gpio-python/wiki/Inputs/
GPIO.add_event_detect(channel, GPIO.BOTH, callback=my_callback)  # add edge detection on a channel
# for testing, use:
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_UP)
# or
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)
time.sleep(31415927) # one year

Categories
Debian Linux Raspberry Pi

My favorite bash scripting tips

Intro

The linux bash shell is great and very flexible. I love to use it and have even installed WSL 2 on my PCs so I can use it as much as possible. When it comes to scripting it’s not exactly my favorite. there is so much history it has absorbed that there are multiple ways to do everything: the really old way, the new way, the alternate way, etc. And your version of bash can also determine what features you can use. nevertheless, I guess if you stick to the basics it makes sense to use bash for simple scripting tasks.

So just like I’ve compiled all the python tips I need for writing my simple python scripts in one convenient, searchable page, I will now do the same for bash. No one but me uses it, but that’s fine.

Iterate (loop) over a range of numbers

END=255 # for instance to loop over an ocetet of an IP address
for i in $(seq 1 $END); do
  echo $i
done
# But if it's OK to just hard-wire start and end, then it's simpler to use:
for i in {1..255}; do echo $i; done

Infinite loop
while /bin/true; do...done

You can always exit to stop it.

Sort IPs in a sensible order

$ sort -n -t . -k1,1 -k2,2 -k 3,3 -k4,4 tmp

What directory is this script in?

DIR=$(cd $(dirname $0);pwd);echo$DIR

Guarantee this script is interpreted (run) by bash and not good ‘ole shell (sh)!
if [ ! "$BASH_VERSION" ] ; then
  exec /bin/bash "$0" "$@"
  exit
fi
Count total occurrences of the word print in a bunch of files which may or may not be compressed, storing the output in a file

print=0
zgrep -c print tst*|cut -d: -f2|while read pline; do prints=$((prints + pline));echo $prints>prints; done

Note that much of the awkwardness of the above line is to get around issues I had with variable scope.

Legal characters in variable names

Don’t use _ as you might in python! Stick to alphanumeric, but also do not begin with a number!

Execute a command

I used to use back ticks ` in the old days. parentheses is more visually appealing:

print1=$(cat prints)

Variable type

No, variables are not typed. Everything is treated as a string.

Function definition

Put function definitions before they are invoked in the script. Invocation is by plain name. function syntax is as in the example.

sendsummary() {
# function execution statements go here, then close it out
} # optionally with a comment like end function sendsummary
sendsummary # invoke our sendsummary function
Indentation

Unlike python, line indentation does not matter. I recommend to indent blocks of code two spaces, for example, for readability.

Booleans and order of execution
[[ "$DEBUG" -eq "1" ]] && echo subject, $subject, intro, "$intro"

The second statement only gets executed if the first one evaluated as true. Now a more complex example.

[[ $day -eq $DAY ]] || [[ -n “$anomalies” ]] && { statements…}

The second expressions get evaluated if the first one is false. If either the first or second expressions are true, then the last expression — a series of statements in what is essentially an unnamed function, hence the enclosing braces — gets executed. The -n is a test to see of length of a string is non-zero. See man test.

Conditionals

Note that clever use of && and || can in many cases obviate the need for a class if…then structure. But you can use if thens. An if block is terminated by a fi. There is an else statement as well as an elif (else if) statement.

grep conditionals
ping -c1 8.8.8.8|grep -iq '1 received'
[ $? -eq 0 ] && echo this host is alive

So the $? variable after grep is run contains 0 if there was a match and 1 if there was no match. -q argument puts grep in “quiet” mode (no output).

More sophisticated example testing exit status and executing multiple commands

#!/bin/bash
# restart mariaDB if home page response becomes greater than one second
curl -m1 -ksH 'Host:drjohnstechtalk.com' https://localhost/blog/ > /dev/null
# if curl didn't have enough time (one sec), its exit status is 28
[ $? -eq 28 ] && (systemctl stop mariadb; sleep 3; systemctl start mariadb; echo mariadb restart at $(date))

Note that I had to group the commands after the conditional test with surrounding parentheses (). That creates a code block. Without those the semicolon ; would have indicated the end of the block! A semicolon ; separates commands. Further note that I nested parentheses and that seems to work as you would hope. also note that STDOUT has been redirected by the greater than sign > to /dev/null in order to silently discard all STDOUT output. /dev/null is linux-specific. The windows equivalent, apparently, is nul. Use curl -so nul suppress output on a Windows system.

One square bracket or two?

I have no idea and I use whatever I get to work. All my samples work and I don’t have time to test all variations.

Variable scope

I really struggled with this so I may come back to this topic!

Variable interpolation

$variable will suffice for simple, i.e., one-word content. But if the variable contains anything a bit complex such as words separated by spaces, or containing unusual characters, better go with double quotes around it, “$variable”. And sometimes syntactically throw in curly braces to separate it from other elements, “${variable}”

Eval
eval="ls -l"
$eval # executes ls -l
Shell expansion
mv Pictures{,.old} # renames directory Pictures to Pictures.old
Poor man’s launch at boot time

Use crontab’s @reboot feature!

@reboot sleep 25; ./recordswitch.sh > recordswitch.log 2>&1

The above expression also shows how to redirect standard error to standard out and have both go into a file.

Use extended regular expressions, retrieving a positional field using awk, and how to subtract (or add) two numbers
t1=`echo -n $line|awk '{print $1}'` 
t2=`echo -n $line|awk '{print $4}'` 
# test for integer inputs 
[[ "$t1" =~ ^[0-9]+$ ]] && [[ "$t2" =~ ^[0-9]+$ ]] && downtime=$(($t1-$t2))

Oops, I used the backticks there! I never claim that my way is the best way, just the way that I know to work! I know of a zillion options to add or subtract numbers…

Get last field using awk
echo hi.there.111|awk -F\. '{print $NF}' # returns 111
Why do assignments have no extra spaces?

It simply doesn’t work if you try to put in spacing around the assignment operator =.

Divert stdout and stderr to a file from within the script
log=/tmp/my-log.log
exec 1>$log 
exec 2>&1
Lists, arrays amd dictionary variables

I don’t think bash is for you if you need these types of variables.

Formatted date

date +%F

produces yyyy-mm-dd, i.e., 2024-01-25

Poor man’s source code versioning

The old EDT/TPU editor on VAX used to do this automatically. Now I want to save a version of whatever little script I’m currently working on in the ~/tmpFRI (if it’s Friday) directory to sort of spread out my work by day of the week. I call this script cpj so it’s easy to type:

#!/bin/bash
# save file using sequential versioning to tmp area named after this day - DrJ
DIR='~'/tmp$(date +%a|tr '[a-z]' '[A-Z]') # ~/tmp + day of the week, e.g., FRI
DIRREAL=$(eval "echo $DIR") # the real diretory we need
mkdir -p $DIRREAL
for file in $*; do
  res=$(ls $DIRREAL|egrep "$file"'\.[0-9]{1,}$') # look for saved version numbers of this filename
  if test -n "$res"; then # we have seen this file...
    suffix=$(echo $res|awk -F\. '{print $NF}')  # pull out just the number at the end
    nxt=$(($suffix+1)) # add one to the version number
    saveFile="${file}"."${nxt}"
  else # new file to archive or no versioned number exists yet
    [[ -f $DIRREAL/$file ]] && saveFile="$file".1
    [[ -f $DIRREAL/$file ]] || saveFile=""
  fi
  cp "$file" $DIRREAL/"$saveFile"
  [[ -n $saveFile ]] && target=$DIR/"$saveFile"
  [[ -n $saveFile ]] || target="$DIR"
  echo copying "$file" to "$target"
done

It is a true mis-mash of programming styles, but it gets the job done. Note the use of eval. I’m still wrapping my head around that. Also note the technique used to upper case a string using tr. Note the use of extended regular expressions and egrep. Note the use of tilde ~ expansion. I insist on showing the target directory as ~/tmpSAT or whatever because that is what my brain is looking for. Note the use of nested $‘s.

Now that cpj is in place I occasionally know I want to make that versioned copy before I launch the vi editor, so I created a vij in my bash alias file thusly:

vij () { cpj "$@";sleep 1;vi "$@"; }

Conclusion

I have documented here most of the tecniques I use from bash to achieve simple yet powerful scripts. My style is not always top form, but as I learn better ways I will adopt and improve.

Categories
Linux Raspberry Pi

Multiple Raspberry Pi photo frames

Intro

I have previously shared my work on displaying pictures in a nice slideshow from a Google Drive to an HDMI monitor. All these years later, it is still working and every day we see a new slideshow or randomly yet thoughtfully chosen pictures.

Building on this, today I extended this solution to display these pictures on a second monitor in a different room.

Equipment
  • RPi 4 (what I happened to have around. RPi 3 would also be fine)
  • HDMI monitor
  • Raspberry Pi Lite OS
The details

I needed to install fbi (sudo apt-get install fbi).

I needed to copy over m4.pl from the primary display, and for good measure black.jpg.

The general idea is to copy the pics over to the second display once per day.

I call this program copyslideshow.sh.

#!/bin/bash
# copy pictures from primary source
rm -rf Pictures.old
mv Pictures{,.old}
sshpass -p raspberry scp -r [email protected]:mediashow .
sshpass -p raspberry scp -r [email protected]:Pictures .
./m4.pl  $HOME/mediashow >> m4.log 2>&1

Yeah so I hard-coded the RPi password which is still set to the default. I’m willing to take the risk

Then in crontab I added this line:

# get yesterday's pictures!
1 5 * * * killall m4.pl; ./copyslideshow.sh > copyslideshow.log 2>&1

Note that these displays are not synced. That would be a whole ordeal. In fact we thought it would be cool to display different pictures. So the second monitor will be showing yesterday’s slideshow from the main monitor.

Automating turn-on, turn-off of the HDMI display based on the ambient room light

Since this second slideshow is in a bedroom, I wanted to have it turn off when the lights were out, and turn back on again during daylight. This was a really interesting challenge for me as I got to use an inexpensive external sensor with my RPi. And I got it to work, and it works quite well if I say so myself. That’s all written up in this post.

Reference and related

I have written many variations on this same topic. I guess this is a good one.

How I automatically turn on the HDMI display in the morning and turn it off again at night.

Categories
Linux

Linux tip: How to sort a list of IP addresses

Intro

Sorting a list of IPs should be easy using some linux utility or another, right? It is, and here is how to do it.

The details

Say your list of IPs, one per line, is contained in the file tmp. Then the proper sort command is:

$ sort -n -t . -k1,1 -k2,2 -k 3,3 -k4,4 tmp

If you want things in descending order just do a -nr instead of -n at the beginning.

This sort solves the problem that for instance 9 is evaluated as being greater than 115, for instance!

References and related

More shell tricks can be gleaned from the way I solved this NPR puzzle

Categories
Firewall Linux Network Technologies

The IT Detective Agency: the case of the mysterious ICMP host administratively prohibited packets

Intro

I haven’t published a new case in a while, not for lack of cases, but more that they they all fall into something I’ve already written about. But today there is definitely something new.

Some details

Thousandeyes agent-to-agent communication was generally working for all our enterprise agents after fixing firewall rules, etc, except for this one agent hosted in Azure US East. Was it something funny about the firewalls on either side of the vpn tunnel to this cloud? Ping tests were working. But a connection to tcp port 49153, which is used for agent-to-agent communication gave a response in the form of an ICMP type 3 code 10 packet which said something like host administratively prohibited. What?

The Cisco TAM suggested to look at iptables. I did a listing with iptables -L. The output is pretty long and I’m not experienced looking at it. Nothing much jumped out at me, but I did note the presence of this line:

REJECT     all  —  anywhere             anywhere             reject-with icmp-host-prohibited

in a couple of the chains, which seemed suspicous.

An Internet search pointed towards firewalld since the agent is a Redhat 7.9 system. Indeed firewalld was running:

systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2023-10-12 15:26:25 UTC; 5h 45min ago

The suggestion is to test with firewalld disabled. Indeed this produced correct results – no more ICMP packets back.

But it’s probably a good security measure to run firewalld, so how to modify it? This note from Redhat was particularly helpful in learning how to add a rule to the firewall. I pretty much just needed to do this to permanently add my rule:

firewall-cmd –add-port 49153/tcp –permanent

Afterwards the agent-to-agent tests began to be run successfully.

Which runs first, tcpdump or firewalld?

tcpdump

This is a good question to ask because if the order had been different, and who knows, you might have your packets dropped before you ever see them on tcpdump. But tcpdump seems to get a pretty clean mirror of what the network interface gets before application or kernel processing.

The new equivalent to netstat -an

If I want to see the listening processes in Redhat I might do a

ss -ln

In the old days I memorized using netstat -an, but that is now frowned upon.

Conclusion

We solved a case where tcp packets were getting returned with an ICMP packet which basically said: prohibited. This was due to the host, a Redhat 7 system, having restricted ports due to firewalld running. Once firewalld was modified this traffic was permitted and Thousandeyes Tests ran successfully. We also proved that tcpdump runs before firewalld.

References and related

How to add rule to firewalld on Redhat-like systems.

Categories
Admin Apache Linux

Cloudflare: an added layer of protection for your personal web site

Intro

I was looking at what Cloudflare could do for my web site. A colleague pointed out that they have a free usage tier which supplies a web application firewall and some anti-bot measures. I checked it out and immedaitely signed up!

The details

What Cloudflare is supplying at no cost (for personal web sites like mine) is amazing. It’s not just a world-class dns service. That would already be amazing. Run dnscheker.org against drjohnstechtalk.com and you will see several different IPs mentioned around the world- just like the big guns! I also get for free some level of mitigation against dns-based attackes.

Web site protections

I don’t fully understand their products so I don’t know what level of protections I am getting in the free tier, but there are at least some! They say they’ve blocked 10 requests in the last few days

Web usage stats

I have to admin using raw linux tools against my apache access file hasn’t bee n the most illuminating until now. Now that I use Cloudflare I get a nice visual presentation showing where (which country) my visitors came from, where the bots come from, how much data was transmitted.

Certificate for HTTPS

Cloudflare automatically takes care of the web site certificate. I had to do nothing at all. So now I can forget my call out to LetsEncrypt. I wonder if GoDaddy is still charging $69 annually for their certificates.

Acceleration

Yeah my web site just feels faster now since the switch. It just does. And Cloudflare stats say that about 30% of the content has been served from their cache – all with zero setup effort on my part! I also believe they use certain tcp acceleration techniques to speed things up.

Cache

And Cloudflare caches some of my objects to boost performance. Considering that I pay for data transfer at Amazon AWS, it’s a fair question to ask if this caching could even be saving me money? I investigated this and found that I get billed maybe $ .02 per GByte, and in a busy month I might use .8 GB or so, so $ .02 per month. So I might occasionally save a penny or so – nothing substantial though!

geoDNS

Even with this free tier you get some geoDNS functionality for free, namely, visitors from around the world will see an IP address which is geographically close to where they are, bossting their performance when using your site. Stop to think about that. That’s a whole lot of infrastructure sophistication that they’re just giving you for free!

Why are they giving this much away?

I think they have the noble aim of improving the security posture of the Internet writ large. Much as letsencrypt greatly accelerated the adoptipon of web page encyrption (https) by making certificates free, Cloudflare hopes to accelerate the adoption of basic security measures for every web site, thereby lifting the security posture of the Internet as a whole. Count me as a booster!

What’s their business model. How will they ever make money?

Well, you’re only supposed to use the free tier for a personal web site, for one. My web sites don’t really have any usage and do not display ads so I think I qualify.

More importantly, the free security protections and acceleration are a kind of teaser and the path to upgrading to profesisonal tier is very visibly marked. So they’re not 100% altruistic.

Why I dislike GoDaddy

Let’s contrast this with offerings from GoDaddy. GoDaddy squeezes cents out of you at every turn. They make it somewhat mysterious what you are actually paying for so they’re counting on fear of screwing up (FOSU, to coin a term). After all, except for the small hit to your wallet, getting that upgraded tier – whois cloaking, anyone? – might be what you need. Who knows. Won’t hurt, right? But I get really tired of it. Amazon AWS is perhaps middle tier in this regards. They do have a free tier virtual server which I used initially. But it really doesn’t work except as a toy. My very modest web site overwhlemed it on too many occasions. So, basically useless. Everything else: you pay for it. But somehow they’re not shaking the pennies out of you at every turn unlike GoDaddy. And AWS even shows you how to optimize your spend.

How I converted my live site to Cloudflare

After signing up for Cloudflare I began to enter my dns domains, e.g., drjohnstechtalk.com, johnstechtalk.com, plsu a few others. They explained how at GoDaddy I had to update the nameserver records for these domains, which I did. Then Cloudflare has to verify these updates. Then my web sites basically stopped working. So I had to switch the encryption mode to full. This is done in Web sites > drjohnstechtalk.com > SSL/TLS > Overview. This mode encrypts the back-end data to my web server, but it accepts a self-signed certificate, no matter if it’s expired or not and no matter who issued it. That is all good because you still get the encrypted channel to your content server.

Then it began to work!

Restoring original visitor IPs to my apache web server logs

Very important to know from a technical standpoint that Cloudflare acts as a reverse proxy to your “content server.” Knowing this, you will also know that your content server’s apache logs get kind of boring because they will only show the Cloudflare IPs. But Cloudflare has a way to fix that so you can see the original IPs, not the Cloudlfare IPs in your apache logs.

Locking down your virtual server

If Internet users can still access the web server of your virtual server directly (bypassing Cloudflare), your security posture is only somewhat improved. To go further you need to use a local firewall. I debated whether to use AWS Network Security Groups or iptables on my centos virtual server. I went with iptables.

I lossely followed this developer article. Did I mention that Cloudflare has an extensive developer community? https://developers.cloudflare.com/fundamentals/get-started/setup/allow-cloudflare-ip-addresses/

Actually I had to install iptables first because I hadn’t been using it. So my little iptables script I created goes like this.

#!/bin/bash
# from https://developers.cloudflare.com/fundamentals/get-started/setup/allow-cloudflare-ip-addresses/
# For IPv4 addresses
curl -s https://www.cloudflare.com/ips-v4|while read ip; do
 echo adding $ip to iptables restrictions
 iptables -I INPUT -p tcp -m multiport --dports http,https -s $ip -j ACCEPT
done
ip=127.0.0.1
iptables -I INPUT -p tcp -m multiport --dports http,https -s $ip -j ACCEPT
# maybe needed it just once??
#iptables -A INPUT -p tcp -m multiport --dports http,https -j DROP
# list all rules
iptables -S

I believe I just need to run it the one time, not, e.g., after every boot. We’ll soon see. The output looks like this:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -s 127.0.0.1/32 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 172.64.0.0/13 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 104.24.0.0/14 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 104.16.0.0/13 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 162.158.0.0/15 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 198.41.128.0/17 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 197.234.240.0/22 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 188.114.96.0/20 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 190.93.240.0/20 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 108.162.192.0/18 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 141.101.64.0/18 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 103.31.4.0/22 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 103.22.200.0/22 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 103.21.244.0/22 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -s 173.245.48.0/20 -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 80,443 -j DROP

Note that this still leaves ssh open, but that’s ok since it is locked down via Network Security Group rules. No urgent need to change those.

Then I made sure that direct access to my content server freezes, which it does, and that access through the official DNS channels which use Cloudflare still works, which it did. So… all good. The setup was not hard at all. But since I have several hosted web sites for the iptables to make any sense I had to be sure to migrate all my hosted sites over to Cloudflare.

Not GoDaddy

I was dreading migrating my other zones (dns domains) over to Cloudflare. Still being in the GoDaddy mindframe I figured, sure, Cloudflare will permit me one zone for free, but then charge me for a second one.

So I plunged ahead. johnstechtalk.com. No charge!

And a third one: vmanswer.com. Also no charge!

And a fourth, and a fifth and a sixth.

I thought perhaps five will be the threshold. But it wasn’t. I only have six “zones” as Cloudflare now calls them. But they are all in my account and all free. Big relief. This is like the anti-GoDaddy.

DNS changes

Making DNS changes is quite fast. The changes are propagated within a minute or two.

api access

Everything you can do in the GUI you can do through the api. I had previously created and shared some model python api scripts.

ipv6

As if all the above weren’t already enough, I see Cloudflare also gives my web site accessibility via ipv6:

$ dig +short aaaa drjohnstechtalk.com

2606:4700:3035::ac43:ad17
2606:4700:3031::6815:3fea

I guess it’s accessible through ipv6 but I haven’t quite proven that yet.

Mail forwarding

I originally forgot that I had set up mail forwarding on GoDaddy. It was one of the few free things you could get. I think they switched native Outlook or something so my mail forwarding wasn’t working. On a lark I checked if Cloudflare has complementary mail forwarding for my domains. And they do! So that’s cool – another free service I will use.

Sending mail FROM this Cloudflare domain using your Gmail account

This is more tricky than simple mail forwarding. But I think I’ve got it working now. You use Gmail’s own server (smtp.gmail.com) as your relay. You also need to set up an app password for Gmail. Even though you need to specify a device such as Windows, it seems once enabled, you can send from this new account from any of your devices. I’ve found that you also need to update your TXT record (see link below) with an expanded SPF information:

v=spf1 include:_spf.google.com include:_spf.mx.cloudflare.net ~all

In words it means the Google and Cloudflare sending servers are authorized to sends emails with this domain in the sender field, mail from elsewhere will be marked.

Even after all that I wasn’t seeing my sent message at work where Microsoft 365 is in use. It landed in the Junk folder! Why? The sending email “appears similar to someone who previously sent you email, but may not be that person.” Since I am a former mail admin I am sympathetic to what they’re trying to do – help hapless users avoid phishing; because it’s true – the characters in my test email did bear similarities to my regular email. My regular email is first_name.last_name @ gmail.com, while mail from this domain was first_name @ last_name + s .com Mail sent to a fellow Gmail user suffered no such fate however. Different providers, different approaches. So I can accept that. Once it’s set up you get a drop-down menu of sending addresses every time you compose a new message! The detailed instructions are at the Cloudflare community site.

Cost savings using Cloudflare

Suppose like me you only use GoDaddy as your registrar and get all your other services in some other way. Well, Cloudflare began to pitch me on transferring my domains to them. I thought, Aha, this is the moment they will make money off me. So I read their pitch. Their offer is to bill me for the charges they incur from ICANN or wherever, i.e., pass-through charges without any additional middleman overhead. It’s like, what? So let’s say at GoDaddy I pay $22 per year per domain. Well with Cloudflare I’d be paying something like $10 per year. For one domain I wouldn’t bother, but since I have more than five, I will be bothering and gladly leaving GoDaddy in the dust. I have just transferred the first two domains. GoDaddy seems to drag out the process as long as possible. I found I could expedite it by approving the transfer in the GoDaddy portal (https://dcc.godaddy.com/control/transfers). The trick there is that that one URL looks very different depending on whether or not a domain transfer is pending. If GoDaddy perceives a domain transfer has been initiated by an other registrar, it will show that page with a Transfer In and Transfer Out tabs. Just select Transfer Out and approve your domain for transfer. Then the transfer happens within five minutes. Otherwise that page is shown with no possibility to do a transfer out. So I guess you have to be patiennt, refresh it, or I don’t know what to get it to draw correctly. Once approved in the GoDaddy transfer out portal, Cloudflare had them within 5 minutes. It’s not super-easy to do a transfer, but also not impossble.

In typical GoDaddy style, executing a domain transfer to another registrar seems essentially impossible if you use their latest Domain portfolio app. Fortunately I eventually noticed the option to switch from “beta” to the old Domain manager, which still has the option and looks a bit more like their documentation. I’ve generated auth codes and unlocked, etc. And I even see the correct domain status (ok as opposed to client transfer prohibited) when I do a whois, but now Cloudflare, which is usually so quick to execute, seems to be lagging in recognizing that the domains have been unlocked and suggests to check back in some hours. Weird. The solution here was to provide my credit card info. Even 12 hours later I was having this trouble where it said none of my domains were eligible for transfer. As soon as I provided my payment information, it recognized two of my domains as eligible for transfer. In other cases Cloudflare recognized that domains were unlocked in a matter of 15 minutes or so. It may help to first unlock the domain in GoDaddy, then to view it in Cloudflare. Not sure.

A plug for GoDaddy

As my favorite sport seems to be bashing GoDaddy I wanted to balance that out and say a few kind words about them. Someone in my houisehold just started a job with a startup who uses GoDaddy. It provides desktop Outlook Email, MS Teams, Sharepoint, helps with consulting, etc. And on day one this person was up and running. So if you use their services, they definitely offer value. My issue is that I tried to restrict my usage to just one service – domain registrar – and they pushed me to use it more extensively, which I resisted. But for a small business which needs those thnigs, it’s fine.

How many domains are you sharing your IP with?

The thnig with Cloudflare is that they assign you to a couple of their IP addresses, often beginning with either 172.67 or 104…. . Now did you ever wonder with how many other web sites you’re sharing those IPs? If not, you should! I found a tool that provides the answer: https://dnslytics.com/ So for this free tier they seem to keep the number around 500 unique domains per IP! Yes that’s a lot, but I’d only be concerned if there was evidence of service degradation, which so far I have not seen. What’s nice about the dnsyltics site is that it lists a few of the domains – far from all of them, but at least it’s 20 or 30 – associated with a given IP. That can be helpful during truobleshooting.

Conclusion

What Cloudflare provides for protective and performance services represents a huge forward advance in the state of the art. They do not niggle you for extra charges (entice is more the word here) for Fear of Screwing Up.

All in all, I am amazed, and I am something of an insider – a professional user of such services. So I heartily endorse using Cloudflare for all personal web servers. I have not been sponsored or even in contact with Cloudflare, by the way!

References and related

Cloudlfare tip: Restoring original visitor IPs to your apache web server.

Locking your virtual server down to just Cloudflare IPs: https://developers.cloudflare.com/fundamentals/get-started/setup/allow-cloudflare-ip-addresses/

Using the Cloudflare python api: working examples

Sending Gmail with your Cloudlflare domain as sending address

Cloudflare’s analysis of the exploit HTTP/2 Rapid Reset is extremely detailed. See https://blog.cloudflare.com/zero-day-rapid-reset-http2-record-breaking-ddos-attack/ and https://blog.cloudflare.com/technical-breakdown-http2-rapid-reset-ddos-attack/ .

I remember being so excited to discover free certificates from LetsEncrypt.

A good explanation of SPF records

Turn an IP addres into a list of associated domain names: https://dnslytics.com/

Categories
Linux Perl Python SLES

Using syslog within python

Intro

We created a convention where-in our scripts log to syslog with a certain style. Originally these were Perl scripts but newer scripts are written in python. My question was, how to do in python what we had done in Perl?

The details

The linux system uses syslog-ng. In /etc/syslog-ng/conf.d I created a test file 03drj.conf with these contents:

destination d_drjtest { file("/var/log/drjtest.log"); };
filter f_drjtest{ program("drjtest"); };
log { source(s_src); filter(f_drjtest); destination(d_drjtest); flags(final); };

So we want that each of our little production scripts has its own log file in /var/log/.

The python test program I wrote which outputs to syslog is this:

[

import syslog
syslog.openlog('jhtest',syslog.LOG_PID,facility=syslog.LOG_LOCAL0)
syslog.syslog(syslog.LOG_NOTICE,'[Notice] Starting')
syslog.syslog(syslog.LOG_ERR,'[Error] We got an error')
syslog.syslog(syslog.LOG_INFO,'[Info] Just informational stuff')

Easy, right? Then someone newer to python showed me what he had done – not using syslog but logger, in which he accomplished pretty much the same goal but by different means. But he had to hard-code a lot more of the values and so it was not as elegant in my opinion.

In any case, the output is in /var/log/drjtest.log which look like this after a test run:

Jul 24 17:45:32 drjohnshost drjtest[928]: [Notice] Starting
Jul 24 17:45:32 drjohnshost drjtest[928]: [Error] We got an error
Jul 24 17:45:32 drjohnshost drjtest[928]: [Info] Just informational stuff
OSes using rsyslog

Today I needed to make this style of logging work on a new system which was running rsyslog. The OS is SLES v 15. For this OS I added a file to /etc/rsyslog.d called drjtest.conf with the contents:

if ($programname == 'drjtest' ) then {
        -/var/log/drjtest.log
        stop
}

But the python program did not need to change in any way.

Conclusion

We show how to properly use the syslog facility within python by using the syslog package. It’s all pretty obvious and barely needs to be mentioned, except when you’re just statring out you want a little hint that you may not find in the man pages or the documentation at syslog-ng.

References and related

I have a neat script which we use to parse all these other scripts and give us once a week summary emails, unless and error has been detected in which case the summary email goes out the day after a script has reported an error. It has some pretty nice logic if I say so myself. Here it is: drjohns script checker.