Categories
Admin Consumer Interest Consumer Tech Firewall Home Computing Linux Scams Security Spam Web Site Technologies

Types of Cyberattacks and other terms from the world of cyber security

Intro

It’s convenient to name drop different types of cyber attacks at a party. I often struggle to name more than a few. I will try to maintain a running list of them.

But I find you cannot speak about cybersecurity unless you also have a basic understanding of information technology so I am including some of those terms as well.

As I write this I am painfully aware that you could simply ask ChatGPT to generate a list of all relevant terms in cybersecurity along with their definitions – at least I think you could – and come up with a much better and more complete list. But I refuse to go that route. These are terms I have personally come across so they have special significance for me personally. In other words, this list has been organically grown. For instance I plowed through a report by a major vendor specializing in reviewing other vendor’s offerings and it’s just incredible just how dense with jargon and acronyms each paragragh is: a mother lode of state-of-the-art tech jargon.

Credential Stuffing Attack

I.e., password re-use. Takes advantage of users re-using passwords for different applications. Nearly three of four consumers re-use password this way. Source: F5. Date: 3/2024

Password spraying

A type of attack in which the threat actor tries the same password with multiple accounts, until one combination works. 

Supply Chain attack
Social Engineering
Hacking
Hacktivist

I suppose that would be an activitst who uses hacking to further their agenda.

Living off the land
Data Breach
Keylogger
Darknet
Captcha
Click farms
Jackpotting

This is one of my favorite terms. Imagine crooks implanted malware into an ATM and were able to convince it to dispense all its available cash to them on the spot! something like this actually happened. Scary.

Overlay Attack

Example: When you open a banking app on your phone, malware loads an HTML phishing page that’s designed to look just like that particular app and the malware’s page is overlaid on top.

Skimmer
bot
Anti-bot, bot defense
Spoofing
Mitigation
SOC
Selenium (Se) or headless browser
WAF
Obfuscation
PII, Personally Identifiable Information
api service
Reverse proxy
Inline
endpoint, e.g., login, checkout
scraping
Layer 7
DDOS
Carpet bombing DDOS attack

Many sources hitting many targets within the same subnet. See:

https://www.a10networks.com/blog/carpet-bombing-attacks-highlight-the-need-for-intelligent-and-automated-ddos-protection/#:~:text=Carpet-bombing%20attacks%20are%20not,entire%20CIDR%20or%20multiple%20ASNs.

SYN flood
DOS
Visibility
Automation
Token
Post
JavaScript
Replay
Browser Fingerprint
OS
Browser
GDPR
PCI DSS
AICPA Trust Services
GUI
(JavaScript) Injection
Command Injection
Hotfix
SDK
URL
GET|POST Request
Method
RegEx
Virtual Server
TLS
Clear text
MTTR
RCA
SD-WAN
PoV
PoC
X-Forwarded-For
JSON
Client/server
Threat Intelligence
Use case
Carding attack
WebHook
Source code
CEO Fraud
Phishing
Vishing

(Voice Phishing) A form of cyber-attack where scammers use phone calls to trick individuals into revealing sensitive information or performing certain actions.

Business email compromise (BEC)
Deepfakes
Threat Intelligence
Social engineering
Cybercriminal
SIM box
Command and control (C2)
Typo squatting
Voice squatting

A technique similar to typo squatting, where Alexa and Google Home devices can be tricked into opening attacker-owned apps instead of legitimate ones.

North-South
East-West
Exfiltrate
Malware
Infostealer
Obfuscation
Antivirus
Payload
Sandbox
Control flow obfuscation
Indicators of Compromise
AMSI (Windows Antimalware Scan Interface)
Polymorphic behavior
WebDAV
Protocol handler
Firewall
Security Service Edge (SSE)
Secure Access Service Edge (SASE)
Zero Trust

Zero Trust is a security model that assumes that all users, devices, and applications are inherently untrustworthy and must be verified before being granted access to any resources or data.

Zero Trust Network Access (ZTNA)
Zero Trust Edge (ZTE)
Secure Web Gateway (SWG)
Cloud Access Security Broker (CASB)
Remote Browser Isolation (RBI)
Content Disarm and Reconstruction (CDR)
Firewall as a service
Egress address
Data residency
Data Loss Prevention (DLP)
Magic Quadrant
Managed Service Provider (MSP)
0-day or Zero day
User Experience (UX)
Watermark
DevOps
Multitenant
MSSP
Remote Access Trojan (RAT)
SOGU

2024. A remote access trojan.

IoC (Indicators of Compromise)
Object Linking and Embedding
(Powershell) dropper
Backdoor
TTP (Tactics, Techniques and Procedures)
Infostealer
Shoulder surfing
Ransomware
Pig butchering

This is particularly disturbing to me because there is a human element, a foreign component, crypto currency, probably a type of slave trade, etc. See the Bloomberg Businessweek story about this.

Forensic analysis
Attack vector
Attack surface
Economic espionage
Gap analysis
AAL (Authentication Assurance Level)
IAL (Identity Assurance Level)
CSPM (Cloud Security Posture Management)
Trust level
Remediation
Network perimeter
DMZ (Demilitarized zone)
Defense in depth
Lateral movement
Access policy
Micro segmentation
Least privilege
Elevated privileges
Breach
Intrusion
Insider threat
Cache poisoning

I know it as DNS cache poisoning. If an attacker manages to fill the DNS resolver’s cache with records that have been altered or “poisoned.”

Verify explicitly
Network-based attack
Adaptive response
Telemetry
Analytics
Identity Provider (IDP)
Consuming entity
Behavior analysis
Authentication
Authorization
Real-time
Lifecycle management
Flat network
Inherent trust
Cloud native
Integrity
Confidentiality
Data encryption
EDR (Endpoint Detection and Response)
BSI (Bundesamt für Sicherheit in der Informationstechnik)

German Federal Office for Information Security (Bundesamt für Sicherheit in der Informationstechnik)

ICS (Industrial Control System)
Reverse shell

A text-based interfaces that allow for remote server control.

Crypto Miner
A RCE (Remote Code Execution)
Threat Actor
APT (Advanced Persistent Threat)
Compromise
Vulnerability
Bug
Worm
Remote Access VPN (RAVPN)
XDR (Extended Detection and Response)
SIEM (Security Information and Event Management)
User Entity Behavior Analytics (UEBA)
Path traversal vulnerability

An attacker can leverage path traversal sequences like “../” within a request to a vulnerable endpoint which ultimately allows access to sensitive files like /etc/shadow.

Tombstoning
Post-exploit persistence technique
MFA bomb

Bombard a user with notifications until they finally accept one.

Use-after-free (UAF)

use-after-free vulnerability occurs when programmers do not manage dynamic memory allocation and deallocation properly in their programs.

Cold boot attack

A cold boot attack focuses on RAM and the fact that it is readable for a short while after a power cycle.

Famous named attacks

Agent Tesla
Cloudbleed
Heartbleed
log4j
Morris Worm

Explanations of exploits

Famous attackers

APT29 (Cozy Bear)

A Russia-nexus threat actor often in the news

Volt Typhoon

2024. A China-nexus threat actor

IT terminology

Active Directory
BGP (Border Gateway Protocol)
Browser
Data at rest
Data in motion
Data Remanence

The residual representation of data that remains even after attempting to erase or initialize RAM.

DDI (DNS, DHCP and IP address management)
DLL
DLP (Data Loss Prevention)
DoH (DNS over HTTPS)
Domain
Enhanced Factory Reset (EFR)
Exact Data Matching (EDM)
ICS
Link
Mandiant
Modbus protocol
.NET
NSA (National Security Agency)
OCR (Optical Character Recognition)
OT (Operational Technology)
Patch
PaaS (Platform as a Service)
PLC (programmable logic controller)
Portable Executable (PE)
Private Cloud
Proof of Concept (POC)
Ray

An open-source unified compute framework used by the likes of OpenAI, Uber, and Amazon which simplifies the scaling of AI and Python workloads, including everything from reinforcement learning and deep learning to tuning and model serving.

Redirect
Retrieval-Augmented Generation (RAG)
SaaS (Software as a Service)
SMTP
URL
VPN – Virtual Private Network
Website
YARA
Categories
Python

Azure DevOps: how to run python in a virtual environment

Intro

My pipeline job has been running without issue for the last year+. All of a sudden I started to receive this error right after my pip3 install -r requirements.txt:

× This environment is externally managed

Well I never had that problem before. I assumed that ADO sprinkles some magic dust onto the agents in the pool and creates an already virtual environment so why bother requiring further vitrualization?? I don’t know. And that’s how I rationalized how it might have ever worked in the first place if I ever bothered to think about it at all.

But I guess I was living on borrowed time and that house of cards came down hard, probably after the agents were upgraded.

The details

With some small effort I have managed to have the pipeline build up a virtual python environment and install my needed packages into it. Here is the relevant code section in the yaml file.

This by the way is my job to check that all pipelines have run correctly in the last hour:

# As of 2/2024 we need to run pytho in a virtual environment
- script: |
    python3 -m venv venv
    source ./venv/bin/activate
    pip3 install -vvv --timeout 60 -r requirements.txt
    python3 check_all_pipelines.py conf_check_all.ini
  displayName: 'Run script'
  workingDirectory: $(System.DefaultWorkingDirectory)/Pipeline_check
  env:
    ADO_AUTH: $(ado_auth)
    PYTHONPATH: $(System.DefaultWorkingDirectory)/Pipeline_check:$(System.DefaultWorkingDirectory)

The requirements.txt simply contains the line

pymsteams==0.2.2

Conclusion

We have shown how to set up a python virtual environment within your yaml file for an Azure DevOps pipeline. You might need this if you rely on any external packages which are not present in the OS version of python on the agents.

References and related

Wrtie-up of my ADO pipeline checker pipeline

How an ADO piepline can modify its own git repository

My favorite python tips

Categories
CentOS Debian Linux

What happened to insert mode on the latest version of vi?

Intro

As a creature of habit, I fall for an editor and can never imagine using something else. In the VAX days there was EDT, which if memory serves was replaced by the even better VPU. On Ultrix we ran a pretty nice editor simply called e from Rand Corporation. Then there was the love affair with emacs, and finally for the last 30 years vi.

Well with my latest server, a Debian 12 machine, I was having trouble with insert mode, specifically, inserting text from my Windows clipboard. Never had problems before….

Well for some reason, if you want to insert text from the clipboard, you now use <CTRL.>-<SHIFT>V in command mode. Well, at least on Windows 11 running WSL 2 that seems to work. I now realize that doesn’t work from Windows 10 with WSL. I had better figure this out soon…

Windows 10 running WSL

The terminal type (check the TERM environment variable) is set toxterm-color256. I tried pasting any and all registers which is the standard thing you would do if you use the standard Internet advice. None of it worked for me. I finally realized on my own that – and this harkens back to my old days with the beloved VAX 780 – that if I set the terminal type to vt100 all was good! Seriously. Back in the day we had physical VT100 terminals. Well, before that I think there was a VT52? Then maybe a VT102. VT202 was a big upgrade. Anyway, initially I added the following line to my .bashrc file:

export TERM=vt100

and now I can insert clipboard text the way I always have (mouse right-click) in vi insert mode. This kludge was how we fixed a lot of terminal display issues in the old days. But now I see display from top is messed up! Probably other curses-based apps as well. So two steps forward, one step back. So now what I’ve done is removed that line from .bashrc and put the following lines in my .bash_aliases file:

# DrJ kludge to get vi to work and keep top working
alias top='export TERM=xterm-256color;\top'
alias vi='export TERM=vt100;\vi'

That \top harkens back to an old linux convention where a command preceded by \ invokes a program but ignores defined aliases for that program.

Conclusion

I have offered one possible solution to the can’t insert text from the clipboard problem into my vi: set the TERM environment variable to the old-fashioned vt100. Now I can once again right-click while in insert mode to paste in clipboard text.

This was a very vexing issue for a creature of habit such as me!

References and related

This whole issue came up only when I switched from CentOS 8 to Debian 12 as my back-end server. Believe me, Debian 12 is so superior in so many ways this little setback would never make a material impact in that decision. Here’s the write-up of my upgrade.

Categories
CentOS Debian Linux Raspberry Pi

drjohnstechtalk now runs on a modern OS

Intro

I’m thrilled to announce that the long-running blog drjohnstechtalk.com has now been migrated to a modern back-end operating system. drjohnetchtalk.com is, a far as I know, the only quality-written technical resource on the Internet which is not supported by ads. Instead it runs on a pay-it-forward approach, embracing the spirit of the old Internet before it was ruined by big money.

drjohnstechtalk.com has been providing solutions to obscure tech questions since 2011.

The details

I like to run my own server which I can use for other purposes as well. I think that approach used to be more common. Now it’s harder to find others using it. Anyway, my old hosting environment is a CentOS server. I had hoped it would last me up to 10 years! 10 years is about the duration of long-term support for Redhat linux. It’s a real pain to migrate a WordPress blog with lots of history where it is important to preserve the articles and the permalinks. This article documents the nightmare I put myself through to get that up and running. Before that there was a CentOS 6 server. Then in 2022 – only about two years later – I learned that CentOS was dead! IBM had killed it. I’m over-simplifying here somewhat, but not by much.

So my blog sort of limped on on this unsupported system, getting riskier by the day to run as I was missing out on security patches. Then my companyt accidentally included one of my blogs in a security scan and I saw I had some vulnerabilities. So I upgraded WordPress versions and plugin versions. So with up to date software, the stage was set to migrate to a newer OS. Further motivation was provided by the fact that after the WP upgrade, the pages loaded more slowly. And sometimes the site just collapsed and crashed.

I have come to love Debian linux due to my positive experience with running it on Raspberry Pis and a few other places. It tends to run more recent versions of open source software, for instance. So I chose a Debian linux server. Then I forget where I learned this. Perhaps I asked someone at work which web server to use, but the advice was to use nginx, not apache! This was very new to me as I had never run nginx, not that I was in love with apache.

So, anyway, here I am writing this on my shiny new Debian 12 bookworm server which is running an nginx web server! And wow my site loads so much faster now. It’s really striking…

Running WordPress in a subdirectory with nginx

There always has to be a hard part, right? This was really, really hard. I run WP in the subdirectory blog as you can see from any of my URLs. I must have scoured a dozen sites on how to do it, none of which completely worked for me. So I had to do at least some of the heavy lifting and work out a working config on my own.

Here it is:

# mostly taken from https://www.nginx.com/resources/wiki/start/topics/recipes/wordpress/
# but with some important mods
upstream php {
    server unix:/var/run/php/php8.2-fpm.sock;
}

server {
  listen 443 ssl;

    include snippets/self-signed.conf;


    server_name drjohnstechtalk.com www.drjohnstechtalk.com;

    root /web/drjohns;
    index index.php index.html;


    access_log /var/log/nginx/drjohns.access.log;
    error_log /var/log/nginx/drjohns.error.log;

    client_max_body_size 100M;

# the following section prevents wp-admin from infintely redirecting to itself!
    location /blog/wp-admin {
            root /web/drjohns;
            try_files $uri $uri/ /blog/wp-admin/index.php?$args;
    }

    location /blog {
            root /web/drjohns/blog;
            try_files $uri $uri/ /blog/index.php?$args;
    }
    location ~ \.php$ {
#NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
         include fastcgi_params;
         fastcgi_intercept_errors on;
         fastcgi_pass php;
         fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    }
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg) {
            expires max;
            log_not_found off;
    }
}

I had to add ths svg file type to ignore, the location directive that matches /blog/wp-admin/. I had to define the upstream label as php and refer to that label in fastcgi_pass. I had to figure out my correct version of fastcgi. I tossed out some location directives which weren’t too important to me.

I disabled the wp-hide-login plugin while I grappled with why I was getting first a 404 not found for /blog/wp-admin/, then later, the too many redirects error. But I still had the issue with it disabled. Once I resolved the problem by adding the /blog/wp-admin location directive – I seem to be the only one on the Internet offering this solution and no other solution worked for me! – then I re-enabled the hide login plugin. The other plugins are working I would say.

Firewall?

I gather the current approach to host-based firewall on Debian 12 is to run ufw. A really good article on setting it up is here: https://www.cyberciti.biz/faq/set-up-a-firewall-with-ufw-on-debian-12-linux/

I’m on the fence about it, fearing it might slow my speedy server. But it looks pretty good. So for now I am relying on AWS Network Security Group rules. Did you know you can ask them to increase your max rule quota frmo 20 to 40? Yes, you can. I did and got approved overnight. I have added the Cloudflare ranges.

Cloudflare

I continue to use Cloudflare as reverse proxy, certificate issuer, DNS provider and light security screening. The change to the new server did not alter that. But I needed a new config file to properly report the origin IP address in my access files. The following file does the trick for me. It is up to date as of February 2024, can be placed in your /etc/nginx/conf.d directory and called, e.g., cloudlfare.conf.

# up to date as of 2/2024
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 104.16.0.0/13;
set_real_ip_from 104.24.0.0/14;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 131.0.72.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 2400:cb00::/32;
set_real_ip_from 2606:4700::/32;
set_real_ip_from 2803:f800::/32;
set_real_ip_from 2405:b500::/32;
set_real_ip_from 2405:8100::/32;
set_real_ip_from 2c0f:f248::/32;
set_real_ip_from 2a06:98c0::/29;

real_ip_header CF-Connecting-IP;

The idea is that if the source IP of the HTTP connection to nginx is from the Cloudflare range of IPs, then this must represent a request proxied through Cloudflare and the original IP of the client is in the HTTP header CF-Connecting-IP, which nginx can report on. If not, just use the normal IP from the TCP connection.

Swap space

On CentOS I had to provide some swap space because otherwise apache + mariaDB + WordPress would easily send its cpu soaring. So far I have not had to do that with my new Debian 12! That is great… So I have a t2.small instance with 25 GB of gp2 storage (100 iops). The server is basically running with a 0.00 load average now. I don’t get a lot of traffic so I hope that infrastructure will suffice.

Set the timezone

My Debian system started out in the UTC timezone. This command confirms that:

sudo timedatectl

This command brings up a menu and i can change the timezone to US Eastern:

sudo dpkg-reconfigure tzdata

Automate patching

It hasn’t run yet, but I’m hoping this root crontab entry will automate the system updates:

59 2 * * 0 (date && apt-get update && apt-get upgrade -y) >> /home/admin/hosting/update.log 2>&1

Debian 12 lifecycle

There should be three years of full support plus two more years of long term support for a stable Debian release, if I’ve undrstood it correctly. So I believe I may hope to get five years out of my Bookworm version, give or take. Debian — Debian Releases

Fixing the vi editor

I’ve never really had a problem with vi until this server. I show how I fixed it in this blog post.

Status after a few days – not all positive news

Well after a few days I feel the server response has noticeably slowed. I could not run top because I messed up the terminal with my fix to vi! So in a panic I restarted mariadb which seemed to help performance a lot. I will have to figure out how to monitor for this problem and how best to address it. I’m sure it will return. Here is my monitor.sh script:

#!/bin/bash
# restart mariaDB if home page response becomes greater than one second
curl -m1 -o /dev/null -ksH 'Host:drjohnstechtalk.com' https://localhost/blog/
# if curl didn't have enough time (one sec), its exit status is 28
[ $? -eq 28 ] && (systemctl stop mariadb; sleep 3; systemctl start mariadb; echo mariadb restart at $(date))

I invoke it from root’s crontab every three minutes:

# check that our load time is within reason or else restart mariadb -DrJ 2/24
*/3 * * * * sleep 25;cd /home/admin/hosting; ./monitor.sh >> monitor.log 2>&1

I do love my kludges. I will be on the lookout for a better long-term solution.

Conclusion

The technical blogging web site drjohnstechtalk.com now runs on new infrastructure: Debian 12 running nginx. It is muich faster than before. The migration was moderately painful! I have shared the technical details on how I managed to do it. I hope that, unlike my previous platform of CentOS 8, this platform lasts me for the next 10 years!

References and related

My second article!

nginx’s own advice about how to configure it to run WordPress

Trying to upgrade WordPress brings a thicket of problems

One of many RPi projects of mine: Raspberry Pi light sensor project

ufw firewall for Debian 12

Debian — Debian Releases

Cloudflare, an added layer of security for your web site

IP Ranges | Cloudflare

What happened to insert mode on the latest version of vi?

Categories
Linux

How an ADO pipeline can modify its own repo

Intro

I wanted to run a job on an Azure DevOps pipeline which did a backup of DNS zones on Cloudflare and write the results, in the form of a compressed tar file, into the ADO repository since everyone on the team has access to it and knows how to make a clone of the repo.

My first attempts produced some stunningly bad results. I was wiping out recently created files in the repo and such. That is very undesirable.

The solution

By “stealing with pride” from colleagues and such, I arrived at this AFAIK working solution. Here is the yaml file.

trigger: none

pool:
  name: backup_agents

steps:
# next two lines needed so we can modify the git repo and add our backups
- checkout: self
  clean: true
  persistCredentials: true
  fetchDepth: 1

- script: pip3 install -vvv --timeout 60 -r Cloudflare-backup/requirements.txt
  displayName: 'Install requirements'

- script: python3 backup-all-zones.py
  displayName: 'Run script'
  workingDirectory: $(System.DefaultWorkingDirectory)/Cloudflare-backup
  env:
    CLOUDFLARE_API_TOKEN: $(cloudflare_api_token)
    PYTHONPATH: $(System.DefaultWorkingDirectory)/Cloudflare-backup:$(System.DefaultWorkingDirectory)
- script: |
    git config --global http.sslVerify false
    git config --global user.email "[email protected]"
    git config --global user.name "pipeline"
    cd Cloudflare-backup
    pwd
    ls
    git add backups/zones-*
    git commit -m "adding todays backup files"
    git push origin HEAD:refs/heads/main

schedules:
- cron: "47 23 * * *"
  displayName: Run the script at 23:47 UTC
  branches:
    include:
    - main

I’m not exactly where all the magic happens. I think the section at the top that does the self checkout must be important. Then, obviously, there are the git add/git commit -m/git push commands. I do not claim to understand the origin HEAD:refs/heads/main argument to git push. I just copied it from a working example.

And branches: include -main. I’m not sure what this does either.

I need a few more days of testing, to be really certain, but I no longer am reverting my repo to an old state as I was with my initial attempts which involved doing a git fetch and probably missed the self checkout step as well.

Conclusion

One day I hope to understand git. But that today is not today! nevertheless I got my ADO pipeline to add backup files to its own ADO repository! So that’s cool.

References and related

My own git cheatsheet

Cloudflare python api examples

Categories
Linux Raspberry Pi

Raspberry Pi light sensor project

Intro

I ultimately want to turn off the connected display when it is nighttime and the lights have been turned off. And I want it to turn itself back on during daylight. The reason is because my RPi-driven slideshow is running and somoeone may be sleeping in that room.

This is till a work in progress. Pardon our dust and gibberish. What I’ve already found out is too important to delay publication.

Equipment and skills
How to turn off and on an HDMI port with a Raspberry Pi 4

Honestly this is the most significant thing I have found in this investigation. The methods used for older RPis do not work! In other words you can run vcgencmd display_power 0 and clever variations of that command until you’re blue in the face and the thnig stubbornly won’t turn off. tvservice -o ? Nope. That’ll suggest that command was deprecated and use kmsprint instead.

But I can say as of this writing (Jan ’24) the kms* commands are not mature and do not permit you to turn off the display. kmsprint tells you some stuff, but it does not allow you to set things. Remember RPi is for the education and hobbyist crowd so we have to give them the slack to experiment and try new things, even when they aren’t fully formed.

Instead they give you a way to restore the old commands. Edit the file /boot/config.txt.

Make it look like this:

#Enable DRM VC4 V3D driver
#use the fake kms driver in place of the native kms driver so we can  control hdmi power -DrJ 1/24
##dtoverlay=vc4-kms-v3d
dtoverlay=vc4-fkms-v3d

Reboot. Congrats you are now using the fake kms driver (fkms) and now have compatibility with the old commands. But instead of using tvservice, for my purposes, I think vcgencmd is better because the frame buffer state is not lost.

So now this command will indeed turn off the display:

vcgencmd display_power 0 # turn hdmi display off

vegcmd display_power 1 # turns the HDMI display back on

vcgencmd display_power 0 # turn hdmi display off

Prepare for our light sensor

I really don’t know if I’ll ever get the light sensor to work or not. Anticipating that it will, I have created this GPIO callback routine in python which in my dreams will turn off the HDMI display when the room is dark and turn it back on when the ambient light crosses a threshold. Who knows… But the code is pretty cool because it permits you to play with it as well using software command you send to a GPIO pin.

I call it gpio_basic.py:

import RPi.GPIO as GPIO
import time
import os
import datetime

GPIO.setmode(GPIO.BCM)
channel=4
GPIO.setup(channel, GPIO.IN)
reading = GPIO.input(channel)
print('Initial Reading',reading,flush=True)
old_reading = reading
while True: # infinite loop
    time.sleep(4)
# rising means ambient light went from light to dark
    reading = GPIO.input(channel) # 1 => dark, D0 LED turns off; 0 if light
    if reading == old_reading: continue

# else section where the state has changed
    print('This is a change in state on channel',channel,'at ',datetime.datetime.now(),flush=True)
    print('Reading',reading)
    if reading == 1:
        print('Turning off the HDMI display...',flush=True)
        os.system("vcgencmd display_power 0")
    else:
        time.sleep(6)
        print('Turn on HDMI display...',flush=True)
        os.system("vcgencmd display_power 1")
    old_reading = reading

Now to play with it, in another window get into python and run these commands:

import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)
channel=4
GPIO.setup(channel, GPIO.IN)
GPIO.input(channel)
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_UP)
#or
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

Those commented out lines at the end are the key ones. By executing them you should see the display turn off or on, and your other program should output some verbiage. Now hopefully the sensor will work something like that!

Wiring

Don’t know if this will pan out. I envision wiring (RPi pin #’s on the left):

  • 1 – Vcc (+ 3.3 V voltage)
  • 6 – Gnd (ground)
  • 7 – D0 (digital out)

At this point I don’t see the point of wiring the fourth lead on the diode sensor (which is A0, analog out – the RPi cannot read analog out), but having only three of them wired doesn’t quite seem right either. My electronics knowledge is weak!

So maybe I’ve wasted my money on these sensors, or I need to read them with a microcontroller and use i2c to talk to the RPi (too much effort and too much expense for my taste), but I hope not. Will know soon…

My first sensor, the photodiode, works! But its threshold is near one end of the control, which isn’t so great. But it’s fun to play with. In other words, its low light sensitivity may not be adequate for my purposes where I need to distinguish low ambient light (in a room with only a glowing TV screen, for instance) from even lower ambient light (lights off). Turns out our human eyes are the best measuring devices! Actually this photo diode, properly tuned, is quite good! I believe there is some jitter in the measurements however. So it can jump around during low light a bit. I have to consider how much of a problem that is.

The photodiode has a power led and a light sensor led. They are both way too bright. I suspect they could even create a feedback loop. I covered both with masking tape, leaving room for the adjustment screw.

But much, much worse than the the photodiode is the photoresistor. That at best distinguishes between a decent amount of light, and quite dark. But the transition between the two is sticky (I believe that would be called hysteresis). It will not work at all for my purposes based on my initial testing. It cannot distinguish between low light and very low light no matter where the dial is set.

Since this is all working, we just need to make it permanent by starting at boot time. So in my crontab file I added this line:

# Turn monitor off and on depending on ambien light! - DrJ 1/9/24
@reboot sleep 42; python3 gpio_basic.py > gpio_basic.log 2>&1
RPi4 shown with the slick metal case and the photodiode during daytime
Measure temperature of the CPU and why it matters

vcgencmd measure_temp

With my new fancy aluminum, heat-dissipating case I get around 40.4° C. On my RPi 3, air-cooled it is around 49° C. Why the sudden concern around heat? I’m just beginning to suspect that you know those times when you use the command line and things just seem to freeze? I always just assumed it was a glitch in the WiFi. But maybe it was actually the cpu getting too hot and having to pause itself to avoid burning up. I would see this when transferring files on my RPi 4. so the RPi 4 probably really does run hot unless you take steps to cool it, and the aluminum case in the equipment list above is really cool, ha , ha.

Tips

The command pinout is very useful. Pinout docs here: https://www.raspberrypi.com/documentation/computers/raspberry-pi.html

References and related

This guide talks about explicit support for generic digital input devices, not just a button: https://randomnerdtutorials.com/raspberry-pi-digital-inputs-python/

This talks about built-in pull-up pull-down resistors configurable via software! Who knew?

This might be useful for simple GPIO stuff: https://tutorials-raspberrypi.com/raspberry-pi-gpio-explanation-for-beginners-programming-part-2/

Gory details with gory circuit diagram: https://www.circuits.dk/everything-about-raspberry-gpio/

Running multiple RPi slideshows

How to deal with th GPIO pins using python: https://sourceforge.net/p/raspberry-gpio-python/wiki/Inputs

This project used the RPi GPIO OUTPUT pins to control a power relay device

Appendix A

I started with this code, gpio_callback.py, but the one condition kept getting called when the GPIO pin was reading 1. So I wrtoe gpio_basic.py and use that instead.

import RPi.GPIO as GPIO
import time
import os

GPIO.setmode(GPIO.BCM)
channel=4
GPIO.setup(channel, GPIO.IN)
reading = GPIO.input(channel)
print('Initial Reading',reading)
def my_callback(channel):
# rising means ambient light went from light to dark
    print('This is a edge event callback function!')
    print('Edge detected on channel %s'%channel)
    print('This is run in a different thread to your main program')
    print('Gonna stop that slideshow now...')
    reading = GPIO.input(channel) # 1 => dark, D0 LED turns off; 0 if light
    print('Reading',reading)
    if reading == 1:
        time.sleep(1)
        print('Turning off the HDMI display...')
        os.system("vcgencmd display_power 0")
    else:
        time.sleep(1)
        print('Turn on HDMI display...')
        os.system("vcgencmd display_power 1")

# RISING,FALLING or BOTH. https://sourceforge.net/p/raspberry-gpio-python/wiki/Inputs/
GPIO.add_event_detect(channel, GPIO.BOTH, callback=my_callback)  # add edge detection on a channel
# for testing, use:
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_UP)
# or
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)
time.sleep(31415927) # one year

Categories
Debian Linux Raspberry Pi

My favorite bash scripting tips

Intro

The linux bash shell is great and very flexible. I love to use it and have even installed WSL 2 on my PCs so I can use it as much as possible. When it comes to scripting it’s not exactly my favorite. there is so much history it has absorbed that there are multiple ways to do everything: the really old way, the new way, the alternate way, etc. And your version of bash can also determine what features you can use. nevertheless, I guess if you stick to the basics it makes sense to use bash for simple scripting tasks.

So just like I’ve compiled all the python tips I need for writing my simple python scripts in one convenient, searchable page, I will now do the same for bash. No one but me uses it, but that’s fine.

Iterate (loop) over a range of numbers

END=255 # for instance to loop over an ocetet of an IP address
for i in $(seq 1 $END); do
  echo $i
done
# But if it's OK to just hard-wire start and end, then it's simpler to use:
for i in {1..255}; do echo $i; done

Infinite loop
while /bin/true; do...done

You can always exit to stop it.

Sort IPs in a sensible order

$ sort -n -t . -k1,1 -k2,2 -k 3,3 -k4,4 tmp

What directory is this script in?

DIR=$(cd $(dirname $0);pwd);echo$DIR

Guarantee this script is interpreted (run) by bash and not good ‘ole shell (sh)!
if [ ! "$BASH_VERSION" ] ; then
  exec /bin/bash "$0" "$@"
  exit
fi
Count total occurrences of the word print in a bunch of files which may or may not be compressed, storing the output in a file

print=0
zgrep -c print tst*|cut -d: -f2|while read pline; do prints=$((prints + pline));echo $prints>prints; done

Note that much of the awkwardness of the above line is to get around issues I had with variable scope.

Legal characters in variable names

Don’t use _ as you might in python! Stick to alphanumeric, but also do not begin with a number!

Execute a command

I used to use back ticks ` in the old days. parentheses is more visually appealing:

print1=$(cat prints)

Variable type

No, variables are not typed. Everything is treated as a string.

Function definition

Put function definitions before they are invoked in the script. Invocation is by plain name. function syntax is as in the example.

sendsummary() {
# function execution statements go here, then close it out
} # optionally with a comment like end function sendsummary
sendsummary # invoke our sendsummary function
Indentation

Unlike python, line indentation does not matter. I recommend to indent blocks of code two spaces, for example, for readability.

Booleans and order of execution
[[ "$DEBUG" -eq "1" ]] && echo subject, $subject, intro, "$intro"

The second statement only gets executed if the first one evaluated as true. Now a more complex example.

[[ $day -eq $DAY ]] || [[ -n “$anomalies” ]] && { statements…}

The second expressions get evaluated if the first one is false. If either the first or second expressions are true, then the last expression — a series of statements in what is essentially an unnamed function, hence the enclosing braces — gets executed. The -n is a test to see of length of a string is non-zero. See man test.

Conditionals

Note that clever use of && and || can in many cases obviate the need for a class if…then structure. But you can use if thens. An if block is terminated by a fi. There is an else statement as well as an elif (else if) statement.

grep conditionals
ping -c1 8.8.8.8|grep -iq '1 received'
[ $? -eq 0 ] && echo this host is alive

So the $? variable after grep is run contains 0 if there was a match and 1 if there was no match. -q argument puts grep in “quiet” mode (no output).

More sophisticated example testing exit status and executing multiple commands

#!/bin/bash
# restart mariaDB if home page response becomes greater than one second
curl -m1 -ksH 'Host:drjohnstechtalk.com' https://localhost/blog/ > /dev/null
# if curl didn't have enough time (one sec), its exit status is 28
[ $? -eq 28 ] && (systemctl stop mariadb; sleep 3; systemctl start mariadb; echo mariadb restart at $(date))

Note that I had to group the commands after the conditional test with surrounding parentheses (). That creates a code block. Without those the semicolon ; would have indicated the end of the block! A semicolon ; separates commands. Further note that I nested parentheses and that seems to work as you would hope. also note that STDOUT has been redirected by the greater than sign > to /dev/null in order to silently discard all STDOUT output. /dev/null is linux-specific. The windows equivalent, apparently, is nul. Use curl -so nul suppress output on a Windows system.

One square bracket or two?

I have no idea and I use whatever I get to work. All my samples work and I don’t have time to test all variations.

Variable scope

I really struggled with this so I may come back to this topic!

Variable interpolation

$variable will suffice for simple, i.e., one-word content. But if the variable contains anything a bit complex such as words separated by spaces, or containing unusual characters, better go with double quotes around it, “$variable”. And sometimes syntactically throw in curly braces to separate it from other elements, “${variable}”

Eval
eval="ls -l"
$eval # executes ls -l
Shell expansion
mv Pictures{,.old} # renames directory Pictures to Pictures.old
Poor man’s launch at boot time

Use crontab’s @reboot feature!

@reboot sleep 25; ./recordswitch.sh > recordswitch.log 2>&1

The above expression also shows how to redirect standard error to standard out and have both go into a file.

Use extended regular expressions, retrieving a positional field using awk, and how to subtract (or add) two numbers
t1=`echo -n $line|awk '{print $1}'` 
t2=`echo -n $line|awk '{print $4}'` 
# test for integer inputs 
[[ "$t1" =~ ^[0-9]+$ ]] && [[ "$t2" =~ ^[0-9]+$ ]] && downtime=$(($t1-$t2))

Oops, I used the backticks there! I never claim that my way is the best way, just the way that I know to work! I know of a zillion options to add or subtract numbers…

Get last field using awk
echo hi.there.111|awk -F\. '{print $NF}' # returns 111
Why do assignments have no extra spaces?

It simply doesn’t work if you try to put in spacing around the assignment operator =.

Divert stdout and stderr to a file from within the script
log=/tmp/my-log.log
exec 1>$log 
exec 2>&1
Lists, arrays amd dictionary variables

I don’t think bash is for you if you need these types of variables.

Formatted date

date +%F

produces yyyy-mm-dd, i.e., 2024-01-25

date +%Y%m%d -> 20240417

Poor man’s source code versioning

The old EDT/TPU editor on VAX used to do this automatically. Now I want to save a version of whatever little script I’m currently working on in the ~/tmpFRI (if it’s Friday) directory to sort of spread out my work by day of the week. I call this script cpj so it’s easy to type:

#!/bin/bash
# save file using sequential versioning to tmp area named after this day - DrJ
DIR='~'/tmp$(date +%a|tr '[a-z]' '[A-Z]') # ~/tmp + day of the week, e.g., FRI
DIRREAL=$(eval "echo $DIR") # the real diretory we need
mkdir -p $DIRREAL
for file in $*; do
  res=$(ls $DIRREAL|egrep "$file"'\.[0-9]{1,}$') # look for saved version numbers of this filename
  if test -n "$res"; then # we have seen this file...
    suffix=$(echo $res|awk -F\. '{print $NF}')  # pull out just the number at the end
    nxt=$(($suffix+1)) # add one to the version number
    saveFile="${file}"."${nxt}"
  else # new file to archive or no versioned number exists yet
    [[ -f $DIRREAL/$file ]] && saveFile="$file".1
    [[ -f $DIRREAL/$file ]] || saveFile=""
  fi
  cp "$file" $DIRREAL/"$saveFile"
  [[ -n $saveFile ]] && target=$DIR/"$saveFile"
  [[ -n $saveFile ]] || target="$DIR"
  echo copying "$file" to "$target"
done

It is a true mis-mash of programming styles, but it gets the job done. Note the use of eval. I’m still wrapping my head around that. Also note the technique used to upper case a string using tr. Note the use of extended regular expressions and egrep. Note the use of tilde ~ expansion. I insist on showing the target directory as ~/tmpSAT or whatever because that is what my brain is looking for. Note the use of nested $‘s.

Now that cpj is in place I occasionally know I want to make that versioned copy before I launch the vi editor, so I created a vij in my bash alias file thusly:

vij () { cpj "$@";sleep 1;vi "$@"; }

Another example

I wrote this to retain one backup per month plus the last 28 days.

#!/bin/bash
# do some date arithmetic to preserve backup from first Monday in the month
#[[ $(date +%a) == "Wed" ]] && { echo hi; }
DEBUG=0
DRYRUN=''
[[ $DEBUG -eq 1 ]] && DRYRUN='--dry-run'
if [[ $(date +%a) == "Mon" ]] && [[ $(date +%-d) -lt 8 ]]; then
# preserve one month ago's backup!
  echo "On this first Monday of the month we are keeping the Monday backup from four weeks ago"
else
  d4wksAgo=$(date +%Y%m%d -d'-4 weeks') # four weeks ago
  oldBackup=zones-${d4wksAgo}.tar.gz
  git rm $DRYRUN backups/$oldBackup
fi
today=$(date +%Y%m%d)
todaysBackup=zones-${today}.tar.gz
git add $DRYRUN backups/$todaysBackup

It incorpoates a lot of the tricks I’ve accumulated over the years, too numerous to recount. But it’s a good example to study.

Output the tab character in an echo statement

Just use the -e switch as in this example:

echo -e “$subnet\t$SSID”

Get top output in a non-interactive (batch) shell

top -b -n 1

Prompting for user input

echo -n “Give your input: “

read userInput

Conclusion

I have documented here most of the tecniques I use from bash to achieve simple yet powerful scripts. My style is not always top form, but as I learn better ways I will adopt and improve.

Categories
Linux Raspberry Pi

Multiple Raspberry Pi photo frames

Intro

I have previously shared my work on displaying pictures in a nice slideshow from a Google Drive to an HDMI monitor. All these years later, it is still working and every day we see a new slideshow or randomly yet thoughtfully chosen pictures.

Building on this, today I extended this solution to display these pictures on a second monitor in a different room.

Equipment
  • RPi 4 (what I happened to have around. RPi 3 would also be fine)
  • HDMI monitor
  • Raspberry Pi Lite OS
The details

I needed to install fbi (sudo apt-get install fbi).

I needed to copy over m4.pl from the primary display, and for good measure black.jpg.

The general idea is to copy the pics over to the second display once per day.

I call this program copyslideshow.sh.

#!/bin/bash
# copy pictures from primary source
rm -rf Pictures.old
mv Pictures{,.old}
sshpass -p raspberry scp -r [email protected]:mediashow .
sshpass -p raspberry scp -r [email protected]:Pictures .
./m4.pl  $HOME/mediashow >> m4.log 2>&1

Yeah so I hard-coded the RPi password which is still set to the default. I’m willing to take the risk

Then in crontab I added this line:

# get yesterday's pictures!
1 5 * * * killall m4.pl; ./copyslideshow.sh > copyslideshow.log 2>&1

Note that these displays are not synced. That would be a whole ordeal. In fact we thought it would be cool to display different pictures. So the second monitor will be showing yesterday’s slideshow from the main monitor.

Automating turn-on, turn-off of the HDMI display based on the ambient room light

Since this second slideshow is in a bedroom, I wanted to have it turn off when the lights were out, and turn back on again during daylight. This was a really interesting challenge for me as I got to use an inexpensive external sensor with my RPi. And I got it to work, and it works quite well if I say so myself. That’s all written up in this post.

Reference and related

I have written many variations on this same topic. I guess this is a good one.

How I automatically turn on the HDMI display in the morning and turn it off again at night.

Categories
Linux

Linux tip: How to sort a list of IP addresses

Intro

Sorting a list of IPs should be easy using some linux utility or another, right? It is, and here is how to do it.

The details

Say your list of IPs, one per line, is contained in the file tmp. Then the proper sort command is:

$ sort -n -t . -k1,1 -k2,2 -k 3,3 -k4,4 tmp

If you want things in descending order just do a -nr instead of -n at the beginning.

This sort solves the problem that for instance 9 is evaluated as being greater than 115, for instance!

References and related

More shell tricks can be gleaned from the way I solved this NPR puzzle

Categories
Consumer Interest

Consumer Tech: how to unfreeze a frozen iPhone screen

Intro

This is just to have this useful tip for my own use.

The details

On my iPhone 13 the screen freezes from time to time. It’s not totally, totally inoperable. I was able to enter my PIN but not to be able to take an incoming call. But mainly just stuck on the screen of one app.

So what you do is:

  • Press and release the up volume button
  • Press and release the down volume button
  • Hold the side button
  • The screen turns black. Keep holding that side button!
  • An Apple logo appears. Finally you can release the side button

You should now be good to go!

Why does this happen?

Wish I knew. If I ever find out I will document it here. At present I can just speculate that the phone runs out of memory.