Categories
Linux Python

How to prevent an image from being rotated

Intro

I still use my home-grown slideshow software based on Raspberry Pi, which is quite a testament to its robustness as it has been running with only minor modifications for many years now. one recent improvement has been my addition of being able to handle photos from recent iPhones which save photos in the new-to-me HEIC format. My original implementation only handles JPEGs and PNG file types, so it was skipping all our recent iPhone photos.

I figured there just had to be a converter our there which would even work on the RPi, which of course there was, heif-convert. But it has an oddity when it comes to rotation. It converts the HEIC to a jpeg, fine, but it rotates them, but it also leaves all the EXIF meta data, including the orientation meta data, as is. This in turn means display software such as fbi may try to rotate the picture a second time. Or at least that’s what happened to my software where one of my steps is an explicit rotate. That step was creating a double rotation.

So I needed a tiny program which left all the EXIF meta data alone except the rotation, which it sets to 0, i.e., do not rotate. Seeing nothing out there, I developed my own.

The details

Here is that script, which I call 0orientation.py:

#!/usr/bin/python3
# DrJ 10/24
# you need the exif package: pip install exif
# https://github.com/kennethleungty/Image-Metadata-Exif?tab=readme-ov-file
#https://exif.readthedocs.io/en/latest/
import sys
from exif import Image
file = sys.argv[1]
with open(file, 'rb') as image_file:
    my_image = Image(image_file)
exif_flag = my_image.has_exif
print(exif_flag)
exifs = my_image.list_all()
print(exifs)
my_image.orientation = 0
with open('modified_image.jpg', 'wb') as new_image_file:
    new_image_file.write(my_image.get_file())

Conclusion

I have shared how to overwrite just the orientation tag frmo a JPEG photo.

Reference and related

My photo frame software based on photos stored in a Google Drive

Categories
Admin Linux Network Technologies

The IT detective agency: The mystery of the non-validating DKIM record

Intro

A colleague of mine in another timezone created the necessary DKIM records in Cloudflare for a new mail domain. There was panic as the mail team realized too late these records were not validating. I was called in to help. Unfortunately at the beginning I only my smartphone to work with. Did you ever try to do this kind of detail work with a smartphone? Don’t.

The details

The smartphone thing is worthy of a separate post. I was getting somewhere, but it is like working with both hands tied behind yuor back.

So the mail team is telling me the dkim record doesn’t validate and showing me a screenshot of something from mxtoolbox to prove it.

I of course want to know the details so I can verify my mistakes before anyone else gets to – that’s how I roll!

Well, mxtoolbox, has a free validator for these dkim records which is pretty useful. Go to Supertool, then click the dropdown and select DKIM. A DKIM record involves a domain and a selector. Here’s a real live example for Hurricane Electric which uses he.net as their sending mail domain. So in their DNS the DKIM txt record for them looks like this when viewed from dig:

"v=DKIM1; k=rsa; p=MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAonNI5HmoWfZntOsU5G3t eKi70HHBhDMe7himvGBNfq119soydCj7KoR9DsFYAMqCcPghLY29ishIbzMKsCFy 68XN4MWOSrFr+ERDHIuLXcFvaYYQ0oI5HVcViKSX85/YLXe+5JUcf5VsKoBLifNy U1NFA3UPa6MHBIOcD+JVF6F67G9m7t+COhsrhcvl9x" "kNq2NAY0OxbBM+CM+V4p0J 6pgt0PqYGnwd9s3/P7TUD2jY9elJLB5CfIec4DDCROj3MgUyTl2JfBcNy0WGzkEl OpFipd5MMesZvgyIVBsgLY58hTPldYhekkKWlOhpMpYbAi8gnvk+aJv2jZcaYHpJ kLNrri+q2gMeEX30JSoXfYNKx+B6m1Udn7Ig2ngHNVTXgNZlCw6SvbfmwXBE97q5 iG1SOnrgLKQvtgZv08Y7k5sp9+2SfoOS5MSYt" "OTfCbtknUi/VbaU4kVE76jFB0xx 6CAoR1SC9lDJBGvyFMuGvyhOXTiYV44tk1fyrV9Ba4yaKi8dhgHwe9vVbCSK8Ebt CeMXrkS/I3Dc33B6+tM1poC06GVhxElpd8rHiWvNImBuqCWwtGDsXm4ulubTcjvS gglJrB7kl4l3+AcTZn15zCrePl6xHWtL29b9vEy1w7whgExoDHaXZl+Svne9pfZ7 esXNu+mfERmGb56OreCEQQMCAwEAAQ=="

This is the value for this record: henet-20240223-153551._domainkey.he.net

To validate this DKIM record in mxtoolbox we pull out the token in front of _domainkey and refer to it as the selector, and drop the _domainkey and enter it like this:

The problem with the DKIM entry I was assigned to rescue was that the DIM syntax check was not passing. Yet it looked just like the way the mail team requested. What is going on? How can this problem be broken down into smaller steps???

To be continued…

Appendix A
How did I know the exact selector for Hurricane Electric?

I looked at the SMTP headers of an email I received from them. I found this section:

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=he.net;
	s=henet-20240223-153551

d must stand for domain and s for selector. This is all considered public information, albeit somewhat obscure. So the domain is he.net and the selector is henet-20240223-153551.

Categories
DNS Linux TCP/IP

The IT Detective Agency: the case of the slow dns server responses to tcp

Intro

This case was solved today. Now I just need to find the time to write it up!

I belong to a team which runs many dozens of dns servers. We have basic but thorough monitoring of these servers using both Zabbix and Thousandeyes. One day I noticed a lot of timeout alerts so I began to look into it. One mystery just led to another without coming any closer to a true root cause. There were many dead ends in the hunt. Finally our vendor came through and discovered something…

The details

The upshot are these settings we arrived at for an ISC BIND server:

   tcp-listen-queue 200;
   tcp-clients 600;
   tcp-idle-timeout 10;

This is in the options section of the named.conf file. That’s it! This is on a four-core server with 16 GB RAM. The default values are:

tcp-listen-queue: 10

tcp-clients: 10

tcp-idle-timeout: 60 seconds

Those defaults will kill you on any reasonably busy server, meaning, one which gets a couple thousand requests per second.

To be continued…

Conclusion

We encountered a tough situation on our ISC BIND DNS servers. TCP queries, and only TCP queries, were responded to slowsly at best or not at all. after many flase starts we found the solution was setting three tcp parameters in the options section of the configuration file, tcp-listen-queue, tcp-clients and tcp-idle-timeout. We’ve never had to mess with those parameters after literally decades of running ISC BIND. Yet we have incontrovertible proof that that is what was needed.

Case: closed!

References and related

A great and very detailed discussion of this type of TCP backlog issues on Redhat systems is found here: https://access.redhat.com/solutions/30453

Categories
Admin Consumer Interest Consumer Tech Firewall Home Computing Linux Scams Security Spam Web Site Technologies

Types of Cyberattacks and other terms from the world of cyber security

Intro

It’s convenient to name drop different types of cyber attacks at a party. I often struggle to name more than a few. I will try to maintain a running list of them.

But I find you cannot speak about cybersecurity unless you also have a basic understanding of information technology so I am including some of those terms as well.

As I write this I am painfully aware that you could simply ask ChatGPT to generate a list of all relevant terms in cybersecurity along with their definitions – at least I think you could – and come up with a much better and more complete list. But I refuse to go that route. These are terms I have personally come across so they have special significance for me personally. In other words, this list has been organically grown. For instance I plowed through a report by a major vendor specializing in reviewing other vendor’s offerings and it’s just incredible just how dense with jargon and acronyms each paragragh is: a motherlode of state-of-the-art tech jargon.

AiTM (Adversary in the Middle)
Baitortion

I guess an attack which has a bait such as a plum job offer combined with some kind of extortion? The usage was not 100% clear.

BYOVD (Bring Your Own Vulnerable Driver)
Clickfix infection chain

Upon visiting compromised websites, victims are redirected to domains hosting fake popup windows that instruct them to paste a script into a PowerShell terminal to fix an issue.

Collision attack

I.e., against the MD5 hash algorithm as done in the Blast RADIUS exploit.

Credential Stuffing Attack

I.e., password re-use. Takes advantage of users re-using passwords for different applications. Nearly three of four consumers re-use password this way. Source: F5. Date: 3/2024

Data Wiper
Evasion

Malicious software built to avoid detection by standard security tools.

Password spraying

A type of attack in which the threat actor tries the same password with multiple accounts, until one combination works. 

Port Scan
Host Sweep
Supply Chain attack
Social Engineering
Hacking
Hacktivist

I suppose that would be an activitst who uses hacking to further their agenda.

Living off the land
Data Breach
Keylogger
Darknet
Captcha
Click farms
Jackpotting

This is one of my favorite terms. Imagine crooks implanted malware into an ATM and were able to convince it to dispense all its available cash to them on the spot! something like this actually happened. Scary.

Overlay Attack

Example: When you open a banking app on your phone, malware loads an HTML phishing page that’s designed to look just like that particular app and the malware’s page is overlaid on top.

Payment fraud attack

In a recent example, the victim experienced “multiple fraudulently induced outbound wire transfers to accounts controlled by unknown third parties.”

Skimmer
bot
Anti-bot, bot defense
Mitigation
SOC
Selenium (Se) or headless browser
WAF
Obfuscation
PII, Personally Identifiable Information
api service
Reverse proxy
Inline
endpoint, e.g., login, checkout
scraping
Layer 7
DDOS
Carpet bombing DDOS attack

Many sources hitting many targets within the same subnet. See:

https://www.a10networks.com/blog/carpet-bombing-attacks-highlight-the-need-for-intelligent-and-automated-ddos-protection/#:~:text=Carpet-bombing%20attacks%20are%20not,entire%20CIDR%20or%20multiple%20ASNs.

SYN flood
DOS
Visibility
Automation
Token
Post
JavaScript
Replay
Browser Fingerprint
OS
Browser
GDPR
PCI DSS
AICPA Trust Services
Grandparent scam

A social engineering attack where scammers target grandparents by pretending to be a grandchild in a bind.

GUI
(JavaScript) Injection
Command Injection
Hotfix
SDK
URL
GET|POST Request
Method
RegEx
Virtual Server
TLS
Clear text
MTTR
RCA
SD-WAN
PoV
PoC
X-Forwarded-For
JSON
Client/server
Threat Intelligence
Use case
Carding attack
Source code
CEO Fraud
Phishing
Vishing

(Voice Phishing) A form of cyber-attack where scammers use phone calls to trick individuals into revealing sensitive information or performing certain actions.

Business email compromise (BEC)
Deepfake
Threat Intelligence
Social engineering
Cybercriminal
SIM box
Command and control (C2)
Typo squatting
Voice squatting

A technique similar to typo squatting, where Alexa and Google Home devices can be tricked into opening attacker-owned apps instead of legitimate ones.

North-South
East-West
Exfiltrate
Malware
Infostealer
Obfuscation
Antivirus
Payload
Sandbox
Control flow obfuscation
Buffer overflow
Use after free
Indicators of Compromise
AMSI (Windows Antimalware Scan Interface)
Polymorphic behavior
WebDAV
Protocol handler
Firewall
Security Service Edge (SSE)
Secure Access Service Edge (SASE)
Zero Trust

Zero Trust is a security model that assumes that all users, devices, and applications are inherently untrustworthy and must be verified before being granted access to any resources or data.

Zero Trust Network Access (ZTNA)
ZTA (Zero Trust Architecture)
Zero Trust Edge (ZTE)
Secure Web Gateway (SWG)
Cloud Access Security Broker (CASB)
Remote Browser Isolation (RBI)
Content Disarm and Reconstruction (CDR)
Firewall as a service
Egress address
Data residency
Data Loss Prevention (DLP)
Magic Quadrant
Managed Service Provider (MSP)
0-day or Zero day
User Experience (UX)
Watermark
DevOps
Multitenant
MSSP
Remote Access Trojan (RAT)
SOGU

2024. A remote access trojan.

IoC (Indicators of Compromise)
Object Linking and Embedding
(Powershell) dropper
Backdoor
Data Bouncing

A technique for data exfiltration that uses external, trusted web hosts to carry out DNS resolution for you

TTP (Tactics, Techniques and Procedures)
Infostealer
Shoulder surfing
Ransomware
Pig butchering

This is particularly disturbing to me because there is a human element, a foreign component, crypto currency, probably a type of slave trade, etc. See the Bloomberg Businessweek story about this.

Forensic analysis
Sitting Ducks

An entirely preventable DNS hijack exploit. See https://blogs.infoblox.com/threat-intelligence/who-knew-domain-hijacking-is-so-easy/

Attack vector
Attack surface
Economic espionage
Gap analysis
AAL (Authentication Assurance Level)
IAL (Identity Assurance Level)
CSPM (Cloud Security Posture Management)
Trust level
Network perimeter
DMZ (Demilitarized zone)
Defense in depth
Lateral movement
Access policy
Micro segmentation
Least privilege
Privilege Escalation (PE)
Breach
Intrusion
Insider threat
Cache poisoning

I know it as DNS cache poisoning. If an attacker manages to fill the DNS resolver’s cache with records that have been altered or “poisoned.”

Verify explicitly
Network-based attack
Adaptive response
Telemetry
Analytics
Consuming entity
Behavior analysis
Authentication
Authorization
Real-time
Lifecycle management
Flat network
Inherent trust
Cloud native
Integrity
Confidentiality
Data encryption
EDR (Endpoint Detection and Response)
BSOD (Blue Screen of Death)

Everyone’s favorite Windows error!

BSI (Bundesamt für Sicherheit in der Informationstechnik)

German Federal Office for Information Security (Bundesamt für Sicherheit in der Informationstechnik)

ICS (Industrial Control System)
Reverse shell

A text-based interfaces that allow for remote server control.

Crypto Miner
RCE (Remote Code Execution)
Threat Actor
APT (Advanced Persistent Threat)
Compromise
Vulnerability
Bug
Worm
Remote Access VPN (RAVPN)
XDR (Extended Detection and Response)
SIEM (Security Information and Event Management)
User Entity Behavior Analytics (UEBA)
Path traversal vulnerability

An attacker can leverage path traversal sequences like “../” within a request to a vulnerable endpoint which ultimately allows access to sensitive files like /etc/shadow.

Tombstoning
Post-exploit persistence technique
Volumetric DDoS
MFA bomb

Bombard a user with notifications until they finally accept one.

Use-after-free (UAF)

use-after-free vulnerability occurs when programmers do not manage dynamic memory allocation and deallocation properly in their programs.

Cold boot attack

A cold boot attack focuses on RAM and the fact that it is readable for a short while after a power cycle.

Random Prefix Attack

A type of DNS attack. https://developers.cloudflare.com/dns/dns-firewall/random-prefix-attacks/

Famous named attacks

Agent Tesla
Cloudbleed
Heartbleed
log4j
Morris Worm

Explanations of exploits

Famous attackers

APT29 (Cozy Bear)

A Russia-nexus threat actor often in the news

Volt Typhoon

2024. A China-nexus threat actor

IT terminology

2FA (2 Factor Authentication)
802.1x
ACL (Access Control List)
AD (Active Directory)
ADO (Azue DevOps)
AFK (Away From Keyboard)
AGI (Artificial General Intelligence)

AGI is the theory and development of computer systems that can act rationally.

ANN (Artificial Neural Network)
Ansible

I would call it an open source orchestrator.

anti-aliasing

When you smooth out color in neighboring pixels.

anycast
apache

A formerly popular open source web server which became bloated with features.

APM (Application Performance Management)
ARIN
ARP (Address Resolution Protocol)
ASN (Autonomous System Number)

Each AS is assigned an autonomous system number, for use in Border Gateway Protocol routing

ASN.1 (Abstract Syntax Notation One)

A standard interface description language (IDL) for defining data structures that can be serialized and deserialized in a cross-platform way.

ASPA (Autonomous System Provider Authorization)

An add-on to RPKI that allows an ASN to create a record that lists which ASNs can be providers for that ASN. The concepts are “customer” (an ASN) and “providers” (a list of ASNs). This is used to do hop by hop checking of AS paths.

ASR (Aggregation Services Router)

A high-end Interent router offered by Cisco for business customers.

AV (anti-virus)
AWS (Amazon Web Services)
Azure AD
BGP (Border Gateway Protocol)
Blast Radius

One of those dreadully overused terms borrowed from the military that mostly only marketing people like to throw around. It means what you think it might mean.

Blue Team – see Red Team
BOM (Bill of Material)
Boot start

A flag for a driver in Windows that tells it to always start on boot.

broadcast
Browser
BSI (The German Federal Office for Information Security)
BYOD (Bring Your Own Device)

I.e., when employees are permitted to use their personal smartphone to conduct company business.

BYOL (Bring Your Own License)

F5 permits this approach to licensing one of their cloud appliances.

CA (Certificate Authority)
CDN (Content Distribution Network)
CDP (Cisco Discovery Protocol)

This protocol allows devices connected to switch ports to learn what switch and which switch port they are connected to. It is a layer 2 protocol.

CGN (Carrier Grade NAT)

The address space 100.64.0.0/10 is handled specially by ISPs for CGN. RFC 6598

CHAP
Chatbot

A computer program that simulates human conversation with and end user.

CI (Configuration Item)

An ITIL term referring to the object upon which changes are made.

CISA (Cybersecurity and Infrastructure Security Agency)
CISO (Chief Information Security Officer)
CMO (Current Mode of Operations)
CNN (Congruential Neural Network)
Computer Vision

A field of AI that leverages machine learning and neutral networks to enable machines to identify and understand visual information such as images and videos.

CPE (Customer Premise Equipment)
CSR (Certificate Signing Request)
CUPS (Common Unix Printing Systems)
Customer Edge (CE)
CVE

CVEs, or Common Vulnerabilities and Exposures, are a maintained list of vulnerabilities and exploits in computer systems. These exploits can affect anything, from phones to PCs to servers or software.  Once a vulnerability is made public, it’s given a name in the format CVE–. There are also scoring systems for CVEs, like the CVSS (Common Vulnerability Scoring System), which assigns a score based on a series of categories, such as how easy the vulnerability is to exploit, whether any prior access or authentication is required, as well as the impact the exploit could have.

DAST (Dynamic Application Security Testing)
Data at rest
Data in motion
Data Remanence

The residual representation of data that remains even after attempting to erase or initialize RAM.

DDI (DNS, DHCP and IP address management)
Deep Learning

A subset of machine learningthat focus on using deep neural networks with multiple layers to model complex patterns in data.

Deepfake

A manipulated video or other digital representation produced by sophisticated machine-learning techniquies that yield seemingly realistic, but fabricated images and sounds.

DHCP (Dynamic Host Control Protocol)
DLL
DLP (Data Loss Prevention)
DNS (Domain Name System)
DNSSEC (Domain Name System Security Extensions)
Docker
DoH (DNS over HTTPS)
Domain
DRM (Digital Rights Management)
EAP
East-West

Data movement with a data center, I believe, as oppose to North-South.

EBITDA (Earnings Before Interest, Taxes, Depreciation and Amortization)

Hey, an IT person needs to know some business terminology!

Eduroam
Enhanced Factory Reset (EFR)
Entra

From Microsoft. The new name for Azure AD

EU AI Act
EULA (End User Licnese Agreement)
Exact Data Matching (EDM)
FAQ (Frequently Asked Questions)
FEX (Fabric Extender)
FIFO (First in, First Out)
FMO (Future Mode of Operation)

As opposed to CMO.

FN (False Negative)
Forensics
FOSS (Free and Open Source Software)
FP (False Positive)
Fritz!Box

A popular home router in Germany.

GA (General Availability)
GBIC

A type of fiber optic transceiver that converts electric signals to optical signals.

GCP (Google Cloud Provider)
GDPR (General Data Protection Regulation)

An EU directive to achieve data privacy.

Generative AI

AI which can create new human-quality content, including text, images, audio or video.

GMT – see UTC
GRE
GSLB (global Server Load Balancing)
GUI (Graphical User Interface)
HA (High Availability)
Hallucination

When an LLM perceives patterns that are non-existent creating nonsensical or inaccurate outputs.

Hands and Eyes

When you don’t have physical access to a server, you need someone who does to be this for you.

HIBP (Have I Been Pwned)

https://haveibeenpwned.com/

HSM (Hardware Security Module)
HTML (HyperText Markup Language)

I started with version 0.9!

Hypervisor
IaaS (Infrastructure as a Service)

E.g., brining up a VM on AWS.

IANA (Internet Assigned Numbers Authority)
ICANN (Internet Corporation for Assigned Numbers and Names)
ICS
IDE (Integrated Development Environment)
IdP (Identity Provider)
IDS (Intrusion Detection System)
Incident Response Team

Variations include: Computer emergency Response team, Security incident Response Team, etc.

IPAM (IP Address Management)
IPI (IP Intelligence)

At least in the world of F5 this means IP Intelligence, i.e., the reputation of a given IP address.

IPS (Intrusion Prevention System)
IPSEC
IPv6 (Internet Protocol version 6)
ISO 9001
ISP (Internet Service Provider)
ITIL (IT Infrastructure Library)
Kerning

Adjusting the spacing between letyters in a proportional font.

Kernel mode
Kubernetes
L2TP (Layer 2 Tunneling Protocol)
L3, L4, L7 (Layer 3, Layer 4, Layer 7)

Refers to ISO 7-layer traffic model.

LAMP (Linux Apache MySQL and PHP)

An application stack which gives a server needed software to do “interesting things.”

LaTEX

A markup language based on TEX I used to use to write a scientific paper. I think it gets transformed into a DVI, and then into a postscript file.

LEC (Local Exchange Carrier)
LLD (Low Level Design)
LLD (Low Level Discovery)

A command-line browser for unix systems.

LLDP (Link layer Discovery Protocol)

See also CDP

LACP (Link Access Control Protocol)
Link
Linux

An open source OS similar to Unix.

LLM (Large Langiuage Model)
lynx

A command-line browser for linux systems.

Machine Learning

A subfield of AI that deals with creating systems that can learn from data and improve their performance without explicit programming.

Mandiant
MD5 (Message Digest 5)
MDM (Mobile Device Management)

Management software used to administer smartphones and tablets.

MFA (Multi Factor Authentication)
MITRE ATT&CK
Modbus protocol
MS-CHAPv2
MSS (Maximum Segment Size)

Set by a TCP option in the beginning of the communcation.

MTU (Maximum transmission unit)

Often 1500 bytes.

multicast
Named pipes

I read it’s a Windows thing. huh. Hardly. It’s been on unix systems long before it was a twinkle in the eye of Bill gates. It acts like a pipe (|) except you give it a name in the filesystem and so it is a special file type. It’s used for inter-process communication.

NDA (Non-Disclosure Agreement)
.NET
NGFW (Next Generation FireWall)

Palo Alto Networks describes their firewalls this way.

NGINX

A web server that is usually superioir to apache for most applications.

NLP (Natural Language Processing)

A branch of AI that uses machine learning to enable computers to understand, interpret, and respond to human language.

NOC (Network Operations Center)
North-South

Data movement from/to the data center. Also see East-West.

NSA (National Security Agency)
OAuth bearer token

A security token with the property that any party in possession of the token (a “bearer“) can use the token in any way that any other party in possession of it can.

OCR (Optical Character Recognition)
Offensive Security – see Red Team
OpenRoaming
openssl

A common open source implementation of SSL/TLS.

OS (Operating System)
OSFP (Open Shortest Path First)
OSS (Open Source Software)
OT (Operational Technology)
OWASP (Open Worldwide Application Security Project)

An online community that produces freely available articles, methodologies, documentation, tools, and technologies in the fields of IoT, system software and web application security.

PAP
Patch
PaaS (Platform as a Service)
PBR (Policy Based Routing)
PDF (Portable Document File)
PDU (Protocol Data Unit)
PEM (Privacy Enhanced Mail)

The format certificates are normally stored in.

PII (Personally Identifiable Information)
PKCS (Public Key Cryptography Standard)
PKI (Public Key Infrastructure)
PLC (programmable logic controller)
PO (Purchase Order)
POC (Point of Contact)
POC (Proof of Concept)
Port Channel
Portable Executable (PE)
POV (Proof of Value)
Private Cloud
Prompt Engineering

The practice of crafting effective prompts that elicit high-quality answers from generative AI tools.

PS (PostScript)

A file type I used to use. It is a vector-oriented language, stack-based, which tells the printer how to move its ink pens around the page. Before there was PDF, there was postscript.

PS (PowerShell)

A versatile scripting language developed by Microsoft and available on all Windows computers.

PTO (Paid Time Off)
Purple Team

Purple teams combine red team and blue team functions. See Red Team.

Python

A popular programming language, not the snake.

QSFP (Quad Small Form factor Pluggable)

A newer kind of SFP.

Rack Unit
RADIUS
RAG (Retrieval Augmented Generation)

A method to train LLMs.

Ray

An open-source unified compute framework used by the likes of OpenAI, Uber, and Amazon which simplifies the scaling of AI and Python workloads, including everything from reinforcement learning and deep learning to tuning and model serving.

RBAC (Role-Based Access Control)
RDP (Remote Desktop Protocol)
Red Team

 In a red team/blue team exercise, the red team is made up of offensive security experts who try to attack an organization’s cybersecurity defenses.

Redirect
Remediation

Addressing a security flaw.

Remote Desktop Licensing (RDL) services

Often deployed on Windows severs with Remote Desktop Services deployed.

Retrieval-Augmented Generation (RAG)
Reverse Proxy

A TCP gateway which terminates a tcp connection and maintains a separate tcp connection to a back-end server.

RFC (Request for Comment)
RFI (Requst for Information)
RFO (Reason for Outage)
RFP (Request for Proposal)
RFQ (Request for Quote)
RIPE
RIR (Regional Internet Registry)
RMA (Return Merchandise Authorization)

You hear this a lot when It guys need to get a replacement for failed equipment.

ROA (Route Origin Authorization)
ROCE (Return on Capital Employed)

Hey, an IT person has to know a few business terms!

RPKI (Resource Public key Infrastructure)

Provides a way to connect Internet number resource information to a trust anchor.

RPi (Raspberry Pi)

A popular small, inexpensive server aimed at the educational crowd.

RPZ (Response Policy Zone)

A concept in DNS for either a DNS firewall or way to overwrite DNS responses.

RR (Resource Record)
RSA

Asymmetric encryption standard named after its creators, Ron Rivest, Adi Shamir and Leonard Adleman.

RTFM (Read The “flippin'” Manual)
SaaS (Software as a Service)
SAML
SANS

Private outfit in the US which specializes in information security and cybersecurity training.

Sans-Serif

A font type which does not have the fancy rounded blobs at the tips of the letter, such as Helvetica.

SASE (Secure Access Service Edge)
SCADA
SDWAN (Software defined WAN)
SEO (Search Engine Optimization)
SFP (Small Form factor Pluggable)

A type of optic transceiver that converts electric signals to optical signals.

SGML (Standard Generalized Markup Language)

If you ask the French they proudly point to this as the predeccesor to the more widely known HTML.

SFTP (Secure file Transfer Protocol)
SHA (Secure Hash Algorithm)
SIEM (Security Information and Event Management)
SRE (Site Reliability Engineer)
SMTP (Secure Mail Transfer Protocol)
SNI (Server Name Indication or similar, I think)

When multiple HTTP[S web sites whare a single IP this technology can be used to identify which certificate to send to a requester.

Spoofing

When a source IP address is faked.

SSH
SSL (Secure Socket Layer)
SOC (System on a Chip)

I believe the RPi is described to be this.

SOC (Security Operations Center)
SSL (Secure Sockets Layer)
SSL labs

A Qualys (so you know it has to be good quality) service where you can test a web site’s SSL certificate.

SSO (Single Sign On)
SVI (Switch virtual Interface)

A layer 3 on-switch routing between vlans on that switch. It’s a Cisco thing.

TAC (Technical Account or something?)
TAM (Technical Account Manager)

Another Cisco term.

Cisco uses this term a lot.

TCP (Transport Control Protocol)
Telnet
Terraform
The epoch

The first moment of January 1st, 1970

TI (Threat Intelligence)
TLP (Traffic Light Protocol)
TLS (Transport Layer Security)
udev rules

udev rules in Linux are used to manage device nodes in the /dev directory. Those nodes are created and removed every time a user connects or disconnects a device.

Unit testing
UPS (Uninterruptible Power Supply)
URL
UTC (Universal Time Coordinated)

What used to be called GMT.

VLAN
VM (Virtual Machine)
VMWare

Will Broadcom destroy this company the way they did to Bluecoat/Symantec?

VNC (Virtual Networking Computer)

VNC is a software used to remotely control a computer.

VPC (Virtual Private Cloud)
vPC (Virtual Port Channel)

A virtual port channel (vPC) allows links that are physically connected to two different Cisco FEXes to appear as a single port channel by a third device.

VPN – Virtual Private Network
Webhook
Website
Wiki

A less formal and usually more collaborative approach to documentation, the prime example being Wikipedia.

Windows PE or Win PE

A small OS for repairing or restoring Windows systems.

WWW (World Wide Web)
YARA
Categories
CentOS Debian Linux

What happened to insert mode on the latest version of vi?

Intro

As a creature of habit, I fall for an editor and can never imagine using something else. In the VAX days there was EDT, which if memory serves was replaced by the even better VPU. On Ultrix we ran a pretty nice editor simply called e from Rand Corporation. Then there was the love affair with emacs, and finally for the last 30 years vi.

Well with my latest server, a Debian 12 machine, I was having trouble with insert mode, specifically, inserting text from my Windows clipboard. Never had problems before….

Well for some reason, if you want to insert text from the clipboard, you now use <CTRL.>-<SHIFT>V in command mode. Well, at least on Windows 11 running WSL 2 that seems to work. I now realize that doesn’t work from Windows 10 with WSL. I had better figure this out soon…

Windows 10 running WSL

The terminal type (check the TERM environment variable) is set toxterm-color256. I tried pasting any and all registers which is the standard thing you would do if you use the standard Internet advice. None of it worked for me. I finally realized on my own that – and this harkens back to my old days with the beloved VAX 780 – that if I set the terminal type to vt100 all was good! Seriously. Back in the day we had physical VT100 terminals. Well, before that I think there was a VT52? Then maybe a VT102. VT202 was a big upgrade. Anyway, initially I added the following line to my .bashrc file:

export TERM=vt100

and now I can insert clipboard text the way I always have (mouse right-click) in vi insert mode. This kludge was how we fixed a lot of terminal display issues in the old days. But now I see display from top is messed up! Probably other curses-based apps as well. So two steps forward, one step back. So now what I’ve done is removed that line from .bashrc and put the following lines in my .bash_aliases file:

# DrJ kludge to get vi to work and keep top working
alias top='export TERM=xterm-256color;\top'
alias vi='export TERM=vt100;\vi'

That \top harkens back to an old linux convention where a command preceded by \ invokes a program but ignores defined aliases for that program.

Conclusion

I have offered one possible solution to the can’t insert text from the clipboard problem into my vi: set the TERM environment variable to the old-fashioned vt100. Now I can once again right-click while in insert mode to paste in clipboard text.

This was a very vexing issue for a creature of habit such as me!

References and related

This whole issue came up only when I switched from CentOS 8 to Debian 12 as my back-end server. Believe me, Debian 12 is so superior in so many ways this little setback would never make a material impact in that decision. Here’s the write-up of my upgrade.

Categories
CentOS Debian Linux Raspberry Pi

drjohnstechtalk now runs on a modern OS

Intro

I’m thrilled to announce that the long-running blog drjohnstechtalk.com has now been migrated to a modern back-end operating system. drjohnetchtalk.com is, a far as I know, the only quality-written technical resource on the Internet which is not supported by ads. Instead it runs on a pay-it-forward approach, embracing the spirit of the old Internet before it was ruined by big money.

drjohnstechtalk.com has been providing solutions to obscure tech questions since 2011.

The details

I like to run my own server which I can use for other purposes as well. I think that approach used to be more common. Now it’s harder to find others using it. Anyway, my old hosting environment is a CentOS server. I had hoped it would last me up to 10 years! 10 years is about the duration of long-term support for Redhat linux. It’s a real pain to migrate a WordPress blog with lots of history where it is important to preserve the articles and the permalinks. This article documents the nightmare I put myself through to get that up and running. Before that there was a CentOS 6 server. Then in 2022 – only about two years later – I learned that CentOS was dead! IBM had killed it. I’m over-simplifying here somewhat, but not by much.

So my blog sort of limped on on this unsupported system, getting riskier by the day to run as I was missing out on security patches. Then my companyt accidentally included one of my blogs in a security scan and I saw I had some vulnerabilities. So I upgraded WordPress versions and plugin versions. So with up to date software, the stage was set to migrate to a newer OS. Further motivation was provided by the fact that after the WP upgrade, the pages loaded more slowly. And sometimes the site just collapsed and crashed.

I have come to love Debian linux due to my positive experience with running it on Raspberry Pis and a few other places. It tends to run more recent versions of open source software, for instance. So I chose a Debian linux server. Then I forget where I learned this. Perhaps I asked someone at work which web server to use, but the advice was to use nginx, not apache! This was very new to me as I had never run nginx, not that I was in love with apache.

So, anyway, here I am writing this on my shiny new Debian 12 bookworm server which is running an nginx web server! And wow my site loads so much faster now. It’s really striking…

Running WordPress in a subdirectory with nginx

There always has to be a hard part, right? This was really, really hard. I run WP in the subdirectory blog as you can see from any of my URLs. I must have scoured a dozen sites on how to do it, none of which completely worked for me. So I had to do at least some of the heavy lifting and work out a working config on my own.

Here it is:

# mostly taken from https://www.nginx.com/resources/wiki/start/topics/recipes/wordpress/
# but with some important mods
upstream php {
    server unix:/var/run/php/php8.2-fpm.sock;
}

server {
  listen 443 ssl;

    include snippets/self-signed.conf;


    server_name drjohnstechtalk.com www.drjohnstechtalk.com;

    root /web/drjohns;
    index index.php index.html;


    access_log /var/log/nginx/drjohns.access.log;
    error_log /var/log/nginx/drjohns.error.log;

    client_max_body_size 100M;

# the following section prevents wp-admin from infintely redirecting to itself!
    location /blog/wp-admin {
            root /web/drjohns;
            try_files $uri $uri/ /blog/wp-admin/index.php?$args;
    }

    location /blog {
            root /web/drjohns/blog;
            try_files $uri $uri/ /blog/index.php?$args;
    }
    location ~ \.php$ {
#NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
         include fastcgi_params;
         fastcgi_intercept_errors on;
         fastcgi_pass php;
         fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    }
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg) {
            expires max;
            log_not_found off;
    }
}

I had to add ths svg file type to ignore, the location directive that matches /blog/wp-admin/. I had to define the upstream label as php and refer to that label in fastcgi_pass. I had to figure out my correct version of fastcgi. I tossed out some location directives which weren’t too important to me.

I disabled the wp-hide-login plugin while I grappled with why I was getting first a 404 not found for /blog/wp-admin/, then later, the too many redirects error. But I still had the issue with it disabled. Once I resolved the problem by adding the /blog/wp-admin location directive – I seem to be the only one on the Internet offering this solution and no other solution worked for me! – then I re-enabled the hide login plugin. The other plugins are working I would say.

Firewall?

I gather the current approach to host-based firewall on Debian 12 is to run ufw. A really good article on setting it up is here: https://www.cyberciti.biz/faq/set-up-a-firewall-with-ufw-on-debian-12-linux/

I’m on the fence about it, fearing it might slow my speedy server. But it looks pretty good. So for now I am relying on AWS Network Security Group rules. Did you know you can ask them to increase your max rule quota frmo 20 to 40? Yes, you can. I did and got approved overnight. I have added the Cloudflare ranges.

Cloudflare

I continue to use Cloudflare as reverse proxy, certificate issuer, DNS provider and light security screening. The change to the new server did not alter that. But I needed a new config file to properly report the origin IP address in my access files. The following file does the trick for me. It is up to date as of February 2024, can be placed in your /etc/nginx/conf.d directory and called, e.g., cloudlfare.conf.

# up to date as of 2/2024
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 104.16.0.0/13;
set_real_ip_from 104.24.0.0/14;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 131.0.72.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 2400:cb00::/32;
set_real_ip_from 2606:4700::/32;
set_real_ip_from 2803:f800::/32;
set_real_ip_from 2405:b500::/32;
set_real_ip_from 2405:8100::/32;
set_real_ip_from 2c0f:f248::/32;
set_real_ip_from 2a06:98c0::/29;

real_ip_header CF-Connecting-IP;

The idea is that if the source IP of the HTTP connection to nginx is from the Cloudflare range of IPs, then this must represent a request proxied through Cloudflare and the original IP of the client is in the HTTP header CF-Connecting-IP, which nginx can report on. If not, just use the normal IP from the TCP connection.

Swap space

On CentOS I had to provide some swap space because otherwise apache + mariaDB + WordPress would easily send its cpu soaring. So far I have not had to do that with my new Debian 12! That is great… So I have a t2.small instance with 25 GB of gp2 storage (100 iops). The server is basically running with a 0.00 load average now. I don’t get a lot of traffic so I hope that infrastructure will suffice.

Set the timezone

My Debian system started out in the UTC timezone. This command confirms that:

sudo timedatectl

This command brings up a menu and i can change the timezone to US Eastern:

sudo dpkg-reconfigure tzdata

Automate patching

It hasn’t run yet, but I’m hoping this root crontab entry will automate the system updates:

59 2 * * 0 (date && apt-get update && apt-get upgrade -y) >> /home/admin/hosting/update.log 2>&1

Debian 12 lifecycle

There should be three years of full support plus two more years of long term support for a stable Debian release, if I’ve undrstood it correctly. So I believe I may hope to get five years out of my Bookworm version, give or take. Debian — Debian Releases

Fixing the vi editor

I’ve never really had a problem with vi until this server. I show how I fixed it in this blog post.

Status after a few days – not all positive news

Well after a few days I feel the server response has noticeably slowed. I could not run top because I messed up the terminal with my fix to vi! So in a panic I restarted mariadb which seemed to help performance a lot. I will have to figure out how to monitor for this problem and how best to address it. I’m sure it will return. Here is my monitor.sh script:

#!/bin/bash
# restart mariaDB if home page response becomes greater than one second
curl -m1 -o /dev/null -ksH 'Host:drjohnstechtalk.com' https://localhost/blog/
# if curl didn't have enough time (one sec), its exit status is 28
[ $? -eq 28 ] && (systemctl stop mariadb; sleep 3; systemctl start mariadb; echo mariadb restart at $(date))

I invoke it from root’s crontab every three minutes:

# check that our load time is within reason or else restart mariadb -DrJ 2/24
*/3 * * * * sleep 25;cd /home/admin/hosting; ./monitor.sh >> monitor.log 2>&1

I do love my kludges. I will be on the lookout for a better long-term solution.

Conclusion

The technical blogging web site drjohnstechtalk.com now runs on new infrastructure: Debian 12 running nginx. It is muich faster than before. The migration was moderately painful! I have shared the technical details on how I managed to do it. I hope that, unlike my previous platform of CentOS 8, this platform lasts me for the next 10 years!

References and related

My second article!

nginx’s own advice about how to configure it to run WordPress

Trying to upgrade WordPress brings a thicket of problems

One of many RPi projects of mine: Raspberry Pi light sensor project

ufw firewall for Debian 12

Debian — Debian Releases

Cloudflare, an added layer of security for your web site

IP Ranges | Cloudflare

What happened to insert mode on the latest version of vi?

Categories
Linux

How an ADO pipeline can modify its own repo

Intro

I wanted to run a job on an Azure DevOps pipeline which did a backup of DNS zones on Cloudflare and write the results, in the form of a compressed tar file, into the ADO repository since everyone on the team has access to it and knows how to make a clone of the repo.

My first attempts produced some stunningly bad results. I was wiping out recently created files in the repo and such. That is very undesirable.

The solution

By “stealing with pride” from colleagues and such, I arrived at this AFAIK working solution. Here is the yaml file.

trigger: none

pool:
  name: backup_agents

steps:
# next two lines needed so we can modify the git repo and add our backups
- checkout: self
  clean: true
  persistCredentials: true
  fetchDepth: 1

- script: pip3 install -vvv --timeout 60 -r Cloudflare-backup/requirements.txt
  displayName: 'Install requirements'

- script: python3 backup-all-zones.py
  displayName: 'Run script'
  workingDirectory: $(System.DefaultWorkingDirectory)/Cloudflare-backup
  env:
    CLOUDFLARE_API_TOKEN: $(cloudflare_api_token)
    PYTHONPATH: $(System.DefaultWorkingDirectory)/Cloudflare-backup:$(System.DefaultWorkingDirectory)
- script: |
    git config --global http.sslVerify false
    git config --global user.email "[email protected]"
    git config --global user.name "pipeline"
    cd Cloudflare-backup
    pwd
    ls
    git add backups/zones-*
    git commit -m "adding todays backup files"
    git push origin HEAD:refs/heads/main

schedules:
- cron: "47 23 * * *"
  displayName: Run the script at 23:47 UTC
  branches:
    include:
    - main

I’m not exactly where all the magic happens. I think the section at the top that does the self checkout must be important. Then, obviously, there are the git add/git commit -m/git push commands. I do not claim to understand the origin HEAD:refs/heads/main argument to git push. I just copied it from a working example.

And branches: include -main. I’m not sure what this does either.

I need a few more days of testing, to be really certain, but I no longer am reverting my repo to an old state as I was with my initial attempts which involved doing a git fetch and probably missed the self checkout step as well.

Conclusion

One day I hope to understand git. But that today is not today! nevertheless I got my ADO pipeline to add backup files to its own ADO repository! So that’s cool.

References and related

My own git cheatsheet

Cloudflare python api examples

Categories
Linux Raspberry Pi

Raspberry Pi light sensor project

Intro

I ultimately want to turn off the connected display when it is nighttime and the lights have been turned off. And I want it to turn itself back on during daylight. The reason is because my RPi-driven slideshow is running and somoeone may be sleeping in that room.

This is till a work in progress. Pardon our dust and gibberish. What I’ve already found out is too important to delay publication.

Equipment and skills
How to turn off and on an HDMI port with a Raspberry Pi 4

Honestly this is the most significant thing I have found in this investigation. The methods used for older RPis do not work! In other words you can run vcgencmd display_power 0 and clever variations of that command until you’re blue in the face and the thnig stubbornly won’t turn off. tvservice -o ? Nope. That’ll suggest that command was deprecated and use kmsprint instead.

But I can say as of this writing (Jan ’24) the kms* commands are not mature and do not permit you to turn off the display. kmsprint tells you some stuff, but it does not allow you to set things. Remember RPi is for the education and hobbyist crowd so we have to give them the slack to experiment and try new things, even when they aren’t fully formed.

Instead they give you a way to restore the old commands. Edit the file /boot/config.txt.

Make it look like this:

#Enable DRM VC4 V3D driver
#use the fake kms driver in place of the native kms driver so we can  control hdmi power -DrJ 1/24
##dtoverlay=vc4-kms-v3d
dtoverlay=vc4-fkms-v3d

Reboot. Congrats you are now using the fake kms driver (fkms) and now have compatibility with the old commands. But instead of using tvservice, for my purposes, I think vcgencmd is better because the frame buffer state is not lost.

So now this command will indeed turn off the display:

vcgencmd display_power 0 # turn hdmi display off

vegcmd display_power 1 # turns the HDMI display back on

vcgencmd display_power 0 # turn hdmi display off

Prepare for our light sensor

I really don’t know if I’ll ever get the light sensor to work or not. Anticipating that it will, I have created this GPIO callback routine in python which in my dreams will turn off the HDMI display when the room is dark and turn it back on when the ambient light crosses a threshold. Who knows… But the code is pretty cool because it permits you to play with it as well using software command you send to a GPIO pin.

I call it gpio_basic.py:

import RPi.GPIO as GPIO
import time
import os
import datetime

GPIO.setmode(GPIO.BCM)
channel=4
GPIO.setup(channel, GPIO.IN)
reading = GPIO.input(channel)
print('Initial Reading',reading,flush=True)
old_reading = reading
while True: # infinite loop
    time.sleep(4)
# rising means ambient light went from light to dark
    reading = GPIO.input(channel) # 1 => dark, D0 LED turns off; 0 if light
    if reading == old_reading: continue

# else section where the state has changed
    print('This is a change in state on channel',channel,'at ',datetime.datetime.now(),flush=True)
    print('Reading',reading)
    if reading == 1:
        print('Turning off the HDMI display...',flush=True)
        os.system("vcgencmd display_power 0")
    else:
        time.sleep(6)
        print('Turn on HDMI display...',flush=True)
        os.system("vcgencmd display_power 1")
    old_reading = reading

Now to play with it, in another window get into python and run these commands:

import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)
channel=4
GPIO.setup(channel, GPIO.IN)
GPIO.input(channel)
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_UP)
#or
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

Those commented out lines at the end are the key ones. By executing them you should see the display turn off or on, and your other program should output some verbiage. Now hopefully the sensor will work something like that!

Wiring

Don’t know if this will pan out. I envision wiring (RPi pin #’s on the left):

  • 1 – Vcc (+ 3.3 V voltage)
  • 6 – Gnd (ground)
  • 7 – D0 (digital out)

At this point I don’t see the point of wiring the fourth lead on the diode sensor (which is A0, analog out – the RPi cannot read analog out), but having only three of them wired doesn’t quite seem right either. My electronics knowledge is weak!

So maybe I’ve wasted my money on these sensors, or I need to read them with a microcontroller and use i2c to talk to the RPi (too much effort and too much expense for my taste), but I hope not. Will know soon…

My first sensor, the photodiode, works! But its threshold is near one end of the control, which isn’t so great. But it’s fun to play with. In other words, its low light sensitivity may not be adequate for my purposes where I need to distinguish low ambient light (in a room with only a glowing TV screen, for instance) from even lower ambient light (lights off). Turns out our human eyes are the best measuring devices! Actually this photo diode, properly tuned, is quite good! I believe there is some jitter in the measurements however. So it can jump around during low light a bit. I have to consider how much of a problem that is.

The photodiode has a power led and a light sensor led. They are both way too bright. I suspect they could even create a feedback loop. I covered both with masking tape, leaving room for the adjustment screw.

But much, much worse than the the photodiode is the photoresistor. That at best distinguishes between a decent amount of light, and quite dark. But the transition between the two is sticky (I believe that would be called hysteresis). It will not work at all for my purposes based on my initial testing. It cannot distinguish between low light and very low light no matter where the dial is set.

Since this is all working, we just need to make it permanent by starting at boot time. So in my crontab file I added this line:

# Turn monitor off and on depending on ambien light! - DrJ 1/9/24
@reboot sleep 42; python3 gpio_basic.py > gpio_basic.log 2>&1
RPi4 shown with the slick metal case and the photodiode during daytime
Measure temperature of the CPU and why it matters

vcgencmd measure_temp

With my new fancy aluminum, heat-dissipating case I get around 40.4° C. On my RPi 3, air-cooled it is around 49° C. Why the sudden concern around heat? I’m just beginning to suspect that you know those times when you use the command line and things just seem to freeze? I always just assumed it was a glitch in the WiFi. But maybe it was actually the cpu getting too hot and having to pause itself to avoid burning up. I would see this when transferring files on my RPi 4. so the RPi 4 probably really does run hot unless you take steps to cool it, and the aluminum case in the equipment list above is really cool, ha , ha.

Tips

The command pinout is very useful. Pinout docs here: https://www.raspberrypi.com/documentation/computers/raspberry-pi.html

References and related

This guide talks about explicit support for generic digital input devices, not just a button: https://randomnerdtutorials.com/raspberry-pi-digital-inputs-python/

This talks about built-in pull-up pull-down resistors configurable via software! Who knew?

This might be useful for simple GPIO stuff: https://tutorials-raspberrypi.com/raspberry-pi-gpio-explanation-for-beginners-programming-part-2/

Gory details with gory circuit diagram: https://www.circuits.dk/everything-about-raspberry-gpio/

Running multiple RPi slideshows

How to deal with th GPIO pins using python: https://sourceforge.net/p/raspberry-gpio-python/wiki/Inputs

This project used the RPi GPIO OUTPUT pins to control a power relay device

Appendix A

I started with this code, gpio_callback.py, but the one condition kept getting called when the GPIO pin was reading 1. So I wrtoe gpio_basic.py and use that instead.

import RPi.GPIO as GPIO
import time
import os

GPIO.setmode(GPIO.BCM)
channel=4
GPIO.setup(channel, GPIO.IN)
reading = GPIO.input(channel)
print('Initial Reading',reading)
def my_callback(channel):
# rising means ambient light went from light to dark
    print('This is a edge event callback function!')
    print('Edge detected on channel %s'%channel)
    print('This is run in a different thread to your main program')
    print('Gonna stop that slideshow now...')
    reading = GPIO.input(channel) # 1 => dark, D0 LED turns off; 0 if light
    print('Reading',reading)
    if reading == 1:
        time.sleep(1)
        print('Turning off the HDMI display...')
        os.system("vcgencmd display_power 0")
    else:
        time.sleep(1)
        print('Turn on HDMI display...')
        os.system("vcgencmd display_power 1")

# RISING,FALLING or BOTH. https://sourceforge.net/p/raspberry-gpio-python/wiki/Inputs/
GPIO.add_event_detect(channel, GPIO.BOTH, callback=my_callback)  # add edge detection on a channel
# for testing, use:
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_UP)
# or
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)
time.sleep(31415927) # one year

Categories
Debian Linux Raspberry Pi

My favorite bash scripting tips

Intro

The linux bash shell is great and very flexible. I love to use it and have even installed WSL 2 on my PCs so I can use it as much as possible. When it comes to scripting it’s not exactly my favorite. there is so much history it has absorbed that there are multiple ways to do everything: the really old way, the new way, the alternate way, etc. And your version of bash can also determine what features you can use. nevertheless, I guess if you stick to the basics it makes sense to use bash for simple scripting tasks.

So just like I’ve compiled all the python tips I need for writing my simple python scripts in one convenient, searchable page, I will now do the same for bash. No one but me uses it, but that’s fine.

Iterate (loop) over a range of numbers

END=255 # for instance to loop over an ocetet of an IP address
for i in $(seq 1 $END); do
  echo $i
done
# But if it's OK to just hard-wire start and end, then it's simpler to use:
for i in {1..255}; do echo $i; done

Infinite loop
while /bin/true; do...done

You can always exit to stop it.

Sort IPs in a sensible order

$ sort -n -t . -k1,1 -k2,2 -k 3,3 -k4,4 tmp

What directory is this script in?

DIR=$(cd $(dirname $0);pwd);echo$DIR

Guarantee this script is interpreted (run) by bash and not good ‘ole shell (sh)!
if [ ! "$BASH_VERSION" ] ; then
  exec /bin/bash "$0" "$@"
  exit
fi
Count total occurrences of the word print in a bunch of files which may or may not be compressed, storing the output in a file

print=0
zgrep -c print tst*|cut -d: -f2|while read pline; do prints=$((prints + pline));echo $prints>prints; done

Note that much of the awkwardness of the above line is to get around issues I had with variable scope.

Permitted characters in variable names

Don’t use _ as you might in python! Stick to alphanumeric, but also do not begin with a number!

Execute a command

I used to use back ticks ` in the old days. parentheses is more visually appealing:

print1=$(cat prints)

Variable type

No, variables are not typed. Everything is treated as a string.

Function definition

Put function definitions before they are invoked in the script. Invocation is by plain name. function syntax is as in the example.

sendsummary() {
# function execution statements go here, then close it out
} # optionally with a comment like end function sendsummary
sendsummary # invoke our sendsummary function
Indentation

Unlike python, line indentation does not matter. I recommend to indent blocks of code two spaces, for example, for readability.

Booleans and order of execution
[[ "$DEBUG" -eq "1" ]] && echo subject, $subject, intro, "$intro"

The second statement only gets executed if the first one evaluated as true. Now a more complex example.

[[ $day == $DAY ]] || [[ -n “$anomalies” ]] && { statements…}

The second expressions get evaluated if the first one is false. If either the first or second expressions are true, then the last expression — a series of statements in what is essentially an unnamed function, hence the enclosing braces — gets executed. The -n is a test to see of length of a string is non-zero. See man test.

Or just use old-fashioned if-then statements?

The huge problem with the approach above is that it may be hard to avoid that multiple statements get executed in their own forked shell. so if they’re trying set a variable, or even do an exit, it may not produce the desired result! I may need further research to refine my approach, but the old if – then clause works for me – no subshell needs to be created.

Conditionals

Note that clever use of && and || can in many cases obviate the need for a class if…then structure, but see thw warning above. But you can use if thens. An if block is terminated by a fi. There is an else statement as well as an elif (else if) statement.

grep conditionals
ping -c1 8.8.8.8|grep -iq '1 received'
[ $? -eq 0 ] && echo this host is alive

So the $? variable after grep is run contains 0 if there was a match and 1 if there was no match. -q argument puts grep in “quiet” mode (no output).

More sophisticated example testing exit status and executing multiple commands

#!/bin/bash
# restart mariaDB if home page response becomes greater than one second
curl -m1 -ksH 'Host:drjohnstechtalk.com' https://localhost/blog/ > /dev/null
# if curl didn't have enough time (one sec), its exit status is 28
[ $? -eq 28 ] && (systemctl stop mariadb; sleep 3; systemctl start mariadb; echo mariadb restart at $(date))

Note that I had to group the commands after the conditional test with surrounding parentheses (). That creates a code block. Without those the semicolon ; would have indicated the end of the block! A semicolon ; separates commands. Further note that I nested parentheses and that seems to work as you would hope. also note that STDOUT has been redirected by the greater than sign > to /dev/null in order to silently discard all STDOUT output. /dev/null is linux-specific. The windows equivalent, apparently, is nul. Use curl -so nul suppress output on a Windows system.

Reading in parameters from a config file

Lots of techniques demoed in this example!

# read in params from file QC.conf
IFS=$'\n'
echo Parameters from file
for line in $(<QC.conf); do
  [[ "$line" =~ ^# ]] || {
  pval=$(echo "$line"|sed 's/ //g')
  lhs=$(echo "$pval"|cut -d= -f1)
  rhs=$(echo "$pval"|cut -d= -f2)
  declare -g $lhs="$rhs"
  echo $lhs is ${!lhs}
  }
done

Note the use of declare with the -g (global) switch to assign a variable to a variable-defined variable name! Note the use of < to avoid creation of a subshell. Note the use of -P argument in grep so that it uses perl-style regex! Note the way to get the value of a variable whose name itself is represented by a variable var is ${!var}.

This script parses a config file with values like a = a_val, where spaces may or may not be present.

One square bracket or two?

I have no idea and I use whatever I get to work. All my samples work and I don’t have time to test all variations.

Variable scope

I really struggled with this so I may come back to this topic!

Variable interpolation

$variable will suffice for simple, i.e., one-word content. But if the variable contains anything a bit complex such as words separated by spaces, or containing unusual characters, better go with double quotes around it, “$variable”. And sometimes syntactically throw in curly braces to separate it from other elements, “${variable}”

Eval
eval="ls -l"
$eval # executes ls -l
Shell expansion
mv Pictures{,.old} # renames directory Pictures to Pictures.old
Poor man’s launch at boot time

Use crontab’s @reboot feature!

@reboot sleep 25; ./recordswitch.sh > recordswitch.log 2>&1

The above expression also shows how to redirect standard error to standard out and have both go into a file.

Use extended regular expressions, retrieving a positional field using awk, and how to subtract (or add) two numbers
t1=`echo -n $line|awk '{print $1}'` 
t2=`echo -n $line|awk '{print $4}'` 
# test for integer inputs 
[[ "$t1" =~ ^[0-9]+$ ]] && [[ "$t2" =~ ^[0-9]+$ ]] && downtime=$(($t1-$t2))

Oops, I used the backticks there! I never claim that my way is the best way, just the way that I know to work! I know of a zillion options to add or subtract numbers…

Get last field using awk
echo hi.there.111|awk -F\. '{print $NF}' # returns 111
Print all but the first field using awk

awk ‘{$1=””; print substr($0,2)}’

Why do assignments have no extra spaces?

It simply doesn’t work if you try to put in spacing around the assignment operator =.

Divert stdout and stderr to a file from within the script
log=/tmp/my-log.log
exec 1>$log 
exec 2>&1
Lists, arrays amd dictionary variables

I don’t think bash is for you if you need these types of variables.

Formatted date

date +%F

produces yyyy-mm-dd, i.e., 2024-01-25

date +%Y%m%d -> 20240417

Poor man’s source code versioning

The old EDT/TPU editor on VAX used to do this automatically. Now I want to save a version of whatever little script I’m currently working on in the ~/tmpFRI (if it’s Friday) directory to sort of spread out my work by day of the week. I call this script cpj so it’s easy to type:

#!/bin/bash
# save file using sequential versioning to tmp area named after this day - DrJ
DIR='~'/tmp$(date +%a|tr '[a-z]' '[A-Z]') # ~/tmp + day of the week, e.g., FRI
DIRREAL=$(eval "echo $DIR") # the real diretory we need
mkdir -p $DIRREAL
for file in $*; do
  res=$(ls $DIRREAL|egrep "$file"'\.[0-9]{1,}$') # look for saved version numbers of this filename
  if test -n "$res"; then # we have seen this file...
    suffix=$(echo $res|awk -F\. '{print $NF}')  # pull out just the number at the end
    nxt=$(($suffix+1)) # add one to the version number
    saveFile="${file}"."${nxt}"
  else # new file to archive or no versioned number exists yet
    [[ -f $DIRREAL/$file ]] && saveFile="$file".1
    [[ -f $DIRREAL/$file ]] || saveFile=""
  fi
  cp "$file" $DIRREAL/"$saveFile"
  [[ -n $saveFile ]] && target=$DIR/"$saveFile"
  [[ -n $saveFile ]] || target="$DIR"
  echo copying "$file" to "$target"
done

It is a true mis-mash of programming styles, but it gets the job done. Note the use of eval. I’m still wrapping my head around that. Also note the technique used to upper case a string using tr. Note the use of extended regular expressions and egrep. Note the use of tilde ~ expansion. I insist on showing the target directory as ~/tmpSAT or whatever because that is what my brain is looking for. Note the use of nested $‘s.

Now that cpj is in place I occasionally know I want to make that versioned copy before I launch the vi editor, so I created a vij in my bash alias file thusly:

vij () { cpj "$@";sleep 1;vi "$@"; }

Complementing these programs is my gitj script which pushes my code changes to my repository after running pyflakes for python files:

#!/bin/bash
file="$@"
status=0

pushfile() {
  git add "$file"
  echo -n "Enter comment: "
  read comment
  fullComment=$(echo -e ${file}: "${comment}\n[skip ci]")
  echo -e "The full comment will be:\n${fullComment}"
  git commit -m "$fullComment"
  git push
  date
}

suffix=$(echo $file|awk -F\. '{print $NF}')  # pull out just the file type
if [[ $suffix == py ]]; then
  echo python file. Now running pyflakes on it;pyflakes $file;status=$?
  if [[ $status -eq 1 ]]; then echo syntax error detected so no git commands will be run
    exit 1
  else # python file checked out
    pushfile
  fi
else # was not a python file
  pushfile
fi

Another example

I wrote this to retain one backup per month plus the last 28 days.

#!/bin/bash
# do some date arithmetic to preserve backup from first Monday in the month
#[[ $(date +%a) == "Wed" ]] && { echo hi; }
DEBUG=0
DRYRUN=''
[[ $DEBUG -eq 1 ]] && DRYRUN='--dry-run'
if [[ $(date +%a) == "Mon" ]] && [[ $(date +%-d) -lt 8 ]]; then
# preserve one month ago's backup!
  echo "On this first Monday of the month we are keeping the Monday backup from four weeks ago"
else
  d4wksAgo=$(date +%Y%m%d -d'-4 weeks') # four weeks ago
  oldBackup=zones-${d4wksAgo}.tar.gz
  git rm $DRYRUN backups/$oldBackup
fi
today=$(date +%Y%m%d)
todaysBackup=zones-${today}.tar.gz
git add $DRYRUN backups/$todaysBackup

It incorpoates a lot of the tricks I’ve accumulated over the years, too numerous to recount. But it’s a good example to study.

Calculate last weekday

today=$(date -u +%Y%m%d) # UTC date
# last weekday calculation
delta="-1"
[[ $(date -u +%a) != "Mon" ]] || delta="-3"
lastday=$(date -u +%Y%m%d -d"${delta} days")

Output the tab character in an echo statement

Just use the -e switch as in this example:

echo -e “$subnet\t$SSID”

Get top output in a non-interactive (batch) shell

top -b -n 1

Prompting for user input

echo -n “Give your input: “

read userInput

Print first 120 characters of each line in a text file

cat file | cut -c -120

Reverse the lines in a file

tac file > file-reversed # tac is cat in reverse!

Send email when there is no mailx, mail or postifx setup

Use curl!

curl –url smtp://mail-relay.com –mail-from $sender –mail-rcpt $recipient -T <(echo -e “$msg”)

Format json into something readable

curl json_api|python3 -m json.tool

Merge every other line in a file

sed ‘N;s/\n/ /’ file

Ending script on compound conditional can be a bad idea

I ended my script with this statement:

# send alerts if needed
[[ $notify -gt 0 ]] && alerting

Problem was, this last statement has normal value of 1 (first condition is false so second expression not evaluated) so whole script exits with value 1 and my ADO pipeline felt that was an error! Guess I’ll add an exit 0 at the end…

Editing file in place with sed

Thge -i switch to sed is designed to do your substitutions right in the file. Here’s an actual crontab entry where I used that switch:

35 22 * * * sed -i s'/enabled=0/enabled=1/' /etc/yum.repos.d/thousandeyes.repo > /dev/null 2>&1
Date of a file in seconds

The output from e.g., ls -l is unparseable. This will do the trick. Technically this reports the last modified time of filename in seconds.

echo $(($(date +%s) - $(date +%s -r "$filename")))

Conclusion

I have documented here most of the tecniques I use from bash to achieve simple yet powerful scripts. My style is not always top form, but as I learn better ways I will adopt and improve.

Categories
Linux Raspberry Pi

Multiple Raspberry Pi photo frames

Intro

I have previously shared my work on displaying pictures in a nice slideshow from a Google Drive to an HDMI monitor. All these years later, it is still working and every day we see a new slideshow or randomly yet thoughtfully chosen pictures.

Building on this, today I extended this solution to display these pictures on a second monitor in a different room.

Equipment
  • RPi 4 (what I happened to have around. RPi 3 would also be fine)
  • HDMI monitor
  • Raspberry Pi Lite OS
The details

I needed to install fbi (sudo apt-get install fbi).

I needed to copy over m4.pl from the primary display, and for good measure black.jpg.

The general idea is to copy the pics over to the second display once per day.

I call this program copyslideshow.sh.

#!/bin/bash
# copy pictures from primary source
rm -rf Pictures.old
mv Pictures{,.old}
sshpass -p raspberry scp -r [email protected]:mediashow .
sshpass -p raspberry scp -r [email protected]:Pictures .
./m4.pl  $HOME/mediashow >> m4.log 2>&1

Yeah so I hard-coded the RPi password which is still set to the default. I’m willing to take the risk

Then in crontab I added this line:

# get yesterday's pictures!
1 5 * * * killall m4.pl; ./copyslideshow.sh > copyslideshow.log 2>&1

Note that these displays are not synced. That would be a whole ordeal. In fact we thought it would be cool to display different pictures. So the second monitor will be showing yesterday’s slideshow from the main monitor.

Automating turn-on, turn-off of the HDMI display based on the ambient room light

Since this second slideshow is in a bedroom, I wanted to have it turn off when the lights were out, and turn back on again during daylight. This was a really interesting challenge for me as I got to use an inexpensive external sensor with my RPi. And I got it to work, and it works quite well if I say so myself. That’s all written up in this post.

Reference and related

I have written many variations on this same topic. I guess this is a good one.

How I automatically turn on the HDMI display in the morning and turn it off again at night.