I still use my home-grown slideshow software based on Raspberry Pi, which is quite a testament to its robustness as it has been running with only minor modifications for many years now. one recent improvement has been my addition of being able to handle photos from recent iPhones which save photos in the new-to-me HEIC format. My original implementation only handles JPEGs and PNG file types, so it was skipping all our recent iPhone photos.
I figured there just had to be a converter our there which would even work on the RPi, which of course there was, heif-convert. But it has an oddity when it comes to rotation. It converts the HEIC to a jpeg, fine, but it rotates them, but it also leaves all the EXIF meta data, including the orientation meta data, as is. This in turn means display software such as fbi may try to rotate the picture a second time. Or at least that’s what happened to my software where one of my steps is an explicit rotate. That step was creating a double rotation.
So I needed a tiny program which left all the EXIF meta data alone except the rotation, which it sets to 0, i.e., do not rotate. Seeing nothing out there, I developed my own.
The details
Here is that script, which I call 0orientation.py:
A colleague of mine in another timezone created the necessary DKIM records in Cloudflare for a new mail domain. There was panic as the mail team realized too late these records were not validating. I was called in to help. Unfortunately at the beginning I only my smartphone to work with. Did you ever try to do this kind of detail work with a smartphone? Don’t.
The details
The smartphone thing is worthy of a separate post. I was getting somewhere, but it is like working with both hands tied behind yuor back.
So the mail team is telling me the dkim record doesn’t validate and showing me a screenshot of something from mxtoolbox to prove it.
I of course want to know the details so I can verify my mistakes before anyone else gets to – that’s how I roll!
Well, mxtoolbox, has a free validator for these dkim records which is pretty useful. Go to Supertool, then click the dropdown and select DKIM. A DKIM record involves a domain and a selector. Here’s a real live example for Hurricane Electric which uses he.net as their sending mail domain. So in their DNS the DKIM txt record for them looks like this when viewed from dig:
This is the value for this record: henet-20240223-153551._domainkey.he.net
To validate this DKIM record in mxtoolbox we pull out the token in front of _domainkey and refer to it as the selector, and drop the _domainkey and enter it like this:
The problem with the DKIM entry I was assigned to rescue was that the DIM syntax check was not passing. Yet it looked just like the way the mail team requested. What is going on? How can this problem be broken down into smaller steps???
To be continued…
Appendix A
How did I know the exact selector for Hurricane Electric?
I looked at the SMTP headers of an email I received from them. I found this section:
d must stand for domain and s for selector. This is all considered public information, albeit somewhat obscure. So the domain is he.net and the selector is henet-20240223-153551.
This case was solved today. Now I just need to find the time to write it up!
I belong to a team which runs many dozens of dns servers. We have basic but thorough monitoring of these servers using both Zabbix and Thousandeyes. One day I noticed a lot of timeout alerts so I began to look into it. One mystery just led to another without coming any closer to a true root cause. There were many dead ends in the hunt. Finally our vendor came through and discovered something…
The details
The upshot are these settings we arrived at for an ISC BIND server:
This is in the options section of the named.conf file. That’s it! This is on a four-core server with 16 GB RAM. The default values are:
tcp-listen-queue: 10
tcp-clients: 10
tcp-idle-timeout: 60 seconds
Those defaults will kill you on any reasonably busy server, meaning, one which gets a couple thousand requests per second.
To be continued…
Conclusion
We encountered a tough situation on our ISC BIND DNS servers. TCP queries, and only TCP queries, were responded to slowsly at best or not at all. after many flase starts we found the solution was setting three tcp parameters in the options section of the configuration file, tcp-listen-queue, tcp-clients and tcp-idle-timeout. We’ve never had to mess with those parameters after literally decades of running ISC BIND. Yet we have incontrovertible proof that that is what was needed.
It’s convenient to name drop different types of cyber attacks at a party. I often struggle to name more than a few. I will try to maintain a running list of them.
But I find you cannot speak about cybersecurity unless you also have a basic understanding of information technology so I am including some of those terms as well.
As I write this I am painfully aware that you could simply ask ChatGPT to generate a list of all relevant terms in cybersecurity along with their definitions – at least I think you could – and come up with a much better and more complete list. But I refuse to go that route. These are terms I have personally come across so they have special significance for me personally. In other words, this list has been organically grown. For instance I plowed through a report by a major vendor specializing in reviewing other vendor’s offerings and it’s just incredible just how dense with jargon and acronyms each paragragh is: a motherlode of state-of-the-art tech jargon.
AiTM (Adversary in the Middle)
Baitortion
I guess an attack which has a bait such as a plum job offer combined with some kind of extortion? The usage was not 100% clear.
BYOVD (Bring Your Own Vulnerable Driver)
Clickfix infection chain
Upon visiting compromised websites, victims are redirected to domains hosting fake popup windows that instruct them to paste a script into a PowerShell terminal to fix an issue.
Collision attack
I.e., against the MD5 hash algorithm as done in the Blast RADIUS exploit.
Credential Stuffing Attack
I.e., password re-use. Takes advantage of users re-using passwords for different applications. Nearly three of four consumers re-use password this way. Source: F5. Date: 3/2024
Data Wiper
Authentication Bypass
See for instance CVE-2024-0012
Email bombing
A threat actor might flood a victom with spam then offer “assistance” to fix it.
Evasion
Malicious software built to avoid detection by standard security tools.
Password spraying
A type of attack in which the threat actor tries the same password with multiple accounts, until one combination works.
Port Scan
Host Sweep
Supply Chain attack
Social Engineering
Hacking
Hacktivist
I suppose that would be an activitst who uses hacking to further their agenda.
Living off the land
Data Breach
Keylogger
Darknet
Captcha
Click farms
Jackpotting
This is one of my favorite terms. Imagine crooks implanted malware into an ATM and were able to convince it to dispense all its available cash to them on the spot! something like this actually happened. Scary.
Overlay Attack
Example: When you open a banking app on your phone, malware loads an HTML phishing page that’s designed to look just like that particular app and the malware’s page is overlaid on top.
Payment fraud attack
In a recent example, the victim experienced “multiple fraudulently induced outbound wire transfers to accounts controlled by unknown third parties.”
Skimmer
XSS (Cross site Scripting)
bot
Anti-bot, bot defense
Mitigation
SOC
Selenium (Se) or headless browser
Obfuscation
PII, Personally Identifiable Information
api service
Reverse proxy
Inline
endpoint, e.g., login, checkout
scraping
Layer 7
DDOS
Carpet bombing DDOS attack
Many sources hitting many targets within the same subnet. See:
A social engineering attack where scammers target grandparents by pretending to be a grandchild in a bind.
GUI
(JavaScript) Injection
Command Injection
Hotfix
SDK
URL
GET|POST Request
Method
RegEx
Virtual Server
TLS
Clear text
RCA
SD-WAN
PoV
PoC
X-Forwarded-For
Client/server
Threat Intelligence
Carding attack
Source code
CEO Fraud
Phishing
Vishing
(Voice Phishing) A form of cyber-attack where scammers use phone calls to trick individuals into revealing sensitive information or performing certain actions.
Business email compromise (BEC)
Deepfake
Threat Intelligence
Social engineering
Cybercriminal
SIM box
Command and control (C2)
Typo squatting
Voice squatting
A technique similar to typo squatting, where Alexa and Google Home devices can be tricked into opening attacker-owned apps instead of legitimate ones.
North-South
East-West
Exfiltrate
Malware
Infostealer
Obfuscation
Antivirus
Payload
Sandbox
Control flow obfuscation
Buffer overflow
Use after free
Indicators of Compromise
AMSI (Windows Antimalware Scan Interface)
Polymorphic behavior
WebDAV
Protocol handler
Firewall
Security Service Edge (SSE)
Secure Access Service Edge (SASE)
Zero Trust
Zero Trust is a security model that assumes that all users, devices, and applications are inherently untrustworthy and must be verified before being granted access to any resources or data.
Zero Trust Network Access (ZTNA)
ZTA (Zero Trust Architecture)
Zero Trust Edge (ZTE)
Secure Web Gateway (SWG)
Cloud Access Security Broker (CASB)
Remote Browser Isolation (RBI)
Content Disarm and Reconstruction (CDR)
Firewall as a service
Egress address
Data residency
Data Loss Prevention (DLP)
Magic Quadrant
Managed Service Provider (MSP)
0-day or Zero day
User Experience (UX)
Watermark
DevOps
Multitenant
MSSP
Remote Access Trojan (RAT)
SOGU
2024. A remote access trojan.
IoC (Indicators of Compromise)
Object Linking and Embedding
(Powershell) dropper
Backdoor
Data Bouncing
A technique for data exfiltration that uses external, trusted web hosts to carry out DNS resolution for you
TTP (Tactics, Techniques and Procedures)
Infostealer
Shoulder surfing
Ransomware
Pig butchering
This is particularly disturbing to me because there is a human element, a foreign component, crypto currency, probably a type of slave trade, etc. See the Bloomberg Businessweek story about this.
A text-based interfaces that allow for remote server control.
Crypto Miner
RCE (Remote Code Execution)
Threat Actor
APT (Advanced Persistent Threat)
Compromise
Vulnerability
Bug
Worm
Remote Access VPN (RAVPN)
XDR (Extended Detection and Response)
SIEM (Security Information and Event Management)
User Entity Behavior Analytics (UEBA)
Path traversal vulnerability
An attacker can leverage path traversal sequences like “../” within a request to a vulnerable endpoint which ultimately allows access to sensitive files like /etc/shadow.
Tombstoning
Post-exploit persistence technique
Volumetric DDoS
MFA bomb
Bombard a user with notifications until they finally accept one.
Use-after-free (UAF)
A use-after-freevulnerability occurs when programmers do not manage dynamic memory allocation and deallocation properly in their programs.
Cold boot attack
A cold boot attack focuses on RAM and the fact that it is readable for a short while after a power cycle.
One of those annoying terms borrowed from the military that only marketing people like to throw around. It means what you think it might mean.
Blue Team – see Red Team
BSI (The German Federal Office for Information Security)
DLS (Data Leak Sites)
Sites where you can see who has had their data stolen.
Red Team
In a red team/blue team exercise, the red team is made up of offensive security experts who try to attack an organization’s cybersecurity defenses.
SNORT (Probably stands for something)
An open source rule-matching engine to scan network traffic and serve as an IDS.
IT terminology
2FA (2 Factor Authentication)
802.1x
ACL (Access Control List)
AD (Active Directory)
ADO (Azue DevOps)
AFK (Away From Keyboard)
AGI (Artificial General Intelligence)
AGI is the theory and development of computer systems that can act rationally.
AIOps
Applying AI to IT operations.
ANN (Artificial Neural Network)
Ansible
I would call it an open source orchestrator.
anti-aliasing
When you smooth out color in neighboring pixels.
anycast
Anydesk
A popular remote management software.
apache
A formerly popular open source web server which became bloated with features.
APM (Application Performance Management)
ARIN
ARM
A processor architecture from ARM Corporation, as opposed to, e.g., x86. Raspberry Pis use ARM. I think Androids do as well.
ARP (Address Resolution Protocol)
ASCII
An early attempt at representing alpha-numeric characters in binary. Was very english-focussed.
ASN (Autonomous System Number)
Each AS is assigned an autonomous system number, for use in Border Gateway Protocol routing
ASN.1 (Abstract Syntax Notation One)
A standard interface description language (IDL) for defining data structures that can be serialized and deserialized in a cross-platform way.
ASPA (Autonomous System Provider Authorization)
An add-on to RPKI that allows an ASN to create a record that lists which ASNs can be providers for that ASN. The concepts are “customer” (an ASN) and “providers” (a list of ASNs). This is used to do hop by hop checking of AS paths.
ASR (Aggregation Services Router)
A high-end Interent router offered by Cisco for business customers.
AV (anti-virus)
AWS (Amazon Web Services)
Azure AD
BGP (Border Gateway Protocol)
BIND (Berkeley Internet Name Daemon)
An open source implementation of DNS, found on many flavors of linux.
BOM (Bill of Material)
Boot start
A flag for a driver in Windows that tells it to always start on boot.
bootp
A predecessor protocol to DHCP.
broadcast
Browser
BYOD (Bring Your Own Device)
I.e., when employees are permitted to use their personal smartphone to conduct company business.
BYOL (Bring Your Own License)
F5 permits this approach to licensing one of their cloud appliances.
CA (Certificate Authority)
Callback
A routine designed to be called when someone else’s code is executing. At least that’s how I understand it.
CDR (Call Detail Record)
Metadata for a phone call.
CDN (Content Distribution Network)
CDP (Cisco Discovery Protocol)
This protocol allows devices connected to switch ports to learn what switch and which switch port they are connected to. It is a layer 2 protocol.
CDSS (Cloud Delivered Security Services)
Only used in Palo Alto Networks land.
CGN (Carrier Grade NAT)
The address space 100.64.0.0/10 is handled specially by ISPs for CGN. RFC 6598
CHAP
Chatbot
A computer program that simulates human conversation with and end user.
CI (Configuration Item)
CI/CD
An ITIL term referring to the object upon which changes are made.
CISA (Cybersecurity and Infrastructure Security Agency)
CISO (Chief Information Security Officer)
Cleartext
Format where no encryption has been applied.
CMDB (Configuration Management Database)
CMO (Current Mode of Operations)
CNN (Congruential Neural Network)
Computer Vision
A field of AI that leverages machine learning and neutral networks to enable machines to identify and understand visual information such as images and videos.
Copilot
Microsoft’s AI built into their productivity software. Sorry, no more Clippy.
Courrier
A well-known fixed-width font.
CPE (Customer Premise Equipment)
CSR (Certificate Signing Request)
CUPS (Common Unix Printing Systems)
Customer Edge (CE)
CVE
CVEs, or Common Vulnerabilities and Exposures, are a maintained list of vulnerabilities and exploits in computer systems. These exploits can affect anything, from phones to PCs to servers or software. Once a vulnerability is made public, it’s given a name in the format CVE–. There are also scoring systems for CVEs, like the CVSS (Common Vulnerability Scoring System), which assigns a score based on a series of categories, such as how easy the vulnerability is to exploit, whether any prior access or authentication is required, as well as the impact the exploit could have.
CVSS (Common Vulnerability Scoring System)
Part of CVE lingo.
DAST (Dynamic Application Security Testing)
Data at rest
Data in motion
Data Plane
A physical security appliance separates data traffic from its management traffic, which transits the managemenbt plane.
Data Remanence
The residual representation of data that remains even after attempting to erase or initialize RAM.
DDI (DNS, DHCP and IP address management)
Debian Linux
A nice distro which I prefer. It is free and open source. Its packages are relatively uptodate.
Deep Learning
A subset of machine learningthat focus on using deep neural networks with multiple layers to model complex patterns in data.
Deepfake
A manipulated video or other digital representation produced by sophisticated machine-learning techniquies that yield seemingly realistic, but fabricated images and sounds.
DHCP (Dynamic Host Control Protocol)
Distributed Cloud
A Gartner term for a SaaS service which runs over multiple cloud environments.
DLL
DLP (Data Loss Prevention)
DNAT (Destination NAT)
DNS (Domain Name System)
DNSSEC (Domain Name System Security Extensions)
DOA (Dead on Arrival)
Usage: That equipment arrived DOA!
Docker
DoH (DNS over HTTPS)
Domain
DRM (Digital Rights Management)
DVI (DeVice Independent file)
See LaTEX entry.
EAP
East-West
Data movement with a data center, I believe, as oppose to North-South.
EBITDA (Earnings Before Interest, Taxes, Depreciation and Amortization)
Hey, an IT person needs to know some business terminology!
Eduroam
Enhanced Factory Reset (EFR)
Entra
From Microsoft. The new name for Azure AD
EU AI Act
EULA (End User Licnese Agreement)
Exact Data Matching (EDM)
FAQ (Frequently Asked Questions)
Fedora Linux
Free and open source linux. New features are introduced here before migrating into Redhat Linux
FEX (Fabric Extender)
FIFO (First in, First Out)
FIPS (Federal Information Processing Standard)
Government security practices. Best to avoid if possible.
FMO (Future Mode of Operation)
As opposed to CMO.
FN (False Negative)
Forensics
Fortran
An ancient procedural programming language popular in the scientific and engineering communities from decades ago.
FOSS (Free and Open Source Software)
FP (False Positive)
freeBSD
A Unix variant which still exists today.
Fritz!Box
A popular home router in Germany.
GA (General Availability)
Gartner Group
A well-regarded research firm which reviews software and SaaS products. They decide which vendors are in the Magic Quadrant.
GBIC
A type of fiber optic transceiver that converts electric signals to optical signals.
GCP (Google Cloud Provider)
GDPR (General Data Protection Regulation)
An EU directive to achieve data privacy.
Generative AI
AI which can create new human-quality content, including text, images, audio or video.
GMP (Good Manufacturing Practice)
FDA lingo that implies their rules are being followed.
GMT – see UTC
GRE
GSLB (global Server Load Balancing)
GUI (Graphical User Interface)
HA (High Availability)
Hallucination
When an LLM perceives patterns that are non-existent creating nonsensical or inaccurate outputs.
Hands and Eyes
When you don’t have physical access to a server, you need someone who does to be this for you.
At least in the world of F5 this means IP Intelligence, i.e., the reputation of a given IP address.
IPS (Intrusion Prevention System)
IPSEC
IPv6 (Internet Protocol version 6)
iRule
F5 specific lingo for programmable control over load-balancing and routing decisions. Uses the TCL language.
ISC (Internet Software Consortium)
A body which maintains an open source reference implementation for DNS (BIND) and DHCP.
ISO 9001
ISP (Internet Service Provider)
ITIL (IT Infrastructure Library)
JSON (JavaScript Object Notation)
Pronounced JAY-son. A popluar format for data exchange. Sort of human-friendly. Example: {“hi”:”there”,”subnets_ignore”:[“10/8″,”192.168/16”]}
Kanban
Agile way of tracking progress on tasks and brief meetings.
Kernel mode
Kerning
Adjusting the spacing between letters in a proportional font.
KEV (Known Exploited vulnerabilities)
CISA maintains this catalog.
K8s (Kubernetes)
Open source system for automating deployment, scaling, and management of containerized applications
KVM (Kernel Virtual Module)
L2TP (Layer 2 Tunneling Protocol)
L3, L4, L7 (Layer 3, Layer 4, Layer 7)
Refers to ISO 7-layer traffic model.
LACP (Link Access Control Protocol)
LAMP (Linux Apache MySQL and PHP)
An application stack which gives a server needed software to do “interesting things.”
LaTEX
A markup language based on TEX I used to use to write a scientific paper. I think it gets transformed into a DVI, and then into a postscript file.
LEC (Local Exchange Carrier)
Link
Linux
LLD (Low Level Design)
LLD (Low Level Discovery)
A command-line browser for unix systems.
LLDP (Link layer Discovery Protocol)
See also CDP
An open source OS similar to Unix.
LLM (Large Langiuage Model)
lynx
A command-line browser for linux systems.
MAC (Media Access Control) Address
Layer 2 address of a device, e.g., fa-2f-36-b4-8c-f5
Machine Learning
A subfield of AI that deals with creating systems that can learn from data and improve their performance without explicit programming.
Magic Quadrant
Gartner’s term for vendors who exceed in both vision and ability to execute.
Management Plane
See Data Plane.
Mandiant
MD5 (Message Digest 5)
MDM (Mobile Device Management)
Management software used to administer smartphones and tablets.
MFA (Multi Factor Authentication)
MITRE ATT&CK
Modbus protocol
MS-CHAPv2
MSS (Maximum Segment Size)
Set by a TCP option in the beginning of the communcation.
MTTI (Mean Time To Identification)
Probably only Cisco uses this acronym e.g., in their ThousandEyes product.
MTTR (Mean Time To Resolution)
MTU (Maximum transmission unit)
Often 1500 bytes.
multicast
NAESAD (North American Energy Software Assurance Database)
Named pipes
I read it’s a Windows thing. huh. Hardly. It’s been on unix systems long before it was a twinkle in the eye of Bill gates. It acts like a pipe (|) except you give it a name in the filesystem and so it is a special file type. It’s used for inter-process communication.
NAT (Network Address Translation)
NDA (Non-Disclosure Agreement)
.NET
Netflow
Think of it like a call detail record for IP communications. Metadata for a communications stream.
NGFW (Next Generation FireWall)
Palo Alto Networks describes their firewalls this way.
NGINX
A web server that is superioir to apache for most applications.
NLP (Natural Language Processing)
A branch of AI that uses machine learning to enable computers to understand, interpret, and respond to human language.
NOC (Network Operations Center)
North-South
Data movement from/to the data center. Also see East-West.
NSA (National Security Agency)
NTLM
Relies on a three-way handshake between the client and server to authenticate a user.
OAuth bearer token
A security token with the property that any party in possession of the token (a “bearer“) can use the token in any way that any other party in possession of it can.
An online community that produces freely available articles, methodologies, documentation, tools, and technologies in the fields of IoT, system software and web application security.
PAP
Patch
PaaS (Platform as a Service)
PBR (Policy Based Routing)
PCI (Payment Card International?)
A standard which seeks to define security practices around the handling of credit cards.
PDF (Portable Document File)
PDU (Protocol Data Unit)
PE (Provider Edge)
Telecom lingo so cisco uses this term a lot.
PEM (Privacy Enhanced Mail)
The format certificates are normally stored in.
PHP (Probably stands for something)
A scripting language often used to program back-end web servers.
PII (Personally Identifiable Information)
PKCS (Public Key Cryptography Standard)
PKI (Public Key Infrastructure)
Plain Text
A human-readable format, i.e., no encyrption and not a binary file.
PLC (programmable logic controller)
PM (Product Manager)
Could also be Project Manager but for me it usually means Product Manager.
PO (Purchase Order)
POC (Point of Contact)
POC (Proof of Concept)
Port Channel
Portable Executable (PE)
POTS (Plain Old Telephone Service)
Voice-grade telephone service employing analog signal transmission over copper
POV (Proof of Value)
Private Cloud
Prompt Engineering
The practice of crafting effective prompts that elicit high-quality answers from generative AI tools.
PS (PostScript)
A file type I used to use. It is a vector-oriented language, stack-based, which tells the printer how to move its ink pens around the page. Before there was PDF, there was postscript.
PS (PowerShell)
A versatile scripting language developed by Microsoft and available on all Windows computers.
PTO (Paid Time Off)
Purple Team
Purple teams combine red team and blue team functions. See Red Team.
PyPi (Python Package Index)
Python
A popular programming language, not the snake.
QSFP (Quad Small Form factor Pluggable)
A newer kind of SFP.
Rack Unit
RADIUS
RAG (Retrieval Augmented Generation)
A method to train LLMs.
Ray
An open-source unified compute framework used by the likes of OpenAI, Uber, and Amazon which simplifies the scaling of AI and Python workloads, including everything from reinforcement learning and deep learning to tuning and model serving.
RBAC (Role-Based Access Control)
RDP (Remote Desktop Protocol)
Recursive
A function which calls itself.
Redhat Linux
A commercialized version of Fedora whose packages are always dated, usually by years.
Redirect
Remediation
Addressing a security flaw.
Remote Desktop Licensing (RDL) services
Often deployed on Windows severs with Remote Desktop Services deployed.
Retrieval-Augmented Generation (RAG)
Reverse Engineer
To figure out the basic building blocks or code by first observing behavior of a system.
Reverse Proxy
A TCP gateway which terminates a tcp connection and maintains a separate tcp connection to a back-end server.
RFC (Request for Comment)
RFI (Requst for Information)
RFO (Reason for Outage)
RFP (Request for Proposal)
RFQ (Request for Quote)
RIPE
RIR (Regional Internet Registry)
RMA (Return Merchandise Authorization)
You hear this a lot when It guys need to get a replacement for failed equipment.
RMM (ReMote Management)
ROA (Route Origin Authorization)
ROCE (Return on Capital Employed)
Hey, an IT person has to know a few business terms!
Route 53
In AWS-land, an intellugent DNS service, i.e., geoDNS +.
RPC (Remote Procedure Call)
RPKI (Resource Public key Infrastructure)
Provides a way to connect Internet number resource information to a trust anchor.
RPi (Raspberry Pi)
A popular small, inexpensive server aimed at the educational crowd.
RPZ (Response Policy Zone)
A concept in DNS for either a DNS firewall or way to overwrite DNS responses.
RR (Resource Record)
RSA
Asymmetric encryption standard named after its creators, Ron Rivest, Adi Shamir and Leonard Adleman.
RTFM (Read The “flippin'” Manual)
SaaS (Software as a Service)
SAML
SANS
Private outfit in the US which specializes in information security and cybersecurity training.
Sans-Serif
A font type which does not have the fancy rounded blobs at the tips of the letter, such as Helvetica.
SASE (Secure Access Service Edge)
SCADA
Scale sets
In cloud, a service which automates the build-up or tear-down of VMs behind a load balancer.
SDWAN (Software defined WAN)
SEO (Search Engine Optimization)
SFP (Small Form factor Pluggable)
A type of optic transceiver that converts electric signals to optical signals.
SGML (Standard Generalized Markup Language)
If you ask the French they proudly point to this as the predeccesor to the more widely known HTML.
SFTP (Secure file Transfer Protocol)
SHA (Secure Hash Algorithm)
SIEM (Security Information and Event Management)
SRE (Site Reliability Engineer)
SMP (Symmetric Multi Processing)
SMTP (Secure Mail Transfer Protocol)
SNAT (Source NAT)
SNI (Server Name Indication or similar, I think)
When multiple HTTP[S web sites whare a single IP this technology can be used to identify which certificate to send to a requester.
SNMP (Simple Network Management Protocol)
All security appliances support this protocol which permits system monitoring.
Spoofing
When a source IP address is faked.
SSH
SSL (Secure Socket Layer)
SOC (System on a Chip)
I believe the RPi is described to be this.
SOC (Security Operations Center)
Solaris
A Unix variant possibly still available. Offered by Oracle and formerly Sun Microsystems Corporation.
SSL (Secure Sockets Layer)
SSL labs
A Qualys (so you know it has to be good quality) service where you can test a web site’s SSL certificate.
TPM, a Microsoft security feature required by Windows 11, is a dedicated chip designed to provide “hardware-level security services for your device,” keeping your private information and credentials safe from unauthorized users.
TSF (Tech Support File)
Palo Alto Networks-specific lingo for a dump file they require for a firewall support case.
Ubuntu Linux
A commercialized implementation of Debian Linux from Canonical.
UC (Unified Communications)
Cisco likes this term.
udev rules
udev rules in Linux are used to manage device nodes in the /dev directory. Those nodes are created and removed every time a user connects or disconnects a device.
UI5
SAP’s UI for HTML 5.
Ultrix
A Unix variant which ran on DEC workstations.
Underlay
SD Wan terminology for the underlying network. As opposed to overlay.
Unit testing
UPS (Uninterruptible Power Supply)
URL
Use case
UTC (Universal Time Coordinated)
What used to be called GMT.
UTF-8
Common representation of common language characters. I think of it as a successor to ASCII.
Validated
In FDA parlance, an adjective used to describe a system which follows FDA controls.
VDI
A virtual desktop offered by Citrix.
VLAN
VM (Virtual Machine)
VMWare
Will Broadcom destroy this company the way they did to Bluecoat/Symantec?
VNC (Virtual Networking Computer)
VNC is a software used to remotely control a computer.
VPC (Virtual Private Cloud)
vPC (Virtual Port Channel)
A virtual port channel (vPC) allows links that are physically connected to two different Cisco FEXes to appear as a single port channel by a third device.
VPG (Virtual Port Group)
A Cisco-ism.
VPN – Virtual Private Network
VRF
A logically separated network when using MPLS.
WAF (Web Application Firewall)
Webhook
Website
Wiki
A less formal and usually more collaborative approach to documentation, the prime example being Wikipedia.
Windows PE or Win PE
A small OS for repairing or restoring Windows systems.
WWW (World Wide Web)
x86
A type of processor architecture. Found in most Windows PCs.
XHR (XMLHttpRequest)
I.e., ajax.
XML (eXtensible Markup Language)
Common file format for data exchange, but not too human-friendly.
As a creature of habit, I fall for an editor and can never imagine using something else. In the VAX days there was EDT, which if memory serves was replaced by the even better VPU. On Ultrix we ran a pretty nice editor simply called e from Rand Corporation. Then there was the love affair with emacs, and finally for the last 30 years vi.
Well with my latest server, a Debian 12 machine, I was having trouble with insert mode, specifically, inserting text from my Windows clipboard. Never had problems before….
Well for some reason, if you want to insert text from the clipboard, you now use <CTRL.>-<SHIFT>V in command mode. Well, at least on Windows 11 running WSL 2 that seems to work. I now realize that doesn’t work from Windows 10 with WSL. I had better figure this out soon…
Windows 10 running WSL
The terminal type (check the TERM environment variable) is set toxterm-color256. I tried pasting any and all registers which is the standard thing you would do if you use the standard Internet advice. None of it worked for me. I finally realized on my own that – and this harkens back to my old days with the beloved VAX 780 – that if I set the terminal type to vt100 all was good! Seriously. Back in the day we had physical VT100 terminals. Well, before that I think there was a VT52? Then maybe a VT102. VT202 was a big upgrade. Anyway, initially I added the following line to my .bashrc file:
export TERM=vt100
and now I can insert clipboard text the way I always have (mouse right-click) in vi insert mode. This kludge was how we fixed a lot of terminal display issues in the old days. But now I see display from top is messed up! Probably other curses-based apps as well. So two steps forward, one step back. So now what I’ve done is removed that line from .bashrc and put the following lines in my .bash_aliases file:
# DrJ kludge to get vi to work and keep top working
alias top='export TERM=xterm-256color;\top'
alias vi='export TERM=vt100;\vi'
That \top harkens back to an old linux convention where a command preceded by \ invokes a program but ignores defined aliases for that program.
Conclusion
I have offered one possible solution to the can’t insert text from the clipboard problem into my vi: set the TERM environment variable to the old-fashioned vt100. Now I can once again right-click while in insert mode to paste in clipboard text.
This was a very vexing issue for a creature of habit such as me!
References and related
This whole issue came up only when I switched from CentOS 8 to Debian 12 as my back-end server. Believe me, Debian 12 is so superior in so many ways this little setback would never make a material impact in that decision. Here’s the write-up of my upgrade.
I’m thrilled to announce that the long-running blog drjohnstechtalk.com has now been migrated to a modern back-end operating system. drjohnetchtalk.com is, a far as I know, the only quality-written technical resource on the Internet which is not supported by ads. Instead it runs on a pay-it-forward approach, embracing the spirit of the old Internet before it was ruined by big money.
drjohnstechtalk.com has been providing solutions to obscure tech questions since 2011.
The details
I like to run my own server which I can use for other purposes as well. I think that approach used to be more common. Now it’s harder to find others using it. Anyway, my old hosting environment is a CentOS server. I had hoped it would last me up to 10 years! 10 years is about the duration of long-term support for Redhat linux. It’s a real pain to migrate a WordPress blog with lots of history where it is important to preserve the articles and the permalinks. This article documents the nightmare I put myself through to get that up and running. Before that there was a CentOS 6 server. Then in 2022 – only about two years later – I learned that CentOS was dead! IBM had killed it. I’m over-simplifying here somewhat, but not by much.
So my blog sort of limped on on this unsupported system, getting riskier by the day to run as I was missing out on security patches. Then my companyt accidentally included one of my blogs in a security scan and I saw I had some vulnerabilities. So I upgraded WordPress versions and plugin versions. So with up to date software, the stage was set to migrate to a newer OS. Further motivation was provided by the fact that after the WP upgrade, the pages loaded more slowly. And sometimes the site just collapsed and crashed.
I have come to love Debian linux due to my positive experience with running it on Raspberry Pis and a few other places. It tends to run more recent versions of open source software, for instance. So I chose a Debian linux server. Then I forget where I learned this. Perhaps I asked someone at work which web server to use, but the advice was to use nginx, not apache! This was very new to me as I had never run nginx, not that I was in love with apache.
So, anyway, here I am writing this on my shiny new Debian 12 bookworm server which is running an nginx web server! And wow my site loads so much faster now. It’s really striking…
Running WordPress in a subdirectory with nginx
There always has to be a hard part, right? This was really, really hard. I run WP in the subdirectory blog as you can see from any of my URLs. I must have scoured a dozen sites on how to do it, none of which completely worked for me. So I had to do at least some of the heavy lifting and work out a working config on my own.
# mostly taken from https://www.nginx.com/resources/wiki/start/topics/recipes/wordpress/
# but with some important mods
upstream php {
server unix:/var/run/php/php8.2-fpm.sock;
}
server {
listen 443 ssl;
include snippets/self-signed.conf;
server_name drjohnstechtalk.com www.drjohnstechtalk.com;
root /web/drjohns;
index index.php index.html;
access_log /var/log/nginx/drjohns.access.log;
error_log /var/log/nginx/drjohns.error.log;
client_max_body_size 100M;
# the following section prevents wp-admin from infintely redirecting to itself!
location /blog/wp-admin {
root /web/drjohns;
try_files $uri $uri/ /blog/wp-admin/index.php?$args;
}
location /blog {
root /web/drjohns/blog;
try_files $uri $uri/ /blog/index.php?$args;
}
location ~ \.php$ {
#NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
include fastcgi_params;
fastcgi_intercept_errors on;
fastcgi_pass php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}
location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg) {
expires max;
log_not_found off;
}
}
I had to add ths svg file type to ignore, the location directive that matches /blog/wp-admin/. I had to define the upstream label as php and refer to that label in fastcgi_pass. I had to figure out my correct version of fastcgi. I tossed out some location directives which weren’t too important to me.
I disabled the wp-hide-login plugin while I grappled with why I was getting first a 404 not found for /blog/wp-admin/, then later, the too many redirects error. But I still had the issue with it disabled. Once I resolved the problem by adding the /blog/wp-admin location directive – I seem to be the only one on the Internet offering this solution and no other solution worked for me! – then I re-enabled the hide login plugin. The other plugins are working I would say.
I’m on the fence about it, fearing it might slow my speedy server. But it looks pretty good. So for now I am relying on AWS Network Security Group rules. Did you know you can ask them to increase your max rule quota frmo 20 to 40? Yes, you can. I did and got approved overnight. I have added the Cloudflare ranges.
Cloudflare
I continue to use Cloudflare as reverse proxy, certificate issuer, DNS provider and light security screening. The change to the new server did not alter that. But I needed a new config file to properly report the origin IP address in my access files. The following file does the trick for me. It is up to date as of February 2024, can be placed in your /etc/nginx/conf.d directory and called, e.g., cloudlfare.conf.
The idea is that if the source IP of the HTTP connection to nginx is from the Cloudflare range of IPs, then this must represent a request proxied through Cloudflare and the original IP of the client is in the HTTP header CF-Connecting-IP, which nginx can report on. If not, just use the normal IP from the TCP connection.
Swap space
On CentOS I had to provide some swap space because otherwise apache + mariaDB + WordPress would easily send its cpu soaring. So far I have not had to do that with my new Debian 12! That is great… So I have a t2.small instance with 25 GB of gp2 storage (100 iops). The server is basically running with a 0.00 load average now. I don’t get a lot of traffic so I hope that infrastructure will suffice.
Set the timezone
My Debian system started out in the UTC timezone. This command confirms that:
sudo timedatectl
This command brings up a menu and i can change the timezone to US Eastern:
sudo dpkg-reconfigure tzdata
Automate patching
It hasn’t run yet, but I’m hoping this root crontab entry will automate the system updates:
There should be three years of full support plus two more years of long term support for a stable Debian release, if I’ve undrstood it correctly. So I believe I may hope to get five years out of my Bookworm version, give or take. Debian — Debian Releases
Fixing the vi editor
I’ve never really had a problem with vi until this server. I show how I fixed it in this blog post.
Status after a few days – not all positive news
Well after a few days I feel the server response has noticeably slowed. I could not run top because I messed up the terminal with my fix to vi! So in a panic I restarted mariadb which seemed to help performance a lot. I will have to figure out how to monitor for this problem and how best to address it. I’m sure it will return. Here is my monitor.sh script:
#!/bin/bash
# restart mariaDB if home page response becomes greater than one second
curl -m1 -o /dev/null -ksH 'Host:drjohnstechtalk.com' https://localhost/blog/
# if curl didn't have enough time (one sec), its exit status is 28
[ $? -eq 28 ] && (systemctl stop mariadb; sleep 3; systemctl start mariadb; echo mariadb restart at $(date))
I invoke it from root’s crontab every three minutes:
# check that our load time is within reason or else restart mariadb -DrJ 2/24
*/3 * * * * sleep 25;cd /home/admin/hosting; ./monitor.sh >> monitor.log 2>&1
I do love my kludges. I will be on the lookout for a better long-term solution.
Conclusion
The technical blogging web site drjohnstechtalk.com now runs on new infrastructure: Debian 12 running nginx. It is muich faster than before. The migration was moderately painful! I have shared the technical details on how I managed to do it. I hope that, unlike my previous platform of CentOS 8, this platform lasts me for the next 10 years!
I wanted to run a job on an Azure DevOps pipeline which did a backup of DNS zones on Cloudflare and write the results, in the form of a compressed tar file, into the ADO repository since everyone on the team has access to it and knows how to make a clone of the repo.
My first attempts produced some stunningly bad results. I was wiping out recently created files in the repo and such. That is very undesirable.
The solution
By “stealing with pride” from colleagues and such, I arrived at this AFAIK working solution. Here is the yaml file.
trigger: none
pool:
name: backup_agents
steps:
# next two lines needed so we can modify the git repo and add our backups
- checkout: self
clean: true
persistCredentials: true
fetchDepth: 1
- script: pip3 install -vvv --timeout 60 -r Cloudflare-backup/requirements.txt
displayName: 'Install requirements'
- script: python3 backup-all-zones.py
displayName: 'Run script'
workingDirectory: $(System.DefaultWorkingDirectory)/Cloudflare-backup
env:
CLOUDFLARE_API_TOKEN: $(cloudflare_api_token)
PYTHONPATH: $(System.DefaultWorkingDirectory)/Cloudflare-backup:$(System.DefaultWorkingDirectory)
- script: |
git config --global http.sslVerify false
git config --global user.email "[email protected]"
git config --global user.name "pipeline"
cd Cloudflare-backup
pwd
ls
git add backups/zones-*
git commit -m "adding todays backup files"
git push origin HEAD:refs/heads/main
schedules:
- cron: "47 23 * * *"
displayName: Run the script at 23:47 UTC
branches:
include:
- main
I’m not exactly where all the magic happens. I think the section at the top that does the self checkout must be important. Then, obviously, there are the git add/git commit -m/git push commands. I do not claim to understand the origin HEAD:refs/heads/main argument to git push. I just copied it from a working example.
And branches: include -main. I’m not sure what this does either.
I need a few more days of testing, to be really certain, but I no longer am reverting my repo to an old state as I was with my initial attempts which involved doing a git fetch and probably missed the self checkout step as well.
Conclusion
One day I hope to understand git. But that today is not today! nevertheless I got my ADO pipeline to add backup files to its own ADO repository! So that’s cool.
I ultimately want to turn off the connected display when it is nighttime and the lights have been turned off. And I want it to turn itself back on during daylight. The reason is because my RPi-driven slideshow is running and somoeone may be sleeping in that room.
This is till a work in progress. Pardon our dust and gibberish. What I’ve already found out is too important to delay publication.
How to turn off and on an HDMI port with a Raspberry Pi 4
Honestly this is the most significant thing I have found in this investigation. The methods used for older RPis do not work! In other words you can run vcgencmd display_power 0 and clever variations of that command until you’re blue in the face and the thnig stubbornly won’t turn off. tvservice -o ? Nope. That’ll suggest that command was deprecated and use kmsprint instead.
But I can say as of this writing (Jan ’24) the kms* commands are not mature and do not permit you to turn off the display. kmsprint tells you some stuff, but it does not allow you to set things. Remember RPi is for the education and hobbyist crowd so we have to give them the slack to experiment and try new things, even when they aren’t fully formed.
Instead they give you a way to restore the old commands. Edit the file /boot/config.txt.
Make it look like this:
#Enable DRM VC4 V3D driver
#use the fake kms driver in place of the native kms driver so we can control hdmi power -DrJ 1/24
##dtoverlay=vc4-kms-v3d
dtoverlay=vc4-fkms-v3d
Reboot. Congrats you are now using the fake kms driver (fkms) and now have compatibility with the old commands. But instead of using tvservice, for my purposes, I think vcgencmd is better because the frame buffer state is not lost.
So now this command will indeed turn off the display:
vcgencmd display_power 0 # turn hdmi display off
vegcmd display_power 1 # turns the HDMI display back on
vcgencmd display_power 0 # turn hdmi display off
Prepare for our light sensor
I really don’t know if I’ll ever get the light sensor to work or not. Anticipating that it will, I have created this GPIO callback routine in python which in my dreams will turn off the HDMI display when the room is dark and turn it back on when the ambient light crosses a threshold. Who knows… But the code is pretty cool because it permits you to play with it as well using software command you send to a GPIO pin.
import RPi.GPIO as GPIO
import time
import os
import datetime
GPIO.setmode(GPIO.BCM)
channel=4
GPIO.setup(channel, GPIO.IN)
reading = GPIO.input(channel)
print('Initial Reading',reading,flush=True)
old_reading = reading
while True: # infinite loop
time.sleep(4)
# rising means ambient light went from light to dark
reading = GPIO.input(channel) # 1 => dark, D0 LED turns off; 0 if light
if reading == old_reading: continue
# else section where the state has changed
print('This is a change in state on channel',channel,'at ',datetime.datetime.now(),flush=True)
print('Reading',reading)
if reading == 1:
print('Turning off the HDMI display...',flush=True)
os.system("vcgencmd display_power 0")
else:
time.sleep(6)
print('Turn on HDMI display...',flush=True)
os.system("vcgencmd display_power 1")
old_reading = reading
Now to play with it, in another window get into python and run these commands:
Those commented out lines at the end are the key ones. By executing them you should see the display turn off or on, and your other program should output some verbiage. Now hopefully the sensor will work something like that!
Wiring
Don’t know if this will pan out. I envision wiring (RPi pin #’s on the left):
1 – Vcc (+ 3.3 V voltage)
6 – Gnd (ground)
7 – D0 (digital out)
At this point I don’t see the point of wiring the fourth lead on the diode sensor (which is A0, analog out – the RPi cannot read analog out), but having only three of them wired doesn’t quite seem right either. My electronics knowledge is weak!
So maybe I’ve wasted my money on these sensors, or I need to read them with a microcontroller and use i2c to talk to the RPi (too much effort and too much expense for my taste), but I hope not. Will know soon…
My first sensor, the photodiode, works! But its threshold is near one end of the control, which isn’t so great. But it’s fun to play with. In other words, its low light sensitivity may not be adequate for my purposes where I need to distinguish low ambient light (in a room with only a glowing TV screen, for instance) from even lower ambient light (lights off). Turns out our human eyes are the best measuring devices! Actually this photo diode, properly tuned, is quite good! I believe there is some jitter in the measurements however. So it can jump around during low light a bit. I have to consider how much of a problem that is.
The photodiode has a power led and a light sensor led. They are both way too bright. I suspect they could even create a feedback loop. I covered both with masking tape, leaving room for the adjustment screw.
But much, much worse than the the photodiode is the photoresistor. That at best distinguishes between a decent amount of light, and quite dark. But the transition between the two is sticky (I believe that would be called hysteresis). It will not work at all for my purposes based on my initial testing. It cannot distinguish between low light and very low light no matter where the dial is set.
Since this is all working, we just need to make it permanent by starting at boot time. So in my crontab file I added this line:
# Turn monitor off and on depending on ambien light! - DrJ 1/9/24
@reboot sleep 42; python3 gpio_basic.py > gpio_basic.log 2>&1
RPi4 shown with the slick metal case and the photodiode during daytime
Measure temperature of the CPU and why it matters
vcgencmd measure_temp
With my new fancy aluminum, heat-dissipating case I get around 40.4° C. On my RPi 3, air-cooled it is around 49° C. Why the sudden concern around heat? I’m just beginning to suspect that you know those times when you use the command line and things just seem to freeze? I always just assumed it was a glitch in the WiFi. But maybe it was actually the cpu getting too hot and having to pause itself to avoid burning up. I would see this when transferring files on my RPi 4. so the RPi 4 probably really does run hot unless you take steps to cool it, and the aluminum case in the equipment list above is really cool, ha , ha.
I started with this code, gpio_callback.py, but the one condition kept getting called when the GPIO pin was reading 1. So I wrtoe gpio_basic.py and use that instead.
import RPi.GPIO as GPIO
import time
import os
GPIO.setmode(GPIO.BCM)
channel=4
GPIO.setup(channel, GPIO.IN)
reading = GPIO.input(channel)
print('Initial Reading',reading)
def my_callback(channel):
# rising means ambient light went from light to dark
print('This is a edge event callback function!')
print('Edge detected on channel %s'%channel)
print('This is run in a different thread to your main program')
print('Gonna stop that slideshow now...')
reading = GPIO.input(channel) # 1 => dark, D0 LED turns off; 0 if light
print('Reading',reading)
if reading == 1:
time.sleep(1)
print('Turning off the HDMI display...')
os.system("vcgencmd display_power 0")
else:
time.sleep(1)
print('Turn on HDMI display...')
os.system("vcgencmd display_power 1")
# RISING,FALLING or BOTH. https://sourceforge.net/p/raspberry-gpio-python/wiki/Inputs/
GPIO.add_event_detect(channel, GPIO.BOTH, callback=my_callback) # add edge detection on a channel
# for testing, use:
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_UP)
# or
#GPIO.setup(channel, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)
time.sleep(31415927) # one year
The linux bash shell is great and very flexible. I love to use it and have even installed WSL 2 on my PCs so I can use it as much as possible. When it comes to scripting it’s not exactly my favorite. there is so much history it has absorbed that there are multiple ways to do everything: the really old way, the new way, the alternate way, etc. And your version of bash can also determine what features you can use. nevertheless, I guess if you stick to the basics it makes sense to use bash for simple scripting tasks.
So just like I’ve compiled all the python tips I need for writing my simple python scripts in one convenient, searchable page, I will now do the same for bash. No one but me uses it, but that’s fine.
END=255 # for instance to loop over an ocetet of an IP address
for i in $(seq 1 $END); do
echo $i
done
# But if it's OK to just hard-wire start and end, then it's simpler to use:
for i in {1..255}; do echo $i; done
Infinite loop
while /bin/true; do...done
You can always exit to stop it.
Sort IPs in a sensible order
$ sort -n -t . -k1,1 -k2,2 -k 3,3 -k4,4 tmp
What directory is this script in?
DIR=$(cd $(dirname $0);pwd);echo$DIR
Guarantee this script is interpreted (run) by bash and not good ‘ole shell (sh)!
if [ ! "$BASH_VERSION" ] ; then
exec /bin/bash "$0" "$@"
exit
fi
Count number of occurrences, even if string occurs multiple times in the same line
grep -o string filename|wc -l
Count total occurrences of the word print in a bunch of files which may or may not be compressed, storing the output in a file
Note that much of the awkwardness of the above line is to get around issues I had with variable scope.
Permitted characters in variable names
Don’t use _ as you might in python! Stick to alphanumeric, but also do not begin with a number!
Execute a command
I used to use back ticks ` in the old days. parentheses is more visually appealing:
print1=$(cat prints)
Variable type
No, variables are not typed. Everything is treated as a string.
Function definition
Put function definitions before they are invoked in the script. Invocation is by plain name. function syntax is as in the example.
sendsummary() {
# function execution statements go here, then close it out
} # optionally with a comment like end function sendsummary
sendsummary # invoke our sendsummary function
Indentation
Unlike python, line indentation does not matter. I recommend to indent blocks of code two spaces, for example, for readability.
The second expressions get evaluated if the first one is false. If either the first or second expressions are true, then the last expression — a series of statements in what is essentially an unnamed function, hence the enclosing braces — gets executed. The -n is a test to see of length of a string is non-zero. See man test.
Or just use old-fashioned if-then statements?
The huge problem with the approach above is that it may be hard to avoid that multiple statements get executed in their own forked shell. so if they’re trying set a variable, or even do an exit, it may not produce the desired result! I may need further research to refine my approach, but the old if – then clause works for me – no subshell needs to be created.
Conditionals
Note that clever use of && and || can in many cases obviate the need for a class if…then structure, but see thw warning above. But you can use if thens. An if block is terminated by a fi. There is an else statement as well as an elif (else if) statement.
grep conditionals
ping -c1 8.8.8.8|grep -iq '1 received'
[ $? -eq 0 ] && echo this host is alive
So the $? variable after grep is run contains 0 if there was a match and 1 if there was no match. -q argument puts grep in “quiet” mode (no output).
More sophisticated example testing exit status and executing multiple commands
#!/bin/bash
# restart mariaDB if home page response becomes greater than one second
curl -m1 -ksH 'Host:drjohnstechtalk.com' https://localhost/blog/ > /dev/null
# if curl didn't have enough time (one sec), its exit status is 28
[ $? -eq 28 ] && (systemctl stop mariadb; sleep 3; systemctl start mariadb; echo mariadb restart at $(date))
Note that I had to group the commands after the conditional test with surrounding parentheses (). That creates a code block. Without those the semicolon ; would have indicated the end of the block! A semicolon ; separates commands. Further note that I nested parentheses and that seems to work as you would hope. also note that STDOUT has been redirected by the greater than sign > to /dev/null in order to silently discard all STDOUT output. /dev/null is linux-specific. The windows equivalent, apparently, is nul. Use curl -so nul suppress output on a Windows system.
# read in params from file QC.conf
IFS=$'\n'
echo Parameters from file
for line in $(<QC.conf); do
[[ "$line" =~ ^# ]] || {
pval=$(echo "$line"|sed 's/ //g')
lhs=$(echo "$pval"|cut -d= -f1)
rhs=$(echo "$pval"|cut -d= -f2)
declare -g $lhs="$rhs"
echo $lhs is ${!lhs}
}
done
Note the use of declare with the -g (global) switch to assign a variable to a variable-defined variable name! Note the use of < to avoid creation of a subshell. Note the use of -P argument in grep so that it uses perl-style regex! Note the way to get the value of a variable whose name itself is represented by a variable var is ${!var}.
This script parses a config file with values like a = a_val, where spaces may or may not be present.
One square bracket or two?
I have no idea and I use whatever I get to work. All my samples work and I don’t have time to test all variations.
Variable scope
I really struggled with this so I may come back to this topic!
Variable interpolation
$variable will suffice for simple, i.e., one-word content. But if the variable contains anything a bit complex such as words separated by spaces, or containing unusual characters, better go with double quotes around it, “$variable”. And sometimes syntactically throw in curly braces to separate it from other elements, “${variable}”
Eval
eval="ls -l"
$eval # executes ls -l
Shell expansion
mv Pictures{,.old} # renames directory Pictures to Pictures.old
Oops, I used the backticks there! I never claim that my way is the best way, just the way that I know to work! I know of a zillion options to add or subtract numbers…
It simply doesn’t work if you try to put in spacing around the assignment operator =.
Divert stdout and stderr to a file from within the script
log=/tmp/my-log.log
exec 1>$log exec 2>&1
Lists, arrays amd dictionary variables
I don’t think bash is for you if you need these types of variables.
Formatted date
date +%F
produces yyyy-mm-dd, i.e., 2024-01-25
date +%Y%m%d -> 20240417
Poor man’s source code versioning
The old EDT/TPU editor on VAX used to do this automatically. Now I want to save a version of whatever little script I’m currently working on in the ~/tmpFRI (if it’s Friday) directory to sort of spread out my work by day of the week. I call this script cpj so it’s easy to type:
#!/bin/bash
# save file using sequential versioning to tmp area named after this day - DrJ
DIR='~'/tmp$(date +%a|tr '[a-z]' '[A-Z]') # ~/tmp + day of the week, e.g., FRI
DIRREAL=$(eval "echo $DIR") # the real diretory we need
mkdir -p $DIRREAL
for file in $*; do
res=$(ls $DIRREAL|egrep "$file"'\.[0-9]{1,}$') # look for saved version numbers of this filename
if test -n "$res"; then # we have seen this file...
suffix=$(echo $res|awk -F\. '{print $NF}') # pull out just the number at the end
nxt=$(($suffix+1)) # add one to the version number
saveFile="${file}"."${nxt}"
else # new file to archive or no versioned number exists yet
[[ -f $DIRREAL/$file ]] && saveFile="$file".1
[[ -f $DIRREAL/$file ]] || saveFile=""
fi
cp "$file" $DIRREAL/"$saveFile"
[[ -n $saveFile ]] && target=$DIR/"$saveFile"
[[ -n $saveFile ]] || target="$DIR"
echo copying "$file" to "$target"
done
It is a true mis-mash of programming styles, but it gets the job done. Note the use of eval. I’m still wrapping my head around that. Also note the technique used to upper case a string using tr. Note the use of extended regular expressions and egrep. Note the use of tilde ~ expansion. I insist on showing the target directory as ~/tmpSAT or whatever because that is what my brain is looking for. Note the use of nested $‘s.
Now that cpj is in place I occasionally know I want to make that versioned copy before I launch the vi editor, so I created a vij in my bash alias file thusly:
#!/bin/bash
file="$@"
status=0
pushfile() {
git add "$file"
echo -n "Enter comment: "
read comment
fullComment=$(echo -e ${file}: "${comment}\n[skip ci]")
echo -e "The full comment will be:\n${fullComment}"
git commit -m "$fullComment"
git push
date
}
suffix=$(echo $file|awk -F\. '{print $NF}') # pull out just the file type
if [[ $suffix == py ]]; then
echo python file. Now running pyflakes on it;pyflakes $file;status=$?
if [[ $status -eq 1 ]]; then echo syntax error detected so no git commands will be run
exit 1
else # python file checked out
pushfile
fi
else # was not a python file
pushfile
fi
Another example
I wrote this to retain one backup per month plus the last 28 days.
#!/bin/bash
# do some date arithmetic to preserve backup from first Monday in the month
#[[ $(date +%a) == "Wed" ]] && { echo hi; }
DEBUG=0
DRYRUN=''
[[ $DEBUG -eq 1 ]] && DRYRUN='--dry-run'
if [[ $(date +%a) == "Mon" ]] && [[ $(date +%-d) -lt 8 ]]; then
# preserve one month ago's backup!
echo "On this first Monday of the month we are keeping the Monday backup from four weeks ago"
else
d4wksAgo=$(date +%Y%m%d -d'-4 weeks') # four weeks ago
oldBackup=zones-${d4wksAgo}.tar.gz
git rm $DRYRUN backups/$oldBackup
fi
today=$(date +%Y%m%d)
todaysBackup=zones-${today}.tar.gz
git add $DRYRUN backups/$todaysBackup
It incorpoates a lot of the tricks I’ve accumulated over the years, too numerous to recount. But it’s a good example to study.
Problem was, this last statement has normal value of 1 (first condition is false so second expression not evaluated) so whole script exits with value 1 and my ADO pipeline felt that was an error! Guess I’ll add an exit 0 at the end…
Editing file in place with sed
Thge -i switch to sed is designed to do your substitutions right in the file. Here’s an actual crontab entry where I used that switch:
I have documented here most of the tecniques I use from bash to achieve simple yet powerful scripts. My style is not always top form, but as I learn better ways I will adopt and improve.
Note that these displays are not synced. That would be a whole ordeal. In fact we thought it would be cool to display different pictures. So the second monitor will be showing yesterday’s slideshow from the main monitor.
Automating turn-on, turn-off of the HDMI display based on the ambient room light
Since this second slideshow is in a bedroom, I wanted to have it turn off when the lights were out, and turn back on again during daylight. This was a really interesting challenge for me as I got to use an inexpensive external sensor with my RPi. And I got it to work, and it works quite well if I say so myself. That’s all written up in this post.
