In full disclosure this case was not one I contributed to in any way, unlike all the others I’ve reported on. Nevertheless, source who did work on this case told me sufficient details and it is an interesting case.
The setup
For this case to make any sense, you need to understand the background. If I got it right, some people were trying to restore a backup version of Windows 11 Professional. When they did this restore, they found the problem that they were not pciking up an IP address via dhcp if they were on a company network. If they did the restore while on a home office network it went OK.
So imagine the comlpexity in a modern IT environment this presents. You have the PV vendor, HP, the OS vendor, Microsoft, the dhcp service operator, in-house, the LAN service provider and the network gear vendor, Cisco. The fault could lie anywhere. They all initially claim their stuff is working fine (which is always the default statement) and look elsewhere.
So what I like to say is that any hypothesis is unlikely, yet one of them will prove to be correct, eventually.
More details
Packet traces showed the DHCP Discover request being sent by the PC, but not arriving to the DHCP server. Ah, you say, simple: the switch is guilty here of dropping the DHCP Discover packet, fix it. After all, “eating” dhcp packets is something misconfigured switches do all the time if dhcp snooping is misconfigured.
Yet the LAN service provider says the switch isn’t misconfigured. So they have to open a case with the switch vendor to understand the drop. I’m not sure where that support case went, meanwhile…
The in-house expert troubleshooters were able to take a second trace from a PC which did pick up an IP address after a restore. This restore feature of course used to work when it was initially released.
I still use my home-grown slideshow software based on Raspberry Pi, which is quite a testament to its robustness as it has been running with only minor modifications for many years now. one recent improvement has been my addition of being able to handle photos from recent iPhones which save photos in the new-to-me HEIC format. My original implementation only handles JPEGs and PNG file types, so it was skipping all our recent iPhone photos.
I figured there just had to be a converter our there which would even work on the RPi, which of course there was, heif-convert. But it has an oddity when it comes to rotation. It converts the HEIC to a jpeg, fine, but it rotates them, but it also leaves all the EXIF meta data, including the orientation meta data, as is. This in turn means display software such as fbi may try to rotate the picture a second time. Or at least that’s what happened to my software where one of my steps is an explicit rotate. That step was creating a double rotation.
So I needed a tiny program which left all the EXIF meta data alone except the rotation, which it sets to 0, i.e., do not rotate. Seeing nothing out there, I developed my own.
The details
Here is that script, which I call 0orientation.py:
While my laptop was being shipped to me I wanted to be as productive as possible using my Samsung Galaxy A35. I was vaguely aware of the availability of Microsoft 365 apps such as Outlook. How far could I take this…?
The recipe
To cut to the chase, I was maybe 60 – 70 % effective. I used equipment found in the typical IT person’s home plus one inexpensive purchase from Walmart.
Here is what I used:
HDMI monitor
old Amazon firestick
cheap bluetooth keyboard purchased from Walmart
phone stand
And here’s what I really wished I had but did not:
bluetooth mouse
Which apps worked well:
Outlook
Teams, especiallt chat, less so the meetings function
One Note
Edge
VPN client
I must say the bluetooth keyboard worked really well for doing some serious typing up of emails.
How the external monitor worked
So I “came up” (in quotes because I’m sure many others figured out this same thin) with the idea of casting my phone screen onto an external monitor by way of the screen mirroring capacibility available on even the oldest amazon Firestick. On the phone you simply go to Smart View Mirror Screen.
So that prevented me from having to hold the phone at least while I was drafting emails.
But, and it’s a big one, is that the external monitor was not a TV and the sound from meetings was killed by this setup. And I did not see a way to keep audio local to the phone while only casting the screen.
A smaller problem is that the refresh lag is quite noticeable under conditions of rapid screen refresh. So it may take a second or two to show what the phone’s screen shows.
Still, it’s pretty cool.
I would have bought a bluetooth mouse but it simply wasn’t available at my local Walmart. I was pretty inconvenienced without it having to constantly touch the phone screen for various things.
And the external keyboard
Pretty well. Even some shortcuts worked. Alt-TAB, which I use a lot to switch between apps has some kind of vaguely similar effect on the phone, but not to the point where I could rely on it usefully. The unlock shortcut button sort of woke the up the phone screen at least.
TAB helped me to pop from one field in the form to the next the way I would use it on a PC.
Overall responseiveness was satisfactory.
The small form factor was not a detriment, and maybe even an advantage since it’s so light and portable.
What if you have an HP G5 docking station lying around?
Well I do. It has a USB-C cord which you normally plug into your HP laptop. But I didn’t have the power supply for it so I couldn’t use it when I would have needed it. Well, it basically works with a Samsung phone – at least the keyboard and mouse worked. In my 10 second testing the attached HDMI display did not automatically show anything. Maybe there are some phone settings which would need to be changed. I didn’t mess with it at all.
But it’s cool seeing a mouse working. It suddenly paints a mouse pointer on your phone screen which you can move around and click to launch an app.
Apps are often baby implmentations
At first I struggeled with the Outlook app, trying to use it as though it were my full-blown Outlook client on my PC. It only had one week’s worth of messages, which was pretty limiting since I was out for more than a week. Then I had a lightbulb moment and remembered that the Web version of Outlook worked on my phone. So I switched to using Outlook through the Edge browser – much better for me. That’s https://outlook.office.com/ . I could get full history and therefore do more reliable searching through messages.
Responsive Design work-around
Sometimes the mobile app version of a web site just doesn’t have the featuires, but looks nice. Edge has a feature you can choose called View Desktop Site which gives you the “real” web site. Now it may look tiny, forcing you to expand and shrink with two fingers. But at least it will generally work.
Where is Notepad or Notepad++
I didn’t look for an app. I suppose there is one. Somtimes you just want to inspect your clipboard. I settled on pasting into a new draft Outlook email to do my visual inspection of my clipboard.
References and related
I prepared the above solution with one day’s notice. If you had a couple days you might check out the Samsung Dex. I guess it would work for modern Samsung Galaxy phones though I haven’t tried it myself.
The web version of business Outlook, which is a pretty good implementtion of the full-blown client is https://outlook.office.com/
A colleague of mine in another timezone created the necessary DKIM records in Cloudflare for a new mail domain. There was panic as the mail team realized too late these records were not validating. I was called in to help. Unfortunately at the beginning I only my smartphone to work with. Did you ever try to do this kind of detail work with a smartphone? Don’t.
The details
The smartphone thing is worthy of a separate post. I was getting somewhere, but it is like working with both hands tied behind yuor back.
So the mail team is telling me the dkim record doesn’t validate and showing me a screenshot of something from mxtoolbox to prove it.
I of course want to know the details so I can verify my mistakes before anyone else gets to – that’s how I roll!
Well, mxtoolbox, has a free validator for these dkim records which is pretty useful. Go to Supertool, then click the dropdown and select DKIM. A DKIM record involves a domain and a selector. Here’s a real live example for Hurricane Electric which uses he.net as their sending mail domain. So in their DNS the DKIM txt record for them looks like this when viewed from dig:
This is the value for this record: henet-20240223-153551._domainkey.he.net
To validate this DKIM record in mxtoolbox we pull out the token in front of _domainkey and refer to it as the selector, and drop the _domainkey and enter it like this:
The problem with the DKIM entry I was assigned to rescue was that the DIM syntax check was not passing. Yet it looked just like the way the mail team requested. What is going on? How can this problem be broken down into smaller steps???
To be continued…
Appendix A
How did I know the exact selector for Hurricane Electric?
I looked at the SMTP headers of an email I received from them. I found this section:
d must stand for domain and s for selector. This is all considered public information, albeit somewhat obscure. So the domain is he.net and the selector is henet-20240223-153551.
This case was solved today. Now I just need to find the time to write it up!
I belong to a team which runs many dozens of dns servers. We have basic but thorough monitoring of these servers using both Zabbix and Thousandeyes. One day I noticed a lot of timeout alerts so I began to look into it. One mystery just led to another without coming any closer to a true root cause. There were many dead ends in the hunt. Finally our vendor came through and discovered something…
The details
The upshot are these settings we arrived at for an ISC BIND server:
This is in the options section of the named.conf file. That’s it! This is on a four-core server with 16 GB RAM. The default values are:
tcp-listen-queue: 10
tcp-clients: 10
tcp-idle-timeout: 60 seconds
Those defaults will kill you on any reasonably busy server, meaning, one which gets a couple thousand requests per second.
To be continued…
Conclusion
We encountered a tough situation on our ISC BIND DNS servers. TCP queries, and only TCP queries, were responded to slowsly at best or not at all. after many flase starts we found the solution was setting three tcp parameters in the options section of the configuration file, tcp-listen-queue, tcp-clients and tcp-idle-timeout. We’ve never had to mess with those parameters after literally decades of running ISC BIND. Yet we have incontrovertible proof that that is what was needed.
It’s been awhile since I have added a case to the canon of It detective stories which I have personally solved. It’s not that things don’t need resolving. They do! But either they look like what has come before, so there’s nothing new, or they are so new I’m still in the middle of them and you never know if they will ever be solved… Such was the situation with today’s subject: WiFi calling.
WiFi calling, which most people are blissfully ignorant of, can be very necessary if you are in a large building which shields you from cell phone tower signals and does not have any in-building signal boosters. In this situation, as long as you’ve enabled WiFi calling on your phone, it will be smart enough upon seeing no cell signal, to switch to using WiFi, assuming an access point and WiFi is reachable.
Well, such is the case at some office building my company has. And wiFi calling was found to be OK for phones using T-Mobile. But not for Verizon. With Verizon (VZ) phones WiFi calling was at best unpredicatble: sometimes the call would go through and sometimes not.
Unfortunately there were a lot of parties involved in the communication path. WLCs (wireless LAN controllers) have access points (APs) connect to them. they in turn tunnel the communication to another site where the anchor controller resides. Then it gets handed off to a perimiter firewall for NATing and egress via Internet routers. The Internet routers have some sort of load-balancing in place. We don’t run them any more the way we used to. A vendor does that now. And firewalls are handled by a different group. And a different group is in charge of mobile devices. The phone also has a Global protect client and hence an always-on VPN connection. That part is run by yet another group! So you see how this gets impossibly messy. I realized I was in a pretty good place – probably th best place compared to anyone else – to do this troubleshooting however because I touched many of the groups or had “good friends” there.
What does failure look like?
On my phone, a failed attempt looks like this. I place a call, and it doesn’t go through. It also doesn’t not go through. I just never hear anything. I wait for up to a minute, because, who is going to wait more than a minute to hear something after they’ve dialed the number?
More details
At the site they convinced themselves that whereas one SSID works, a second SSID which actually uses the same path, does not. For my part I wasn’t so sure. Eventually under my fairly extensive testing I could produce the problem every time by rebooting my phone and then placing a WiFi call very quickly afterwards.
Fun aside: how to force WiFi calling even when you have signal
On an Android device go to airplace mode. Your WiFi is then disabled. But you can re-enable your WiFi and airplace mode will stay on! Now when you bring up the built-in voice calling app, you will see the green phone icon with a WiFi icon superimposed over it. That’s how you know you are placing a WiFi call.
But then if I did nothing for about 30 minutes, often my next attempted WiFi call would go through! Go figure. And the call after that would work as well, etc. But maybe a couple hours later the whole thing would break again. I don’t think they were that systematic in their testing.
Verizon to the rescue
After spinning our wheels helplessly we finally got a call with a tech engineer from Verizon who was helpful. Because at some point you think to yourself, the app developer of the phone should be able to instrument the voice app with verbose logging to say what it thinks the problem is. Let’s switch to the firewall where I have good access to the logs as well as a good colleague willing to grind it out with me. Well this is a Checkpoint firewall and the logs are filled with drops. Checkpoint logging says First packet isn’t SYN. So what the VZ guy said which helped us focus is that you want to look for the tunnels to 14.20.0.0/16 or something like that. maybe it’s more like 14.20.128.0/17, or something that rhymes with that! In any case, we didn’t believe the First packet isn’t SYN drops were hurting us too much as we get those a lot, yet things just work.
Then there were dns requests to 8.8.8.8. Why? That’s not the dns server we configured in dhcp (another one of my sub-specialties). And even if the right dns server was being used, it was always possible it was hitting a dns firewall rule. So that had to be ruled out. And it did seem dns did not play into this. Then there was the worrisome matter of the vpn tunnel created by GPC. What if, somehow, these packets were going over that tunnel? They shouldn’t, but what if they do? Well, then we should see that traffic in the GPC logs (another of my sub-specialties). We didn’t. So I became somewhat comfortable ruling out GPC.
So back to VZ. The guy said on our test call that he saw the tunnel initially established, then there was no more communication over it. And so the tester did not receive the test call for him. So when we looked for destination 141.207…, yeah we could see IKE and IPSEC communication. We could see a tunnel being estabvlished over udp port 500, thn further communication to that same destination over udp port 4500. These are pretty much the standard ports for IKE. the VZ guy said he did not have access to be able to do a trace on the IKE peer. We could do a packet trace on our firewall however.
More testing
So we never did see an official drop in the checkpoint logs. Still, I began to suspect that firewall and my colleague agreed with me, or at least agreed to try some things. But first, another red herring. the VZ guy suggested we could trace the packets on the phone with pcapdroid or something like that. So I got that running on my phone. But to work it creates its own IKE tunnel, uses completely different IP addressing, and just generally makes it impossible to account for these IKE packets going to VZ.
On Checkpoint you have a general setting for how it will handle “NAT traversal” for IKE connections. It looks like this:
By the way, tracing on the firewall isn’t all that easy since there are two interfaces. We actually were running tcpdump on the inward-facing interface while running fw monitor on the outbound interface! That’s not so easy to coordinate. Neither D nor I had ever done it before. We never did reach that Aha moment where you say, look, the packet destined for the tunnel enters here, and doesn’t go out here. There was just too much competing traffic. But anyway, D wanted to play with the NAT traversal settings, which seemed easier.
First adjustment: aggressive aging
The first thing D did was to turn off aggressive aging. Well, that helped a lot. With that, I was able to place my WiFi calls successfully every time after a reboot!
But this thing is tricky. We were chatting. Some time had passed. I placed another test call. Nope. that one didn’t go through! Drat. We had more homework to do. I had been recording the exact times of the calls pretty carefully. About 16 minutes had elapsed between the two calls.
To be continued…
Conclusion
In one of our most difficult cases, we got WiFi calling working reliably on Verizon phones. There were a lot of parties involved and a lot of false leads: look for asymmetric routing, etc.. The real problem was the IKE NAT traversal settings on a Checkpoint firewall. everyone involved is much happier now.
It’s convenient to name drop different types of cyber attacks at a party. I often struggle to name more than a few. I will try to maintain a running list of them.
But I find you cannot speak about cybersecurity unless you also have a basic understanding of information technology so I am including some of those terms as well.
As I write this I am painfully aware that you could simply ask ChatGPT to generate a list of all relevant terms in cybersecurity along with their definitions – at least I think you could – and come up with a much better and more complete list. But I refuse to go that route. These are terms I have personally come across so they have special significance for me personally. In other words, this list has been organically grown. For instance I plowed through a report by a major vendor specializing in reviewing other vendor’s offerings and it’s just incredible just how dense with jargon and acronyms each paragragh is: a motherlode of state-of-the-art tech jargon.
AiTM (Adversary in the Middle)
Baitortion
I guess an attack which has a bait such as a plum job offer combined with some kind of extortion? The usage was not 100% clear.
BYOVD (Bring Your Own Vulnerable Driver)
Clickfix infection chain
Upon visiting compromised websites, victims are redirected to domains hosting fake popup windows that instruct them to paste a script into a PowerShell terminal to fix an issue.
Collision attack
I.e., against the MD5 hash algorithm as done in the Blast RADIUS exploit.
Credential Stuffing Attack
I.e., password re-use. Takes advantage of users re-using passwords for different applications. Nearly three of four consumers re-use password this way. Source: F5. Date: 3/2024
Data Wiper
Authentication Bypass
See for instance CVE-2024-0012
Evasion
Malicious software built to avoid detection by standard security tools.
Password spraying
A type of attack in which the threat actor tries the same password with multiple accounts, until one combination works.
Port Scan
Host Sweep
Supply Chain attack
Social Engineering
Hacking
Hacktivist
I suppose that would be an activitst who uses hacking to further their agenda.
Living off the land
Data Breach
Keylogger
Darknet
Captcha
Click farms
Jackpotting
This is one of my favorite terms. Imagine crooks implanted malware into an ATM and were able to convince it to dispense all its available cash to them on the spot! something like this actually happened. Scary.
Overlay Attack
Example: When you open a banking app on your phone, malware loads an HTML phishing page that’s designed to look just like that particular app and the malware’s page is overlaid on top.
Payment fraud attack
In a recent example, the victim experienced “multiple fraudulently induced outbound wire transfers to accounts controlled by unknown third parties.”
Skimmer
bot
Anti-bot, bot defense
Mitigation
SOC
Selenium (Se) or headless browser
Obfuscation
PII, Personally Identifiable Information
api service
Reverse proxy
Inline
endpoint, e.g., login, checkout
scraping
Layer 7
DDOS
Carpet bombing DDOS attack
Many sources hitting many targets within the same subnet. See:
A social engineering attack where scammers target grandparents by pretending to be a grandchild in a bind.
GUI
(JavaScript) Injection
Command Injection
Hotfix
SDK
URL
GET|POST Request
Method
RegEx
Virtual Server
TLS
Clear text
MTTR
RCA
SD-WAN
PoV
PoC
X-Forwarded-For
JSON
Client/server
Threat Intelligence
Use case
Carding attack
Source code
CEO Fraud
Phishing
Vishing
(Voice Phishing) A form of cyber-attack where scammers use phone calls to trick individuals into revealing sensitive information or performing certain actions.
Business email compromise (BEC)
Deepfake
Threat Intelligence
Social engineering
Cybercriminal
SIM box
Command and control (C2)
Typo squatting
Voice squatting
A technique similar to typo squatting, where Alexa and Google Home devices can be tricked into opening attacker-owned apps instead of legitimate ones.
North-South
East-West
Exfiltrate
Malware
Infostealer
Obfuscation
Antivirus
Payload
Sandbox
Control flow obfuscation
Buffer overflow
Use after free
Indicators of Compromise
AMSI (Windows Antimalware Scan Interface)
Polymorphic behavior
WebDAV
Protocol handler
Firewall
Security Service Edge (SSE)
Secure Access Service Edge (SASE)
Zero Trust
Zero Trust is a security model that assumes that all users, devices, and applications are inherently untrustworthy and must be verified before being granted access to any resources or data.
Zero Trust Network Access (ZTNA)
ZTA (Zero Trust Architecture)
Zero Trust Edge (ZTE)
Secure Web Gateway (SWG)
Cloud Access Security Broker (CASB)
Remote Browser Isolation (RBI)
Content Disarm and Reconstruction (CDR)
Firewall as a service
Egress address
Data residency
Data Loss Prevention (DLP)
Magic Quadrant
Managed Service Provider (MSP)
0-day or Zero day
User Experience (UX)
Watermark
DevOps
Multitenant
MSSP
Remote Access Trojan (RAT)
SOGU
2024. A remote access trojan.
IoC (Indicators of Compromise)
Object Linking and Embedding
(Powershell) dropper
Backdoor
Data Bouncing
A technique for data exfiltration that uses external, trusted web hosts to carry out DNS resolution for you
TTP (Tactics, Techniques and Procedures)
Infostealer
Shoulder surfing
Ransomware
Pig butchering
This is particularly disturbing to me because there is a human element, a foreign component, crypto currency, probably a type of slave trade, etc. See the Bloomberg Businessweek story about this.
A text-based interfaces that allow for remote server control.
Crypto Miner
RCE (Remote Code Execution)
Threat Actor
APT (Advanced Persistent Threat)
Compromise
Vulnerability
Bug
Worm
Remote Access VPN (RAVPN)
XDR (Extended Detection and Response)
SIEM (Security Information and Event Management)
User Entity Behavior Analytics (UEBA)
Path traversal vulnerability
An attacker can leverage path traversal sequences like “../” within a request to a vulnerable endpoint which ultimately allows access to sensitive files like /etc/shadow.
Tombstoning
Post-exploit persistence technique
Volumetric DDoS
MFA bomb
Bombard a user with notifications until they finally accept one.
Use-after-free (UAF)
A use-after-freevulnerability occurs when programmers do not manage dynamic memory allocation and deallocation properly in their programs.
Cold boot attack
A cold boot attack focuses on RAM and the fact that it is readable for a short while after a power cycle.
AGI is the theory and development of computer systems that can act rationally.
ANN (Artificial Neural Network)
Ansible
I would call it an open source orchestrator.
anti-aliasing
When you smooth out color in neighboring pixels.
anycast
apache
A formerly popular open source web server which became bloated with features.
APM (Application Performance Management)
ARIN
ARM
A processor architecture from ARM Corporation, as opposed to, e.g., x86. Raspberry Pis use ARM. I think Androids do as well.
ARP (Address Resolution Protocol)
ASN (Autonomous System Number)
Each AS is assigned an autonomous system number, for use in Border Gateway Protocol routing
ASN.1 (Abstract Syntax Notation One)
A standard interface description language (IDL) for defining data structures that can be serialized and deserialized in a cross-platform way.
ASPA (Autonomous System Provider Authorization)
An add-on to RPKI that allows an ASN to create a record that lists which ASNs can be providers for that ASN. The concepts are “customer” (an ASN) and “providers” (a list of ASNs). This is used to do hop by hop checking of AS paths.
ASR (Aggregation Services Router)
A high-end Interent router offered by Cisco for business customers.
AV (anti-virus)
AWS (Amazon Web Services)
Azure AD
BGP (Border Gateway Protocol)
Blast Radius
One of those dreadully overused terms borrowed from the military that mostly only marketing people like to throw around. It means what you think it might mean.
Blue Team – see Red Team
BOM (Bill of Material)
Boot start
A flag for a driver in Windows that tells it to always start on boot.
broadcast
Browser
BSI (The German Federal Office for Information Security)
BYOD (Bring Your Own Device)
I.e., when employees are permitted to use their personal smartphone to conduct company business.
BYOL (Bring Your Own License)
F5 permits this approach to licensing one of their cloud appliances.
CA (Certificate Authority)
CDN (Content Distribution Network)
CDP (Cisco Discovery Protocol)
This protocol allows devices connected to switch ports to learn what switch and which switch port they are connected to. It is a layer 2 protocol.
CGN (Carrier Grade NAT)
The address space 100.64.0.0/10 is handled specially by ISPs for CGN. RFC 6598
CHAP
Chatbot
A computer program that simulates human conversation with and end user.
CI (Configuration Item)
CI/CD
An ITIL term referring to the object upon which changes are made.
CISA (Cybersecurity and Infrastructure Security Agency)
CISO (Chief Information Security Officer)
CMDB (Configuration Management Database)
CMO (Current Mode of Operations)
CNN (Congruential Neural Network)
Computer Vision
A field of AI that leverages machine learning and neutral networks to enable machines to identify and understand visual information such as images and videos.
CPE (Customer Premise Equipment)
CSR (Certificate Signing Request)
CUPS (Common Unix Printing Systems)
Customer Edge (CE)
CVE
CVEs, or Common Vulnerabilities and Exposures, are a maintained list of vulnerabilities and exploits in computer systems. These exploits can affect anything, from phones to PCs to servers or software. Once a vulnerability is made public, it’s given a name in the format CVE–. There are also scoring systems for CVEs, like the CVSS (Common Vulnerability Scoring System), which assigns a score based on a series of categories, such as how easy the vulnerability is to exploit, whether any prior access or authentication is required, as well as the impact the exploit could have.
CVSS (Common Vulnerability Scoring System)
Part of CVE lingo.
DAST (Dynamic Application Security Testing)
Data at rest
Data in motion
Data Plane
A physical security appliance separates data traffic from its management traffic, which transits the managemenbt plane.
Data Remanence
The residual representation of data that remains even after attempting to erase or initialize RAM.
DDI (DNS, DHCP and IP address management)
Deep Learning
A subset of machine learningthat focus on using deep neural networks with multiple layers to model complex patterns in data.
Deepfake
A manipulated video or other digital representation produced by sophisticated machine-learning techniquies that yield seemingly realistic, but fabricated images and sounds.
DHCP (Dynamic Host Control Protocol)
DLL
DLP (Data Loss Prevention)
DNS (Domain Name System)
DNSSEC (Domain Name System Security Extensions)
Docker
DoH (DNS over HTTPS)
Domain
DRM (Digital Rights Management)
EAP
East-West
Data movement with a data center, I believe, as oppose to North-South.
EBITDA (Earnings Before Interest, Taxes, Depreciation and Amortization)
Hey, an IT person needs to know some business terminology!
Eduroam
Enhanced Factory Reset (EFR)
Entra
From Microsoft. The new name for Azure AD
EU AI Act
EULA (End User Licnese Agreement)
Exact Data Matching (EDM)
FAQ (Frequently Asked Questions)
FEX (Fabric Extender)
FIFO (First in, First Out)
FMO (Future Mode of Operation)
As opposed to CMO.
FN (False Negative)
Forensics
FOSS (Free and Open Source Software)
FP (False Positive)
Fritz!Box
A popular home router in Germany.
GA (General Availability)
GBIC
A type of fiber optic transceiver that converts electric signals to optical signals.
GCP (Google Cloud Provider)
GDPR (General Data Protection Regulation)
An EU directive to achieve data privacy.
Generative AI
AI which can create new human-quality content, including text, images, audio or video.
GMT – see UTC
GRE
GSLB (global Server Load Balancing)
GUI (Graphical User Interface)
HA (High Availability)
Hallucination
When an LLM perceives patterns that are non-existent creating nonsensical or inaccurate outputs.
Hands and Eyes
When you don’t have physical access to a server, you need someone who does to be this for you.
At least in the world of F5 this means IP Intelligence, i.e., the reputation of a given IP address.
IPS (Intrusion Prevention System)
IPSEC
IPv6 (Internet Protocol version 6)
ISO 9001
ISP (Internet Service Provider)
ITIL (IT Infrastructure Library)
Kanban
Agile way of tracking progress on tasks and brief meetings.
Kerning
Adjusting the spacing between letyters in a proportional font.
Kernel mode
Kubernetes
L2TP (Layer 2 Tunneling Protocol)
L3, L4, L7 (Layer 3, Layer 4, Layer 7)
Refers to ISO 7-layer traffic model.
LAMP (Linux Apache MySQL and PHP)
An application stack which gives a server needed software to do “interesting things.”
LaTEX
A markup language based on TEX I used to use to write a scientific paper. I think it gets transformed into a DVI, and then into a postscript file.
LEC (Local Exchange Carrier)
LLD (Low Level Design)
LLD (Low Level Discovery)
A command-line browser for unix systems.
LLDP (Link layer Discovery Protocol)
See also CDP
LACP (Link Access Control Protocol)
Link
Linux
An open source OS similar to Unix.
LLM (Large Langiuage Model)
lynx
A command-line browser for linux systems.
MAC (Media Access Control) Address
Layer 2 address of a device, e.g., fa-2f-36-b4-8c-f5
Machine Learning
A subfield of AI that deals with creating systems that can learn from data and improve their performance without explicit programming.
Management Plane
See Data Plane.
Mandiant
MD5 (Message Digest 5)
MDM (Mobile Device Management)
Management software used to administer smartphones and tablets.
MFA (Multi Factor Authentication)
MITRE ATT&CK
Modbus protocol
MS-CHAPv2
MSS (Maximum Segment Size)
Set by a TCP option in the beginning of the communcation.
MTU (Maximum transmission unit)
Often 1500 bytes.
multicast
Named pipes
I read it’s a Windows thing. huh. Hardly. It’s been on unix systems long before it was a twinkle in the eye of Bill gates. It acts like a pipe (|) except you give it a name in the filesystem and so it is a special file type. It’s used for inter-process communication.
NDA (Non-Disclosure Agreement)
.NET
NGFW (Next Generation FireWall)
Palo Alto Networks describes their firewalls this way.
NGINX
A web server that is usually superioir to apache for most applications.
NLP (Natural Language Processing)
A branch of AI that uses machine learning to enable computers to understand, interpret, and respond to human language.
NOC (Network Operations Center)
North-South
Data movement from/to the data center. Also see East-West.
NSA (National Security Agency)
OAuth bearer token
A security token with the property that any party in possession of the token (a “bearer“) can use the token in any way that any other party in possession of it can.
An online community that produces freely available articles, methodologies, documentation, tools, and technologies in the fields of IoT, system software and web application security.
PAP
Patch
PaaS (Platform as a Service)
PBR (Policy Based Routing)
PDF (Portable Document File)
PDU (Protocol Data Unit)
PEM (Privacy Enhanced Mail)
The format certificates are normally stored in.
PII (Personally Identifiable Information)
PKCS (Public Key Cryptography Standard)
PKI (Public Key Infrastructure)
PLC (programmable logic controller)
PM (Product Manager)
Could also be Project Manager but for me it usually means Product Manager.
PO (Purchase Order)
POC (Point of Contact)
POC (Proof of Concept)
Port Channel
Portable Executable (PE)
POV (Proof of Value)
Private Cloud
Prompt Engineering
The practice of crafting effective prompts that elicit high-quality answers from generative AI tools.
PS (PostScript)
A file type I used to use. It is a vector-oriented language, stack-based, which tells the printer how to move its ink pens around the page. Before there was PDF, there was postscript.
PS (PowerShell)
A versatile scripting language developed by Microsoft and available on all Windows computers.
PTO (Paid Time Off)
Purple Team
Purple teams combine red team and blue team functions. See Red Team.
Python
A popular programming language, not the snake.
QSFP (Quad Small Form factor Pluggable)
A newer kind of SFP.
Rack Unit
RADIUS
RAG (Retrieval Augmented Generation)
A method to train LLMs.
Ray
An open-source unified compute framework used by the likes of OpenAI, Uber, and Amazon which simplifies the scaling of AI and Python workloads, including everything from reinforcement learning and deep learning to tuning and model serving.
RBAC (Role-Based Access Control)
RDP (Remote Desktop Protocol)
Red Team
In a red team/blue team exercise, the red team is made up of offensive security experts who try to attack an organization’s cybersecurity defenses.
Redirect
Remediation
Addressing a security flaw.
Remote Desktop Licensing (RDL) services
Often deployed on Windows severs with Remote Desktop Services deployed.
Retrieval-Augmented Generation (RAG)
Reverse Proxy
A TCP gateway which terminates a tcp connection and maintains a separate tcp connection to a back-end server.
RFC (Request for Comment)
RFI (Requst for Information)
RFO (Reason for Outage)
RFP (Request for Proposal)
RFQ (Request for Quote)
RIPE
RIR (Regional Internet Registry)
RMA (Return Merchandise Authorization)
You hear this a lot when It guys need to get a replacement for failed equipment.
ROA (Route Origin Authorization)
ROCE (Return on Capital Employed)
Hey, an IT person has to know a few business terms!
RPKI (Resource Public key Infrastructure)
Provides a way to connect Internet number resource information to a trust anchor.
RPi (Raspberry Pi)
A popular small, inexpensive server aimed at the educational crowd.
RPZ (Response Policy Zone)
A concept in DNS for either a DNS firewall or way to overwrite DNS responses.
RR (Resource Record)
RSA
Asymmetric encryption standard named after its creators, Ron Rivest, Adi Shamir and Leonard Adleman.
RTFM (Read The “flippin'” Manual)
SaaS (Software as a Service)
SAML
SANS
Private outfit in the US which specializes in information security and cybersecurity training.
Sans-Serif
A font type which does not have the fancy rounded blobs at the tips of the letter, such as Helvetica.
SASE (Secure Access Service Edge)
SCADA
SDWAN (Software defined WAN)
SEO (Search Engine Optimization)
SFP (Small Form factor Pluggable)
A type of optic transceiver that converts electric signals to optical signals.
SGML (Standard Generalized Markup Language)
If you ask the French they proudly point to this as the predeccesor to the more widely known HTML.
SFTP (Secure file Transfer Protocol)
SHA (Secure Hash Algorithm)
SIEM (Security Information and Event Management)
SRE (Site Reliability Engineer)
SMTP (Secure Mail Transfer Protocol)
SNI (Server Name Indication or similar, I think)
When multiple HTTP[S web sites whare a single IP this technology can be used to identify which certificate to send to a requester.
Spoofing
When a source IP address is faked.
SSH
SSL (Secure Socket Layer)
SOC (System on a Chip)
I believe the RPi is described to be this.
SOC (Security Operations Center)
SSL (Secure Sockets Layer)
SSL labs
A Qualys (so you know it has to be good quality) service where you can test a web site’s SSL certificate.
udev rules in Linux are used to manage device nodes in the /dev directory. Those nodes are created and removed every time a user connects or disconnects a device.
UI5
SAP’s UI for HTML 5.
Unit testing
UPS (Uninterruptible Power Supply)
URL
UTC (Universal Time Coordinated)
What used to be called GMT.
VLAN
VM (Virtual Machine)
VMWare
Will Broadcom destroy this company the way they did to Bluecoat/Symantec?
VNC (Virtual Networking Computer)
VNC is a software used to remotely control a computer.
VPC (Virtual Private Cloud)
vPC (Virtual Port Channel)
A virtual port channel (vPC) allows links that are physically connected to two different Cisco FEXes to appear as a single port channel by a third device.
VPN – Virtual Private Network
WAF (Web Application Firewall)
Webhook
Website
Wiki
A less formal and usually more collaborative approach to documentation, the prime example being Wikipedia.
Windows PE or Win PE
A small OS for repairing or restoring Windows systems.
WWW (World Wide Web)
x86
A type of processor architecture. Found in most Windows PCs.
My pipeline job has been running without issue for the last year+. All of a sudden I started to receive this error right after my pip3 install -r requirements.txt:
× This environment is externally managed
Well I never had that problem before. I assumed that ADO sprinkles some magic dust onto the agents in the pool and creates an already virtual environment so why bother requiring further vitrualization?? I don’t know. And that’s how I rationalized how it might have ever worked in the first place if I ever bothered to think about it at all.
But I guess I was living on borrowed time and that house of cards came down hard, probably after the agents were upgraded.
The details
With some small effort I have managed to have the pipeline build up a virtual python environment and install my needed packages into it. Here is the relevant code section in the yaml file.
This by the way is my job to check that all pipelines have run correctly in the last hour:
# As of 2/2024 we need to run pytho in a virtual environment
- script: |
python3 -m venv venv
source ./venv/bin/activate
pip3 install -vvv --timeout 60 -r requirements.txt
python3 check_all_pipelines.py conf_check_all.ini
displayName: 'Run script'
workingDirectory: $(System.DefaultWorkingDirectory)/Pipeline_check
env:
ADO_AUTH: $(ado_auth)
PYTHONPATH: $(System.DefaultWorkingDirectory)/Pipeline_check:$(System.DefaultWorkingDirectory)
The requirements.txt simply contains the line
pymsteams==0.2.2
Conclusion
We have shown how to set up a python virtual environment within your yaml file for an Azure DevOps pipeline. You might need this if you rely on any external packages which are not present in the OS version of python on the agents.
As a creature of habit, I fall for an editor and can never imagine using something else. In the VAX days there was EDT, which if memory serves was replaced by the even better VPU. On Ultrix we ran a pretty nice editor simply called e from Rand Corporation. Then there was the love affair with emacs, and finally for the last 30 years vi.
Well with my latest server, a Debian 12 machine, I was having trouble with insert mode, specifically, inserting text from my Windows clipboard. Never had problems before….
Well for some reason, if you want to insert text from the clipboard, you now use <CTRL.>-<SHIFT>V in command mode. Well, at least on Windows 11 running WSL 2 that seems to work. I now realize that doesn’t work from Windows 10 with WSL. I had better figure this out soon…
Windows 10 running WSL
The terminal type (check the TERM environment variable) is set toxterm-color256. I tried pasting any and all registers which is the standard thing you would do if you use the standard Internet advice. None of it worked for me. I finally realized on my own that – and this harkens back to my old days with the beloved VAX 780 – that if I set the terminal type to vt100 all was good! Seriously. Back in the day we had physical VT100 terminals. Well, before that I think there was a VT52? Then maybe a VT102. VT202 was a big upgrade. Anyway, initially I added the following line to my .bashrc file:
export TERM=vt100
and now I can insert clipboard text the way I always have (mouse right-click) in vi insert mode. This kludge was how we fixed a lot of terminal display issues in the old days. But now I see display from top is messed up! Probably other curses-based apps as well. So two steps forward, one step back. So now what I’ve done is removed that line from .bashrc and put the following lines in my .bash_aliases file:
# DrJ kludge to get vi to work and keep top working
alias top='export TERM=xterm-256color;\top'
alias vi='export TERM=vt100;\vi'
That \top harkens back to an old linux convention where a command preceded by \ invokes a program but ignores defined aliases for that program.
Conclusion
I have offered one possible solution to the can’t insert text from the clipboard problem into my vi: set the TERM environment variable to the old-fashioned vt100. Now I can once again right-click while in insert mode to paste in clipboard text.
This was a very vexing issue for a creature of habit such as me!
References and related
This whole issue came up only when I switched from CentOS 8 to Debian 12 as my back-end server. Believe me, Debian 12 is so superior in so many ways this little setback would never make a material impact in that decision. Here’s the write-up of my upgrade.
I’m thrilled to announce that the long-running blog drjohnstechtalk.com has now been migrated to a modern back-end operating system. drjohnetchtalk.com is, a far as I know, the only quality-written technical resource on the Internet which is not supported by ads. Instead it runs on a pay-it-forward approach, embracing the spirit of the old Internet before it was ruined by big money.
drjohnstechtalk.com has been providing solutions to obscure tech questions since 2011.
The details
I like to run my own server which I can use for other purposes as well. I think that approach used to be more common. Now it’s harder to find others using it. Anyway, my old hosting environment is a CentOS server. I had hoped it would last me up to 10 years! 10 years is about the duration of long-term support for Redhat linux. It’s a real pain to migrate a WordPress blog with lots of history where it is important to preserve the articles and the permalinks. This article documents the nightmare I put myself through to get that up and running. Before that there was a CentOS 6 server. Then in 2022 – only about two years later – I learned that CentOS was dead! IBM had killed it. I’m over-simplifying here somewhat, but not by much.
So my blog sort of limped on on this unsupported system, getting riskier by the day to run as I was missing out on security patches. Then my companyt accidentally included one of my blogs in a security scan and I saw I had some vulnerabilities. So I upgraded WordPress versions and plugin versions. So with up to date software, the stage was set to migrate to a newer OS. Further motivation was provided by the fact that after the WP upgrade, the pages loaded more slowly. And sometimes the site just collapsed and crashed.
I have come to love Debian linux due to my positive experience with running it on Raspberry Pis and a few other places. It tends to run more recent versions of open source software, for instance. So I chose a Debian linux server. Then I forget where I learned this. Perhaps I asked someone at work which web server to use, but the advice was to use nginx, not apache! This was very new to me as I had never run nginx, not that I was in love with apache.
So, anyway, here I am writing this on my shiny new Debian 12 bookworm server which is running an nginx web server! And wow my site loads so much faster now. It’s really striking…
Running WordPress in a subdirectory with nginx
There always has to be a hard part, right? This was really, really hard. I run WP in the subdirectory blog as you can see from any of my URLs. I must have scoured a dozen sites on how to do it, none of which completely worked for me. So I had to do at least some of the heavy lifting and work out a working config on my own.
# mostly taken from https://www.nginx.com/resources/wiki/start/topics/recipes/wordpress/
# but with some important mods
upstream php {
server unix:/var/run/php/php8.2-fpm.sock;
}
server {
listen 443 ssl;
include snippets/self-signed.conf;
server_name drjohnstechtalk.com www.drjohnstechtalk.com;
root /web/drjohns;
index index.php index.html;
access_log /var/log/nginx/drjohns.access.log;
error_log /var/log/nginx/drjohns.error.log;
client_max_body_size 100M;
# the following section prevents wp-admin from infintely redirecting to itself!
location /blog/wp-admin {
root /web/drjohns;
try_files $uri $uri/ /blog/wp-admin/index.php?$args;
}
location /blog {
root /web/drjohns/blog;
try_files $uri $uri/ /blog/index.php?$args;
}
location ~ \.php$ {
#NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
include fastcgi_params;
fastcgi_intercept_errors on;
fastcgi_pass php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}
location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg) {
expires max;
log_not_found off;
}
}
I had to add ths svg file type to ignore, the location directive that matches /blog/wp-admin/. I had to define the upstream label as php and refer to that label in fastcgi_pass. I had to figure out my correct version of fastcgi. I tossed out some location directives which weren’t too important to me.
I disabled the wp-hide-login plugin while I grappled with why I was getting first a 404 not found for /blog/wp-admin/, then later, the too many redirects error. But I still had the issue with it disabled. Once I resolved the problem by adding the /blog/wp-admin location directive – I seem to be the only one on the Internet offering this solution and no other solution worked for me! – then I re-enabled the hide login plugin. The other plugins are working I would say.
I’m on the fence about it, fearing it might slow my speedy server. But it looks pretty good. So for now I am relying on AWS Network Security Group rules. Did you know you can ask them to increase your max rule quota frmo 20 to 40? Yes, you can. I did and got approved overnight. I have added the Cloudflare ranges.
Cloudflare
I continue to use Cloudflare as reverse proxy, certificate issuer, DNS provider and light security screening. The change to the new server did not alter that. But I needed a new config file to properly report the origin IP address in my access files. The following file does the trick for me. It is up to date as of February 2024, can be placed in your /etc/nginx/conf.d directory and called, e.g., cloudlfare.conf.
The idea is that if the source IP of the HTTP connection to nginx is from the Cloudflare range of IPs, then this must represent a request proxied through Cloudflare and the original IP of the client is in the HTTP header CF-Connecting-IP, which nginx can report on. If not, just use the normal IP from the TCP connection.
Swap space
On CentOS I had to provide some swap space because otherwise apache + mariaDB + WordPress would easily send its cpu soaring. So far I have not had to do that with my new Debian 12! That is great… So I have a t2.small instance with 25 GB of gp2 storage (100 iops). The server is basically running with a 0.00 load average now. I don’t get a lot of traffic so I hope that infrastructure will suffice.
Set the timezone
My Debian system started out in the UTC timezone. This command confirms that:
sudo timedatectl
This command brings up a menu and i can change the timezone to US Eastern:
sudo dpkg-reconfigure tzdata
Automate patching
It hasn’t run yet, but I’m hoping this root crontab entry will automate the system updates:
There should be three years of full support plus two more years of long term support for a stable Debian release, if I’ve undrstood it correctly. So I believe I may hope to get five years out of my Bookworm version, give or take. Debian — Debian Releases
Fixing the vi editor
I’ve never really had a problem with vi until this server. I show how I fixed it in this blog post.
Status after a few days – not all positive news
Well after a few days I feel the server response has noticeably slowed. I could not run top because I messed up the terminal with my fix to vi! So in a panic I restarted mariadb which seemed to help performance a lot. I will have to figure out how to monitor for this problem and how best to address it. I’m sure it will return. Here is my monitor.sh script:
#!/bin/bash
# restart mariaDB if home page response becomes greater than one second
curl -m1 -o /dev/null -ksH 'Host:drjohnstechtalk.com' https://localhost/blog/
# if curl didn't have enough time (one sec), its exit status is 28
[ $? -eq 28 ] && (systemctl stop mariadb; sleep 3; systemctl start mariadb; echo mariadb restart at $(date))
I invoke it from root’s crontab every three minutes:
# check that our load time is within reason or else restart mariadb -DrJ 2/24
*/3 * * * * sleep 25;cd /home/admin/hosting; ./monitor.sh >> monitor.log 2>&1
I do love my kludges. I will be on the lookout for a better long-term solution.
Conclusion
The technical blogging web site drjohnstechtalk.com now runs on new infrastructure: Debian 12 running nginx. It is muich faster than before. The migration was moderately painful! I have shared the technical details on how I managed to do it. I hope that, unlike my previous platform of CentOS 8, this platform lasts me for the next 10 years!
