Author: john

Categories
DNS Network Technologies

The IT Detective Agency: The mystery of the bad TE DNS server test

Intro

One of my colleagues, someone who does not moonlight as a DNS team member unlike myself, set up what shuold be a simple DNS test in ThousandEyes Network and App Synthetics. This was in response to a customer complaint that resolution of a particular FQDN occasionally failed. Well, the test results didn’t look right. They indicated very spotty resolution of the FQDN.

So often I’ve come to take TE results with a grain of salt, as though you need an asterisk behind each result so you self-train to ignore most of it and look for mainly secular trends in the hopes they might be meaningful.

The details

This picture sums it up.

A DNS test in ThousandEyes

It’s not looking to good, right? But come on, this is DNS – about as simple s it gets. Either it’s working or something is seriously wrong.

Well, I have access to both ends of the conversation so I do what any network engineer would do and I start to take packet traces on both the source – agent in TE-speak – and destination – one of the DNS servers being tested and showing an error.

For the record, the particular error is

RCODE:0 – No resource records of type A found

Funny thing. After I started taking my traces, I noticed the communication began to work in TE. So I switch to another broken DNS server and set up and then start the traces. And then it too starts working in TE. Finally, I run out of broken DNS servers and all of them are now working! What the…

A little misdirection

So I opened a support case mentioning this strange phenomenon. You never know with support. But TE has made it exceedingly easy to create cases – you never have to leave the app which is super convenient. I marked it with the lowest possible priority since it was mostly a matter of curiosity at this point.

However within a few hours I got a response. Those tests weren’t sending recursive queries. You can turn on recursive queries in the Advanced section.

So what was going on? I fooled you as I fooled myself with a little misdirection and I failed to tell you all the relevant facts. But how can you when you don’t know them yourself? In fact I threw in some additional red herrings into the ticket. I stated that this FQDN is an internal domain name and not resolveable on the Internet. I felt that that might have something to do with it. Wrong. Whenever I started those traces I also ran a dig by hand to make sure that by hand the dns server really could resolve that domain name. And it always could. Then I would fire up the packet traces. Well, dig by default has recursive query enabled by default. In fact, and I’ve never used this until today, you can only turn it off by adding the +norecurse switch! Who knew?

So I was filling that DNS server’s cache with the answer by sending a single recursive query, and it had a fairly long, TTL, about six hours. But then it would revert back to the original bad behavior.

So in that picture above, at the end you see availability has climbed to 100% in the last measurement all the way to the right? I changed the test to do a recursive query and all is good now.

Case: closed.

Conclusion

Although I still feel ThousandEyes produces many test results with artifacts which have to be discounted, at least with regards to a simple DNS resolution test, it probably can be trusted as long as you know what you are doing and in the advanced properties of the test, check the Send recursive queries when you are testing on a recursive server.

References and related

More info concerning ThousandEyes: https://www.thousandeyes.com/

Categories
Admin DNS Firewall

The IT detective agency: the mystery of the quadrupled packets

Intro

This is an active investigation and I cannot stamp case closed at the end as is my wont!

I have access to both source and destination servers. They are virtualy DNS servers running linux and BIND. pretty standard stuff.

We had been measuring response times of DNS queries and were all too often getting high values – like 3000 ms high! But not always. Also 50 ms.

And there were these ICMP messages to this effect:

ICMP 10.13.24.21 udp port 51343 unreachable, length 233

Fortunately I had the ability to do a packet tarce at both source and destination servers – that is a rare luxury these days in the age of granular access control.

Well, what I was seeing was the source DNS server sending a simple UDP query to the destination DNS server, just one UDP packet while the destination server saw that same packet four times! But not always. Slometimes it only saw one packet. Sometimes two packets. So it is random but occurring so often a trace for a few seconds shows the problem.

To be continued…

Categories
Network Technologies

The IT Detective Agency: the mystery of the F5 BigIP restart

Intro

Unlike your home PC, critical security servers basically never just restart on their own. But today I got a Zabbix notification that indeed one of ours had done just that. I set out to investigate to learn the root cause and prevent it from recurring. things took a weird turn at some point.

The details

I asked my colleague to investigate the server, He dutifully looked at various logs. There didn’t seem to be anything amiss according to the logs. Then he showed me a screenshot of the CLI prompt. It had been up for 497 days and some hours. So there was no restart.

We speculated that maybe just the ASM module had restarted, but without any real evidence.

System uptime is an SNMP MIB which we monitor in Zabbix.

The graph of this uptime looks like this:

What is the Zabbix trigger expression for this problem? it is:

change(/f5host/snmp.agent[F5-BIGIP-SYSTEM-MIB::sysSystemUptime.0])<0

In words, if the uptime, a linearly increasing value over time, becomes suddenly smaller, then the change from the previous value is negative and it is assumed that signifies a restart.

Then I remembered another way to satisfy that condition. Consider this python math:

>>> 86400*497
42940800
>>> 2**32
4294967296

The first few digits of those two different calculations agree, right? Let me back up. A day has 3600*24 = 86400 seconds. What I’m getting at is that if you assume the uptime units in this MIB are hundredths of a second, then the two numbers become basically the same.

So what happened is that this counter filled up after 497 days and rolled over! So there was no restart after all.

Conclusion

A Zabbix-reported F5 restart on an F5 BIG-IP which had been up for more than 497 days was shown to be a false positive and explained as an artifact of the finite size of the SNMP counter, which rolled over to 0.

Case closed!

Categories
Admin

An IT Nightmare

Intro

This is an actual dream, or more like a nightamre, that I had recently. I guess it’s illustrative of an IT person’s worst fears.

The dream

Well I don’t remember dreams very well and I’m not the kind ot embellish stories to make them sound more interesting so this is going to be brief.

So in this dream I am at the office. My work situation seems to be that I have slightly more access than I need to various systems. The workpllace is a lagre corporate office where processes are followed but still individual contributors want to make a difference so there is some self-imposed pressure to do something of value.

So anyway I find myself in this dream needing to make a configuration change to a monitoring system. Something like a Zabbix implementation. Now I know that I am not chiefly responsible for it, and in fact I should not be modifying it, but I have some idea that what I plan to do will make it better in some way. It’s in fairly widespread usage – about 150 users.

While doing this improvement it asks me for a new administrator password. But it wasn’t exactly that. It changed the administrator password, displayed the new one, and suggested I copy it and save it, which I did. This dialog only displayed for about 10 seconds and that screen went away.

Actually I hadn’t quite had time to save that new admin password, I just had it in my clipboard. I went to notepad++ to paste it and preserve it. There was nothing in my clipboard! That deep, sinking feeling set in. Even if it continues to work, we won’t be able to do patches so it is as good as killed, it will just take a little longer.

I guess we’ve all been there, right?

Inspired by

IRL I was purchasing tickets for Shen Yun. I selected seats and was at the part where you enter credit card info. My Edge browser proposed to enter a generated credit card number, which I generally approve of as an anti-fraud measure, so I let it fill in the info. It needed biometric authentication – face. The next thing I knew the whole browser screen vanished. Edge running on a completely patched Windows 11 PC simply crashed without a word. I think it bears mentioning because unlike the bad old days where crashes were customary, these days it’s not such a quotidien occurence. And when I restarted things, there was no memory of my seat selection but they were blocked from being purchaseable. Kind of a worst case scenario there. I didn’t wish to wait for the 20 minute purchase timer to timeout as few seats were available. Fortunately there were comparable seats in another row. Second time through it did not propose to fill in with a random card and things went through.

Categories
3D Printing

Trying to develop a 3d object with the help of chatgpt

Intro

I develop simple 3d objects using an approach which could be called 3d-object-as-code. The language is Openscad, and my few objects are all documented here: 3d printing some parts for the house. But that was started long before generative AI took off. So it was incumbent on me to explore what assistance I could get from use of chatgpt.

The details

I wanted an object which would push against our utensil holder to pin it in place within the kitchen drawer, and at the same time make that space available for additional kitchen gadget junk. The final picture is further down below. I started having chatgpt generate the base based on my desired dimensions. So far so good. Then I had it re-do it using less material by making the base a lattice. Things already begin to fall apart. It dropped two of the ends though it did create a lattice pattern.

Now mind you no serious developer would proceed as I have done. I get the code from chatgpt, and paste it into the openscad app to render it to see what it created. Very indirect and inefficient! But this is just for me, so why not…

So anyway, I add the missing side by hand.

A time saver?

Yes! At this point chatgpt has gotten me started. on my own, I get psyched out by decisions as to whether or not to created centered versions of cubes, etc, and my issue when doing it by hand is getting lost in translation, literally. I often find myself three translations deep, and it gets to be overwhelming to think it through. chatgpt unilaterally decided the initial cube would not be translated, so I went with that simpler approach, and that helped.

I think I did manage to get chatgpt to add the legs as well, so that was a help.

chatgpt was good at creating modular and therefore reuseable code which even had nice comments! It’s like looking at your colleague’s code who writes better code than you and picking up a few pointers.

But not after awhile

I realized I needed to stabilize those legs. But what words to use in the prompt? I’m not an engineer. I said something to this effect

Starting from this openscad code, add a small cross arm having the same thickness as the leg in order to anchor the leg more firmly (openscad code...)

The result was laughable as it produced a horizontal piece attached to the bottom of the leg on the one end and attached to nothing at all on the other!

Second attempt: 

Starting from this openscad code, add a small brace having the same thickness as the leg in order to anchor the leg more firmly. It should begin at (0,0,20) and end at (20,0,15) (openscad code...)

Still, it ignored these very direct start and end directives. I tried once more with no better results.

I also tried an approach requesting to add a bracing triangle of material to help stabilize the leg, but it laughably added an extruded triangle along the whole length of the leg!

At this point clearly the ai was not acting like an assistant, but a text language generator. It had clearly zero idea what it was doing.

So at that point, it was time negative exercise, useful only for this blog post and to make me humbly admit I do not know how to get the most out of chatgpt.

Finally

I had to add those bracing bits by hand-coding that part. That involved a rotation, a translation and a difference. It could have been worse.

The final product
The code
Copy Code 


// DrJ 1/2025. Parameters in mm

// Dimensions of the box

width = 110;

length = 150;

thickness = 3;

epsilon = 1;

brace_angle = 30;

brace_z = 20;

spacing = 53; // Spacing between the lines in the criss-cross pattern

line_thickness = 4; // Thickness of the lines in the pattern

leg_height = 53; // Height of the legs

leg_width = 6; // Width of the legs

leg_length = 10; // Length of the legs

//width = width – line_thickness; // correction

side_height = thickness; // Height of the side leg

side_width = width; // Width of the side leg

side_length = 10; // Length of the side leg

module leg() {

    // Create a leg

    cube([leg_width, leg_length, leg_height]);

}

module shave_cube(){

    translate([0,-epsilon,thickness]){cube([side_width,side_length+2*epsilon,leg_height]);};

}

module leg_brace() {

    // Create a leg brace

    difference(){

    translate([0,0,-brace_z]){rotate(a=[0,brace_angle,0]){cube([leg_width, leg_length, leg_height]);}};

    shave_cube();

    }

}

module side_leg() {

    // Create a leg

    cube([side_width, side_length, side_height]);

}

module criss_cross_pattern() {

    for (i = [0 : spacing : length]) {

        // Horizontal lines

        translate([0, i, 0]) {

            cube([width, line_thickness, thickness]);

        }

    }

    for (i = [0 : spacing : width]) {

        // Vertical lines

        translate([i, 0, 0]) {

            cube([line_thickness, length, thickness]);

        }

    }

}


// Create the criss-cross box

criss_cross_pattern();

// Add legs at two corners

translate([0, 0, -(leg_height-thickness)]) {

    leg(); // Leg at the bottom-left corner

}

translate([0, length – leg_length, -(leg_height-thickness)]) {

    leg(); // Leg at the bottom-left corner

}

// add stronger sides

translate([0, 0, 0]) {

    side_leg(); // Leg at the bottom-left corner

}

translate([0, length – side_length, 0]) {

    side_leg(); // Leg at the bottom-left corner

}

leg_brace();

translate([0,length – side_length,0]){

    leg_brace();}

//shave_cube();
Conclusion

I got not-so-great results in my attempt to use the chatgpt o4 generative ai offered by Duckduckgo. The basic stuff, yes, it got me started and taught me how to make good modular openscad code. Anything remotely complex, forget about it. You want to treat ai like an assistant, right, but this assistant has near zero understanding of what I want and did not learn even after multiple attempts within the same chat session. It should be put out to pasture…

However, I am always willing to take the fall. I was just going by the seat of my pants with regards to prompt engineering. Maybe if I had chosen better prompts, or let ai have freer reign to do the whole design I would have experienced better results. But shouldn’t my “assistant” be better at understanding me?

References and related

All about Openscad: https://openscad.org/

The original post: 3d printing some parts for the house.

You can use chatgpt directly: chat.com or the way I used it, from the menu of https://duckduckgo.com/

The Etsy shop of the person who printed this for me for only $5: https://www.etsy.com/shop/gizmoswidgets

Categories
Cloud

Azure DevOps pipeline run limits

Intro

We wished to run a pipeline every five minutes, but when you do the math, this will result in its running more than 1000 times per week, which according to the documentation is forbidden. On the other hand, we are using private agents – our own – so why should Micrsoft put limits on how often we run jobs on them??

The details

Given that there are 10080 minutes in a week, to arrive at fewer than 1000 pipelkine runs per week you’d need to pace out your jobs at no more than run once every 12 minutes. And that’s what I had been doing. So then I would create a second pipelikne running the same code, but running it an the inbetween times to end up with a logical job which runs every six minutes. But is this approach really required for our private agents?

We decided to put this to a real test. I created a Hello World yaml file and ran it every minute. The results are not at all what we expected!

The results

Essentially, the job runs 10 times out of every 15 minutes. This is another published limit. And you see this effect right away. So this is like some kind of burst rate limiting you might say, and it applies.

And during those times when it’s not being run, you don’t see it paused or anything. It simply isn’t run. But you can run it by hand (I think) and it will run.

So then you think, OK, limits apply, even to private agent pools. Then we left it running, and something funny happened.

After about 640 runs in the course of 24 hours, it simply stopped. Then about three days later it started up again, ran about 637 times, then stopped again.

So there seems to be an additional unpublished limit of something like 640 runs in a 72 hour interval.

But, we were able to exceed 1000 runs in a week, for what it’s worth.

Then I let the job run awhile. It seemed to want to run 635 times on Mondays, then stop the whole rest of the week. IDK…

Alternatives

I guess we were not using pipelines for what it was intended. It’s not really to be considered cron on steroids. We’ll be looking at Azure Functions to see if it’s a better fit for our requirements.

Conclusion

Treating pipelines like “cron on steroids” is not what it was designed for. Even when you use your own agents in your pool, your Azure Pipeline job will be rate limited to about 10 runs per 15 minutes, about 640 runs per three day interval (unpublished limit), though you can exceed 1000 runs per week. These limits prevent you from executing a run every five minutes! If you need to execute a job so often, consider finding a different approach!

References and related

How a pipeline can modify its own repository.

Categories
Consumer Tech

The IT Detective Agency: The case of the iPhone mystery alarm

Intro

My wife asked my assistance to find the source of the daily alarm which was nagging her at 6:20 AM every morning. I don’t use an iPhone so I was pretty clueless myself.

The details

Of course she had done the obvious things like look at the clock for set alarms. And at installed apps for alarms. Nothing.

Yet every day – unless the iPhone was turned completely off – this alarm would go off at 6:20 AM. And her Apple iWatch, or whatever it’s called, also had some message about this alarm.

We searched all installed apps for “alarm” and “clock” but there was nothing left to look at. Maybe one of her health apps? Nope. doesn’t seem to be. Maybe the Army Knife app with all its little useful gadgets? Nope, no alarm clock there.

The breakthrough

Then I got an idea. Since the wake-up screen mentioned domething about sleep, I decdied to search the phone for sleep. And voila, there is a sleep app, or at least sleep settings. And it was set to end her sleep at 6:20 AM.

So you see the misdirection at work? We kept thinking in terms of clock and alarm. But Apple just thinks of it as sleep and calls it as such.

Case: closed

Conclusion

Two people were frustrated for days trying to find the source of an iPhone alarm, which eventually was found. Beware that there is a sleep app. We followed the leads on the Internet about turning off certain notifications, which led nowhere.

Categories
Network Technologies

The IT detective agency: the case of the failed dhcp request

Intro

In full disclosure this case was not one I contributed to in any way, unlike all the others I’ve reported on. Nevertheless, source who did work on this case told me sufficient details and it is an interesting case.

The setup

For this case to make any sense, you need to understand the background. If I got it right, some people were trying to restore a backup version of Windows 11 Professional. When they did this restore, they found the problem that they were not pciking up an IP address via dhcp if they were on a company network. If they did the restore while on a home office network it went OK.

So imagine the comlpexity in a modern IT environment this presents. You have the PV vendor, HP, the OS vendor, Microsoft, the dhcp service operator, in-house, the LAN service provider and the network gear vendor, Cisco. The fault could lie anywhere. They all initially claim their stuff is working fine (which is always the default statement) and look elsewhere.

So what I like to say is that any hypothesis is unlikely, yet one of them will prove to be correct, eventually.

More details

Packet traces showed the DHCP Discover request being sent by the PC, but not arriving to the DHCP server. Ah, you say, simple: the switch is guilty here of dropping the DHCP Discover packet, fix it. After all, “eating” dhcp packets is something misconfigured switches do all the time if dhcp snooping is misconfigured.

Yet the LAN service provider says the switch isn’t misconfigured. So they have to open a case with the switch vendor to understand the drop. I’m not sure where that support case went, meanwhile…

The in-house expert troubleshooters were able to take a second trace from a PC which did pick up an IP address after a restore. This restore feature of course used to work when it was initially released.

Categories
Linux Python

How to prevent an image from being rotated

Intro

I still use my home-grown slideshow software based on Raspberry Pi, which is quite a testament to its robustness as it has been running with only minor modifications for many years now. one recent improvement has been my addition of being able to handle photos from recent iPhones which save photos in the new-to-me HEIC format. My original implementation only handles JPEGs and PNG file types, so it was skipping all our recent iPhone photos.

I figured there just had to be a converter our there which would even work on the RPi, which of course there was, heif-convert. But it has an oddity when it comes to rotation. It converts the HEIC to a jpeg, fine, but it rotates them, but it also leaves all the EXIF meta data, including the orientation meta data, as is. This in turn means display software such as fbi may try to rotate the picture a second time. Or at least that’s what happened to my software where one of my steps is an explicit rotate. That step was creating a double rotation.

So I needed a tiny program which left all the EXIF meta data alone except the rotation, which it sets to 0, i.e., do not rotate. Seeing nothing out there, I developed my own.

The details

Here is that script, which I call 0orientation.py:

Copy Code 


#!/usr/bin/python3
# DrJ 10/24
# you need the exif package: pip install exif
# https://github.com/kennethleungty/Image-Metadata-Exif?tab=readme-ov-file
#https://exif.readthedocs.io/en/latest/
import sys
from exif import Image
file = sys.argv[1]
with open(file, 'rb') as image_file:
    my_image = Image(image_file)
exif_flag = my_image.has_exif
print(exif_flag)
exifs = my_image.list_all()
print(exifs)
my_image.orientation = 0
with open('modified_image.jpg', 'wb') as new_image_file:
    new_image_file.write(my_image.get_file())
Conclusion

I have shared how to overwrite just the orientation tag frmo a JPEG photo.

Reference and related

My photo frame software based on photos stored in a Google Drive

Categories
Home Computing Uncategorized Web Site Technologies

How to use your phone as an ersatz workstation with equipment lying around your house

Intro

While my laptop was being shipped to me I wanted to be as productive as possible using my Samsung Galaxy A35. I was vaguely aware of the availability of Microsoft 365 apps such as Outlook. How far could I take this…?

The recipe

To cut to the chase, I was maybe 60 – 70 % effective. I used equipment found in the typical IT person’s home plus one inexpensive purchase from Walmart.

Here is what I used:

  • HDMI monitor
  • old Amazon firestick
  • cheap bluetooth keyboard purchased from Walmart
  • phone stand

And here’s what I really wished I had but did not:

  • bluetooth mouse

Which apps worked well:

  • Outlook
  • Teams, especiallt chat, less so the meetings function
  • One Note
  • Edge
  • VPN client

I must say the bluetooth keyboard worked really well for doing some serious typing up of emails.

How the external monitor worked

So I “came up” (in quotes because I’m sure many others figured out this same thin) with the idea of casting my phone screen onto an external monitor by way of the screen mirroring capacibility available on even the oldest amazon Firestick. On the phone you simply go to Smart View Mirror Screen.

So that prevented me from having to hold the phone at least while I was drafting emails.

But, and it’s a big one, is that the external monitor was not a TV and the sound from meetings was killed by this setup. And I did not see a way to keep audio local to the phone while only casting the screen.

A smaller problem is that the refresh lag is quite noticeable under conditions of rapid screen refresh. So it may take a second or two to show what the phone’s screen shows.

Still, it’s pretty cool.

I would have bought a bluetooth mouse but it simply wasn’t available at my local Walmart. I was pretty inconvenienced without it having to constantly touch the phone screen for various things.

And the external keyboard

Pretty well. Even some shortcuts worked. Alt-TAB, which I use a lot to switch between apps has some kind of vaguely similar effect on the phone, but not to the point where I could rely on it usefully. The unlock shortcut button sort of woke the up the phone screen at least.

TAB helped me to pop from one field in the form to the next the way I would use it on a PC.

Overall responseiveness was satisfactory.

The small form factor was not a detriment, and maybe even an advantage since it’s so light and portable.

What if you have an HP G5 docking station lying around?

Well I do. It has a USB-C cord which you normally plug into your HP laptop. But I didn’t have the power supply for it so I couldn’t use it when I would have needed it. Well, it basically works with a Samsung phone – at least the keyboard and mouse worked. In my 10 second testing the attached HDMI display did not automatically show anything. Maybe there are some phone settings which would need to be changed. I didn’t mess with it at all.

But it’s cool seeing a mouse working. It suddenly paints a mouse pointer on your phone screen which you can move around and click to launch an app.

Apps are often baby implmentations

At first I struggeled with the Outlook app, trying to use it as though it were my full-blown Outlook client on my PC. It only had one week’s worth of messages, which was pretty limiting since I was out for more than a week. Then I had a lightbulb moment and remembered that the Web version of Outlook worked on my phone. So I switched to using Outlook through the Edge browser – much better for me. That’s https://outlook.office.com/ . I could get full history and therefore do more reliable searching through messages.

Responsive Design work-around

Sometimes the mobile app version of a web site just doesn’t have the featuires, but looks nice. Edge has a feature you can choose called View Desktop Site which gives you the “real” web site. Now it may look tiny, forcing you to expand and shrink with two fingers. But at least it will generally work.

Where is Notepad or Notepad++

I didn’t look for an app. I suppose there is one. Somtimes you just want to inspect your clipboard. I settled on pasting into a new draft Outlook email to do my visual inspection of my clipboard.

References and related

I prepared the above solution with one day’s notice. If you had a couple days you might check out the Samsung Dex. I guess it would work for modern Samsung Galaxy phones though I haven’t tried it myself.

The web version of business Outlook, which is a pretty good implementtion of the full-blown client is https://outlook.office.com/