Author: john

Categories
Cloud

Azure DevOps: pipeline tips

Intro

I’ve made a lot of mistakes and learned something from every one. I am trynig to pass on some of what I learned.

Limit of 1000 pipeline runs per week

This seems pretty crazy. In my unit we were going all gung-ho and imagining Azure DevOps pipelines as an elegant replacement for running cron jobs on our own linux servers. But something as simple as “I need to run this job every five minutes” seems to be overwhelming for a pipeline. What? Yes there is a hard limit of 1000 pipeline jobs a week. This limit is discussed here.

I wanted a job to run every six minutes, which will still hit that limit. So what I am trynig is to create two pipelines. Each is scheduled to run every 12 minutes. The yaml files are almost the same except in the one I sleep for six minutes. I also needed to remember to re-create the pipeline variables I was using.

Getting the raw logs from your pipeline runs

I wrote a little script to get the raw logs and copy them to a linux filesystem where I can use the linux command-line tools I know and love to examine them in bulk. The main point I wish to share right now is that it is not at all obvious that you need to use the get builds section of the api, not the log section! Who would have guessed? https://learn.microsoft.com/en-us/rest/api/azure/devops/build/builds/get-build-log?view=azure-devops-rest-7.1

Errors I am seeing in my pipeline

[error] We stopped hearing from agent dsc-adosonar-drjohns4servicescoresystems-agent-549c476959-whd72. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error. For more information, see: https://go.microsoft.com/fwlink/?linkid=846610

We still need to figure this one out. The error appears only randomly.

I also saw a lot of more subtle errors which amounted to my variables not being defined correctly in the yaml section. Indentation is important! I had variables set up secret environment variables amongst other things. The behavior which results does not always make it obvious what the root cause is.

Don’t run the pipeline for every commit

In your commit comment, put

[skip ci]

somewhere on its own line. This will avoid that the pipeline runs each time you do a commit, which quickly gets annoying.

References and related

How to use the Azure DevOps api to for instance fetch the raw logs

Microsoft’s api documentation pertinent to this topic:

https://learn.microsoft.com/en-us/rest/api/azure/devops/build/builds/get-build-log?view=azure-devops-rest-7.1

Categories
Cloud Python

Azure DevOps: How to work in a subfolder of a project

Intro

Our repo corresponds to a project. Within it are subfolders corresponding to individual apps we want to run in a pipeline.

Someone provided me with a starter yaml file to run my code in a pipeline. Originally my code was running just fine on my own linux server. In the pipeline, not so much as it became apparent it was expecting the current working directory to be the subfolder (directory in linux-speak) for references to modules, config files, etc. The documentation is kind of thin. So I spent hours checking things out and creating the solution which I now present.

The solution

The critical thing is to set the workingDirectory. Here is that section of the yaml file.

 script: python getlogdata.py 'PAN4-5min.aql'
  displayName: 'Run script'
  workingDirectory: $(System.DefaultWorkingDirectory)/PAN_Usage_4_Mgrs
  env:
    AUTH_TOKEN: $(auth_token)
#    PYTHONPATH: $(System.DefaultWorkingDirectory)/PAN_Usage_4_Mgrs/modules

Note that that PYTHONPATH environment variable is another possible way out – if all you need is to include modules, but it won’t help you with other things like finding your config file.

Errors

Now suppose you see an error like I got:

ImportError: cannot import name 'ZabbixMetric' from 'jhpyzabbix' (unknown location).

I had tried to put jhpyzabbix folder at the same level as my subfolder, so, right under the top project level. At first I was getting module not found errors. So I put back my PYTHONPATH like so

    PYTHONPATH: $(System.DefaultWorkingDirectory)/PAN_Usage_4_Mgrs:$(System.DefaultWorkingDirectory)

And that’s when I got that cannot import name error. Whar caused that is that although I had copied over the needed .py files to jhpyzabbix, I forgot one whose purpose seemed irrelevant to me. __init__.py. Turns out that tiny python file is quite important after all. School of hard knocks… It sets up the namespace mapping, I guess. To be concrete, mine looks like this:

from .api import ZabbixAPI, ZabbixAPIException, ssl_context_compat
from .sender import ZabbixMetric, ZabbixSender, ZabbixResponse
References and related

Passing secure variable in Azure DevOps to your program

Categories
Cloud

Securing a sensitive variable in Azure DevOps

Intro

I’m a newbie at Azure DevOps. They sort of have to drag me kicking and screaming to give up the comfort of my personal linux to endure the torture of learning something brand new. But I iguess that’s the new way so I gotta go along with it. I work within a project that has a repo. And we intend to run a pipeline. Question is, how to deal with sensitive parameters like passwords?

One approach

If you just do raw Internet searches you can fall into any number of rabbit holes. Key vault, anyone? How about relying on secure files within the library? I guess – very tentatively – that using a key vault might be the right way to deal with this issue, but it all depends on the ACLs available and I do not know that.. I also do not see a simple-minded way to set up a key vault. So what the guy who seems to know a lot more than I do is to set up a hidden variable for the pipeline itself.

The thing is that even that has its own gotchas. I find that depending on where you start from, you may ior may not see the option to declare a pipline variable as hidden.

If I am in the Pipeline section and looking at Recently Run Pipelines, and click on my pipeline, before I run it I can add variables. Doing it that way, you only get the option to include a name and Value. No option for declaring it to be hidden.

So how do you do it?

Instead of Run Pipline > variables go to Edit Pipeline > Variables. Then variables you create will have the option to be kept secret.

It is a bad idea to pass a sensitive information on the command line. Anyone with access to that agent could do a process listing to read the secret. So instead you use an environment variable. It’s a little tricky as you have to understand yaml variable interpolation versus script interpolation.

My secret variable is auth_token for the Zabbix api token. So in my yaml file I have a reference to it which sets up an environment variable:

- script: python Delete_Unused_VEdges/activity_check.py 'SA'
  displayName: 'Run script'
  env:
    AUTH_TOKEN: $(auth_token)

And in activity_check.py I read that environment variable like this:

token_zabbix = os.environ['AUTH_TOKEN']

And, voila, it’s all good.

But how secure is it really?

Question is, is this just performative security? Because I’m working with a project. Anyone else with access to the project could alter my python program temporarily to print out the value of the secret variable. I’m assuming they can both alter the repo I use as well as run a pipeline. Yes, they make it a tiny bit challenging because you need to break up a secret variable into two pieces in order to print it out, but come on. So maybe we’ve mostly achieved security by obscurity.

But again, I don’t really know the richness of this environment and perhaps more can be done to restrict who can alter the repo and run it in a pipeline.

Conclusion

People who know more than I, a newbie, have sugested a certain approach to dealing with sensitive variables with our Azure DevOps pipelines. I have described here what I as a newbie see as potential pitfalls as well as how to implement their chosen method.

Categories
CentOS Web Site Technologies

CentOS is dead

Intro

As my devoted followers will be aware, I nearly kiled myself converting my Centos 6 VM to CentOS 8. For instance see Upgrading WordPress brings a thicket of problems. That is an experience I only want to go through every 10 years, and fortunately, CentOS was just the right platform as its support was supposed to last 10 years I started this blog in either 2011 I believe. I went to CentOS 8 in 2020.

But instead of eight more good years, I’ve learned that CentOS is basically a dead product. EOL in industry parlance. IBM killed it. The last upgrades to CentOS 8 came at the end of 2021. There is a sort of CentOS, now called CentOS Stream, but it should be basically thought of a another Fedora. Probably IBM was losing too much money with people choosing CentOS (free) over RedHat paid subscription.

But anyway, I’ve come to resent how out-of-date the packages are on CentOS and I am much more favorable to plain old Debian linux, largely due to my work on the Raspberry Pi. There the packages like python are much more uptodate. I guess the support is for five years.

The other VM I would consider for my next iteration is Amazon Linux. It has a lot of what I need already installed, so less fuss. But I think they’re only supported for three years.

Rocky Linux, the CentOS replacement

After three years I finally hard about the best CentOS replacement. Rocky Linux. I guess it’s still a bit obscure, but you can find it as an AMI on the marketplace. It has no cost and its stated aim is to be bug-for-bug compatible with Redhat! See the Wikipedia article.

References and related

There is a snarky commentary about this topic which inspired this article. I don’t have the link right now but it is enlightening. I will post it if I ever find it.

Upgrading WordPress brings a thicket of problems

Rocky Linux the CentOS successor on Wikipedia

Categories
Linux

libavddvice.so.58 not found running ffprobe

Intro

I use ffmpeg regularly on my RPi 4. I’ve written several articles. For no apparent reason, today ffprobe, which I use to get the length or a recording, stopped working for no apparent reason. The error was something like

libavddvice.so.58 not found

I hadn’t done any upgrades or anything at all, I think.

The solution

$ sudo apt-get install libavdevice-dev

Even though it looked like it was already present as shown by a dpkg -l, running this command installed a bunch of related libraries and seemed to fix things up just fine.

ffmpeg version 4.3.4-0

Raspbian 11 bullseye

References and relate

https://drjohnstechtalk.com/blog/2021/10/from-audio-recording-to-youtube-with-two-button-clicks-and-a-raspberry-pi/

Categories
Consumer Tech Web Site Technologies

Starlink Internet service: a first look: UPDATED

Intro

Many of us were quite enthusiastically awaiting the availability of SpaceX’s Starlink Internet service. On paper it sounded promising. the first results came in and the reality was far less impressive, but the update I got yesterday (July 2023) is that the service got better and better.

July 2023 Update

I guess they continued to add more satellites making the coverage better and better. When there is an outage it is only for a second – short enough for even real-time media to easily recover.

The original post, written when the service was newer, is below.

The details

I do not have this service but spoke with someone who does. He lives in Puerto Rico where the broadband option are limited. There’s the local cable company, then maybe some boutique services where you use microwave dishes, and this year, finally, Starlink. He had just a couple users on it. I think the net results are that it basically works, but with a big caveat. It sucks for real-time communication. And that’s precisely what he needed it for.

So you know when you’re streaming a movie, that downloads the movie in six(?) second chunks, so it’s a bit robust in the face of brief outages. But when you’re doing web conferencing an outage is very noticeable. And that’s what they experienced, time and again. Brief outages that interrupted their real-time applications. Perhaps lasting for a few seconds, but enough to spoil the broadcast.

Then one night, knowing their cable provider, Liberty, was out, they tested it again. It seemed fine at night. But during the day next day it failed in the same way – brief, disruptive outages.

Maybe some of it is due to holes in the satellite coverage and will get better as the fleet fills out. We’re not sure at this point.

And, yes, the dish was placed in a place where the app showed something like 98% visibility to the satellites in the sky.

Some interesting screenshots of what a Starlink IP looks like in Puerto Rico
ipinfo.io
ipchicken.com
speedtest results

That speedtest looks quite good to me!

Results of Starlink app for this actual user in Puerto Rico
Running PING to google.com shows a single dropped packet
A few words from the actual user

Liberty Cable is not working even after a change of the cable modem. So this past two weeks his household has been exclusively using Starlink. In his own words:

“The single most important thing to consider when using Starlink is how obstructed your northern facing view of the sky is. I am attaching a screenshot from my Starlink app. The red shows the obstructed area. My placement is 2.5% obstructed but I still get an interruption every 4 minutes the app says. In reality it might be every 20 minutes for a few seconds. 

“While my Liberty service has been out Starlink has been a life saver for us. A second user can still do her video calls but it will freeze during those 20-ish minute intervals for a few seconds. It’s not the end of the world for her but not totally idea.


“For me, my VPN will disconnect for those same few seconds and then will reconnect. If I’m entering a trade that can be a crucial few seconds while my vpn and software reconnects, but it’s workable. 


“The Starlink app is free and available for everyone to use. I would suggest that anyone who is interested in the service to first download the app and scan the sky where they think they have the freest point of view north. They will only get purely uninterrupted service if the app registers 100% obstacle-free view. The properties that are most suited for Starlink are the ones at the top of a hill, with a field, or a roof taller than the surrounding trees, especially the trees to the north. An obstructed view like mine is perfectly suitable for streaming movies as they tend to buffer a few minutes in advance, downloading files, and surfing the web. Without a completely obstructed view of the sky, video calls, VPN connection, Remote desktop connection, and online gaming will be interrupted in a frustrating manner. 


“I am also attaching a text file of my results from running a  ping -t to google.com from my Starlink connection. This test ran for about 25 minutes. The request time outs are the times when the Starlink satellite was not able to connect due to my obstructions. However, also notice that during these times it only lost 1 ping and was immediately able to reconnect. Again, somewhat frustrating but it’s a usable product. 

And during Hurricane Fiona?

Starlink performed like a champ during the hurricane. I assumed that coverage would be spotty during the drenching downpour but the user said no he was streaming Netflix. It was just a little more spotty than usual. Now that the island is without power as I write this, his Internet service is as good as usual and the day after the hurricane was a normal (remote) work day like any other.

Conclusion

Don’t throw away your cable modem*. In general as of this writing in June 2022, Starlink is a good solution for those working from home, but be prepared to be bumped every 20 minutes or so from your video conferencing or other real time uses. And of course it’s good for surfing the web or on-demand streaming.

I don’t cnosider this the final word however. There’s still hope. I’ll update this post if the quality of service ever improves.

*Unless you’re one of the many whose cable modem service isn’t all that great to begin with.

References and related

https://en.wikipedia.org/wiki/Starlink

This is a fascinating article providing insight into how the StarLink network of satellites is being built and the problems that can occur: https://www.msn.com/en-us/news/technology/destruction-event-from-sun-annihilated-dozens-of-spacex-satellites/ar-AA11TIwE?ocid=winp1taskbar&cvid=4c9b1b5306484cdcd2ab6bf99748bf01

Categories
Consumer Tech Web Site Technologies

Consumer tech: Edge new tab in Chinese

Intro

If you’ve ever had the misfortune to access a web site in China in your Edge browser, you may find that from that point onwards all your new tab pages display in Chinese despite of your best efforts to eradicate it.

The details

I was in that same boar until today. There are many bad leads out there on the Internet. In fact I never did find the solution on the Internet. I got it from a colleague.

You click on the three dots, go to Settings and search for reset.

Do the Reset. It is a little disruptive, as i have found. It does not delete everything, but it certainly resets some things. As soon as that’s done you will no longer have new tab pages be in Chinese.

Categories
Consumer Interest Consumer Tech

Consumer tech: Android phone tip of the day

Intro

My wife was stuck while using the WW app on her Samsung A51 smartphone. She needed to lookup a nearby “studio.” We’ve all seen these forms – you enter a zipcode and up pops their nearby locations. But in this case there was a problem. No keyboard was popping up! Instead the bottom of the screen below the search field was filled with some blather which we could not get rid of to reveal the presumably hidden software keyboard.

The (kludge) solution

Please note that I am a specialist in doing things the wrong way that manage to get it done. I noticed the field still permitted long touch, and hence paste (from the clipboard). So I told her to enter the zipcode into another app such as Evernote, copy that text into the clipboard, return to the WW app and paste it into that field.

And do you know – that actually worked!

Conclusion

So if you’re in a jam and just need to fill out a field on your Android phone but your software keyboard isn’t appearing, a way out is to paste the desired content from another app such as Evernote or Onenote.

References and related

I use this one all the time: find my phone – no BS apps, just the straight-up Google URL for this built-in service.

Categories
Admin Linux Network Technologies Web Site Technologies

The IT Detective Agency: This site can’t be reached

Intro

It’s been awhile since I’ve had the opportunity to relatean IT mystery. After awhile they are repates of what’s already happened in the past, or it’s too complex to relate, or I was only peripherally involved. But today I came across a good one. It falls into the never been seen before category.

The details

A web server behind my web application firewall became unreachable. In the browser they get a message This site can’t be reached. The app owners came to me looking for input. I checked the WAF and it was fine. The virtual server was looking healthy. So I took a packet trace, something to this effect:

$ tcpdump -nni 0.0 host 192.168.2.124

14:00:45.180349 IP 192.68.1.13.42045 > 192.68.2.124.443: Flags [S], seq 1106553901, win 23360, options [mss 1460,sackOK,TS val 3715803515 ecr 0], length 0 out slot1/tmm3 lis=/Common/was90extqa.drjohn.com.app/was90extqa.drjohn.com_vs port=0.53 trunk=
14:00:45.181081 IP 192.68.2.124 > 192.68.1.13: ICMP host 192.68.2.124 unreachable - admin prohibited filter, length 64 in slot1/tmm2 lis= port=0.47 trunk=
14:00:45.181239 IP 192.68.1.13.42045 > 192.68.2.124.443: Flags [R.], seq 1106553902, ack 0, win 0, length 0 out slot1/tmm3 lis=/Common/was90extqa.drjohn.com.app/was9
0extqa.drjohn.com port=0.53 trunk=

I’ve never seen that before, ICMP host 192.68.2.124 unreachable – admin prohibited filter. But I know ICMP can be used to relay out-of-band routing information on occasion, though I do not see it often. I suspect it is a BAD THING and forces the connection to be shut down. Question is, where was it coming from?

The communication is via a firewall so I check the firewall. I see a little more traffic so I narrow the filter down:

$ tcpdump -nni 0.0 host 192.168.2.124 host 443

And then I only see the initial SYN packet followed by the RST – from the same source IP! So since I didn’t see the bad ICMP packet on the firewall, but I do see it on the WAF, I preliminarily conclude the problem exists on the WAF.

Rookie mistake! Did you fall for it? So very, very often, in the heat of debugging, we invent some unit test which we’ve never done before, and we have to be satisified with the uncertainty in the testing method and hope to find a control test somehow, somewhere to validate our new unit test.

Although I very commonly do compound filters, in this case it makes no sense, as I realized a few minutes later. My port 443 filter would of course exclude logging the bad ICMP packets because ICMP does not use tcp port 443! So I took that out and re-run it. Yup. bad ICMP packet still present on the firewall, even on the interface of the firewall directly connected to the server.

So at this point I have proven to my satisfaction that this packet, which is ruining the communication, really comes frmo the server.

What the server guys say

Server support is outsourced. The vendor replies

As far as the patching activities go , there is nothing changed to the server except distro upgrading from 15.2 to 15.3. no other configs were changed. This is a regular procedure executed on almost all 15.2 servers in your environment. No other complains received so far…

So, the usual It’s not us, look somewhere else. So the app owner asks me for further guidance. I find it’s helpful to create a test that will convince the other party of the error with their service. And what is one test I would have liked to have seen but didn’t cnoduct? A packet trace on the server itself. So I write

I would suggest they (or you) do a packet trace on the server itself to prove to themselves that this server is not behaving ini an acceptable way, network-wise, if they see that same ICMP packet which I see.

The resolution

This kind of thing can often come to a stand-off, or many days can be wasted as an issue gets escalated to sufficiently competent technicians. In this case it wasn’t so bad. A few hours later the app owners write and mention that the home-grown local firewall seemed suspect to them. They dsabled it and this traffic began to work.

They are reaching out to the vendor to understand what may have happened.

Case: closed!

Conclusion

An IT mystery was resolved today – something we’ve never seen but were able to diagnose and overcome. We learned it’s sometimes a good thing to throw a wider net when seeing unexpected reset packets because maybe just maybe there is an ICMP host unreachable packet somewhere in the mix.

Most firewalls would just drop packets and you wait for a timeout. But this was a homegrown firewall running on SLES 15. So it abides by its own ways of working, I guess. So because of the RST, your connection closes quickly, not timing out as with a normal network firewall.

As always, one has to maintain an open mind as to the true source of an issue. What was working yesterday does not today. No one admits to changing anything. Finding clever ad hoc unit tests is the way forward, and don’t forget to validate the ad hoc test. We use curl a lot for these kinds of tests. A browser is a complex beast and too much of a black box.

Categories
Network Technologies

How to force snmpwalk to convert strings to numeric OIDs

Intro

It’s a little hard to find this information on the Internet, so I’m amplifying the correct answer here by using my blog.

The details

I’m not super-competent with MIBs and such, but I manage for my purposes with my basic understanding. I have access to an F5 bigip with various IPSEC tunnels on it. I want to use Zabbix to check the status of those tunnels. So I do an SMPwalk like this:

snmpwalk -v3 … -c public 127.0.0.1 F5-BIGIP-SYSTEM-MIB::sysIpsecSpdStatTunnelState

which produces output like this line:

F5-BIGIP-SYSTEM-MIB::sysIpsecSpdStatTunnelState.”/Common/tunnel-01″.58401 = STRING: up

But I cannot take that as it is and use it in an snmpget like this:

snmpget -v3 … -c public 127.0.0.1 F5-BIGIP-SYSTEM-MIB::sysIpsecSpdStatTunnelState.”/Common/tunnel-01″.58401

That produces an error like this:

Unknown Object Identifier (Index out of range: /Common/tunnel-01 (sysIpsecSpdStatTrafficSelectorName))

So we need to convert the string into a numeric OID. But how?

The answer

Use the -On switch as an additional argument in your snmpwalk.

You will get a scary long OID, but it will at least be numeric.

Gonig further

You can then deconstruct the response and reconstitute the section at the beginning with a nice name. For my F5 example

.1.3.6.1.4.1.3375.2.1.2.17.1.3.1.14

becomes

F5-BIGIP-SYSTEM-MIB::sysIpsecSpdStatBytes

I think. Then preserve the following digits as is.

Conclusion

We have shown how to output a numeric OID from an snmpwalk. This, specifically, is sueful in converting a string embedded in the output into a numeric OID, which may then be used by other SNMP applications such as Zabbix which may or may not have the MIB file loaded. The secret is simply to use the -On switch in snmpwalk.

References and related

My Zabbix FAQ – questions you wish they had answered, can be very helpful