We created a convention where-in our scripts log to syslog with a certain style. Originally these were Perl scripts but newer scripts are written in python. My question was, how to do in python what we had done in Perl?
The details
The linux system uses syslog-ng. In /etc/syslog-ng/conf.d I created a test file 03drj.conf with these contents:
import syslog
syslog.openlog('jhtest',syslog.LOG_PID,facility=syslog.LOG_LOCAL0)
syslog.syslog(syslog.LOG_NOTICE,'[Notice] Starting')
syslog.syslog(syslog.LOG_ERR,'[Error] We got an error')
syslog.syslog(syslog.LOG_INFO,'[Info] Just informational stuff')
Easy, right? Then someone newer to python showed me what he had done – not using syslog but logger, in which he accomplished pretty much the same goal but by different means. But he had to hard-code a lot more of the values and so it was not as elegant in my opinion.
In any case, the output is in /var/log/drjtest.log which look like this after a test run:
Jul 24 17:45:32 drjohnshost drjtest[928]: [Notice] Starting
Jul 24 17:45:32 drjohnshost drjtest[928]: [Error] We got an error
Jul 24 17:45:32 drjohnshost drjtest[928]: [Info] Just informational stuff
OSes using rsyslog
Today I needed to make this style of logging work on a new system which was running rsyslog. The OS is SLES v 15. For this OS I added a file to /etc/rsyslog.d called drjtest.conf with the contents:
if ($programname == 'drjtest' ) then {
-/var/log/drjtest.log
stop
}
But the python program did not need to change in any way.
Conclusion
We show how to properly use the syslog facility within python by using the syslog package. It’s all pretty obvious and barely needs to be mentioned, except when you’re just statring out you want a little hint that you may not find in the man pages or the documentation at syslog-ng.
References and related
I have a neat script which we use to parse all these other scripts and give us once a week summary emails, unless and error has been detected in which case the summary email goes out the day after a script has reported an error. It has some pretty nice logic if I say so myself. Here it is: drjohns script checker.
I used the Flux language available within Grafana for my InfluxDB data source to do a number of things which, while easy, are not easy to find. So here I am doing a brain dump to document the language features I’ve so far used, and especially to describe these features with terms common to other more well-known languages such as python.
A word about Flux
Nothing is straightforward when it comes to Flux. And to make matters worse, the terminology is also strange. I am at the beginning level of beginning, so I don’t know their terminology, much less positioned to defend it.
Flux vs InfluxQL
It seems InfluxQL is the old stuff. It had SQL-like statements. Flux syntax seems rather different. But its capabilities have been enhanced. But since flux is a common word I am never sure how to formulate an Internet search. Flux lang? So when you do searches you’ll often see references to the old stuff (SELET statements and the like), which I always avoid.
Versions
I believe, here in June 2023 as I write this, that our InfluxDB is v 2.x, Grafana is version 10 (as of today!), so that gives us Flux v 0.x, I guess???
Time Series data
One has to say a word about time series data. But I’m not the one to say it. I simply muddle along until my stuff comes out as I wish it to. Read this somewhat helpful page to attempt to learn about time series: https://docs.influxdata.com/flux/v0.x/get-started/data-model/
Debugging
Grafana is very helpful in pointing out errors. You create your Flux query, try to refresh, and it throws up a traingular-shaped warning icon which says which exact characters of which exact line of your Flux query it doesn’t like. It it weren’t for this I’d have wasted many more hours beyond the ones already wasted.
Print out variable values to debug
In python I print out variable values to understand what’s happening. I have no idea how to do this in Influx. Of course I’m not even sure variable is the right word.
Example
This working example will be heplful to illustrate some of the things I’ve learned.
Begin a line with // to indicate it is a comment line. But // can also be used to comment at the end of a line.
Concatenating strings
Let’s start out easy. Concatenating strings is one of the few things which work the way you expect:
niceDate = “Date: ” + dayW + ” “+ year + “-” + month + “-” + day
Variablesare strongly typed
Let’s back up. Yes there are variables and they have one of a few possible types and the assignment operator = works for a few simple cases. So far I have used the types int, string, duration, time and dict. Note that I define all my variables before I get to the range statement. Not sure if that’s essential or not. If a function expects a duration, you cannot fake it out by using a string or an int! There are conversion functions to convert from one type to another.
literal string syntax: use double quotes
I’m so used to python’s indifference as to whether your string literals are enclosed by either single quotes or double quotes, and I prefer single quotes. But it’s only double quotes in Flux.
Add an additional column to the table
|> set(key:”hi”, value:niceDate)
Why do this? I want that in my stat visualization, when you hover over a stat, you learn the day and day-of-the-week.
Copy (duplicate) a column
|> duplicate(column: “_value”, as:”value”)
Convert the _value column to a string
I guess this only works on the _value column.
|> toString()
For arbitary conversion to a string:
string = string(v: variable)
But if you’re inside an fn function, you need something like this:
|> map(fn: (r) => ({r with valueString: string(v: r._value)}))
I guess it makes sense in a time series to devote a lot of attention to date arithmetic and such. In my script above I do some of the following things:
truncate the current day down to Midnight
add a full day to a date
pull out the year, month, date
convert a date object to a string
template variables
day is a template variable (I think that’s the term). It is set up as a hidden variable with the (hand-coded) values of 0,1,2,3,4,…32.
dropping columns
We all know about the drop(columns:[…, but how about if you have so many columns it’d be more economical to simply keep the ones you need? That is my situation, hence the line:
|> keep(columns:[“_time”,”_value”,”item”])
Lumping data together into every hour, aka data aggregation or windowing
Note the keep columns is in fron of the aggregation. Things didn’t turn out so well if I flipped the order.
Adding additional columns
|> set(key:”hi”, value:niceDate)
So when you hover over a row with the mouse, this will produce a pleasant Wednesday 2023-6-14
Appending additional series together
I have had success with union when the tables all had the same colunms.
union(tables: [meanTbl,n95Tbl,maxTbl])
Outputting additional series
Use yield. Let’s say you’ve outputted one table and want to output a second table based on different criteria. The second table can be output using
|> yield(name: “second”)
Regular Expressions (RegEx)
Should be supported. I haven’t had opportunity to use them yet however.
A short word on what I’m doing
I have a stat visualization where the stat blocks repeat vertically and there are 24 per row. Get it? Each stat contains a single number representing the value for that hour of the day. Then there is a repeat over template variable day.
Just above this panel is another thin panel which contains the numbers 0 through 23 so you know which hour of the day it is. That is another stat visualization which contains this code:
import "generate"
data = generate.from(
count: 24,
fn: (n) => n,
start: 2021-01-01T00:00:00Z,
stop: 2021-01-06T00:00:00Z,
)
data
|> range(start: 2021-01-01T00:00:00Z, stop: 2021-01-24T05:00:00Z)
|> set(key:"hi", value:"local vEdge hour")
Generate your own test data– generate.from()
The non-intuitive code snippet above shows one example of generating my own data into a table which is used to display stat blocks which contain 0, 1, 2, …, 23 and when you mouseover the pop-up text says “local vEdge hour.” The time references are mostly throw-away dummy values, I guess.
Loops or iterations
Flux sucks when it comes to program loops. And say yuo wanted nested loops? Forget about it. It’s not happening. I have no idea how to do a simple loop. I know, seriously? Except I only needed it for the data generation where I do show how to do it within a generate.from code block.
Refer to the time picker’s time range
This is surprisingly hard to find. I don’t know why.
Setting Grafana time picker to a stop time in the future
This is actually a Grafana thing, but since I’m on a roll, here goes. From now-5d To now+10h
Is Flux only good for Grafana?
No. They have made it its own stand-alone thing so it could be used in other contexts.
How to subtract two times
Intuitively since there is a date.add function you would expect a date.sub function so that you cuold subtract two times and get a duration as a result. But they messed up and omitted this obvious function. Also, just for the record, the subtract operator is not overloaded and therefor you canno simply do a c = b – a if a and b are of type time. You will get a
invalid: error @4:5-4:10: time is not Subtractable
Yet I wanted to do a comparison of the difference between two times and a known duration. How to do it? So what you can do is convert the two times into, e.g., day of the year with a date.yearDay function. The days are integers. Then subtract them and compare the difference to another integer (representing the number of days in this example) using simple integer comparison operator such as >.
Is flux dead?
Now, in October 2023, when I get online help on flux I see this information:
Flux is dead message
I have a lot invested in flux at this point so this could be a huge development for me. I’m seeking to get more details.
The examples provided on github are kind of wrong. I created an example script which actually works. If you simply copy their example and try the one where you add a DNS record using the python interface to the api, you will get this error:
CloudFlare.exceptions.CloudFlareAPIError: Requires permission “com.cloudflare.api.account.zone.create” to create zones for the selected account
Read on to see the corrected script.
Then some months later I created a script – still using the python api – to do a DNS export of all the zone files our account owns on Cloudflare. I will also share that.
The details
I call the program below listrecords.py. This one was copied from somewhere and it worked without modification:
import CloudFlare
import sys
def main():
zone_name = sys.argv[1]
cf = CloudFlare.CloudFlare()
# query for the zone name and expect only one value back
try:
zones = cf.zones.get(params = {'name':zone_name,'per_page':1})
except CloudFlare.exceptions.CloudFlareAPIError as e:
exit('/zones.get %d %s - api call failed' % (e, e))
except Exception as e:
exit('/zones.get - %s - api call failed' % (e))
if len(zones) == 0:
exit('No zones found')
# extract the zone_id which is needed to process that zone
zone = zones[0]
zone_id = zone['id']
# request the DNS records from that zone
try:
dns_records = cf.zones.dns_records.get(zone_id)
except CloudFlare.exceptions.CloudFlareAPIError as e:
exit('/zones/dns_records.get %d %s - api call failed' % (e, e))
# print the results - first the zone name
print("zone_id=%s zone_name=%s" % (zone_id, zone_name))
# then all the DNS records for that zone
for dns_record in dns_records:
r_name = dns_record['name']
r_type = dns_record['type']
r_value = dns_record['content']
r_id = dns_record['id']
print('\t', r_id, r_name, r_type, r_value)
exit(0)
if __name__ == '__main__':
main()
The next script adds a DNS record. This is the one which I needed to modify.
# kind of from https://github.com/cloudflare/python-cloudflare
# except that most of their python examples are wrong. So this is a working version...
import sys
import CloudFlare
def main():
zone_name = sys.argv[1]
print('input zone name',zone_name)
cf = CloudFlare.CloudFlare()
# zone_info is a list: [{'id': '20bd55fbc94ff155c468739', 'name': 'johnstechtalk-2.com', 'status': 'pending',
zone_info = cf.zones.get(params={'name': zone_name})
zone_id = zone_info[0]['id']
dns_records = [
{'name':'foo', 'type':'A', 'content':'192.168.0.1'},
]
for dns_record in dns_records:
r = cf.zones.dns_records.post(zone_id, data=dns_record)
exit(0)
if __name__ == '__main__':
main()
The zone_id is where the original program’s wheels fell off. Cloudflare Support does not support this python api, at least that’s what they told me. So I was on my own. What gave me confidence that it really should work is that when you install the python package, it also installs cli4. And cli4 works pretty well! The examples work. cli4 is a command line program for linux. But when you examine it you realize it’s (I think) using the python behind the scenes. And in the original bad code there was a POST just to get the zone_id – that didn’t seem right to me.
Backup all zones in the Cloudflare account by doing a DNS export
import os
import CloudFlare
from datetime import datetime
def listzones(cf):
allzones = list()
page_number = 0
while True:
page_number += 1
raw_results = cf.zones.get(params={'per_page':20,'page':page_number})
#print(raw_results)
zones = raw_results['result']
for zone in zones:
zone_id = zone['id']
zone_name = zone['name']
print("zone_id=%s zone_name=%s" % (zone_id, zone_name))
allzones.append([zone_id,zone_name])
total_pages = raw_results['result_info']['total_pages']
if page_number == total_pages:
break
#print('allzones',allzones)
return allzones
# main program
today = datetime.today().date() # today's date
date = today.strftime('%Y%m%d') # formatted date
print('Begin backup of zones on this day:',date)
newdir = 'zones-' + date
os.makedirs(newdir,exist_ok=True)
cf = CloudFlare.CloudFlare(raw=True)
print('Getting list of all zones and zone ids')
allzones = listzones(cf)
print('Begin export of the zone data')
for zone in allzones:
zone_id,zone_name = zone
print('Doing dns export of',zone_id,zone_name)
# call to do a BIND-style export of the zone, specified by zoneid
res = cf.zones.dns_records.export.get(zone_id)
dns_records = res['result']
with open(f'{newdir}/{zone_name}','w') as f:
f.write(dns_records)
# create compressed tar file and delete temp directory
print('Create compressed tar file')
os.system(f'tar czf backups/{newdir}.tar.gz {newdir}')
print(f'Remove directory {newdir} and all its contents')
os.system(f'rm -rf {newdir}')
As mentioned in the comments the cool thing in this backup is that the format output is the BIND style of files, which are quite readable. Obviously this script is designed for linux systems because that’s all I use.
Backup all zones again, without using the CF python api
After we upgraded to python 3.12 the CF api didn’t really work. It may just be we needed the private certificates installed. Regardless, that led us to create this backup script which does not depend on the CF python api. Well, it includes it, but it’s not material and could be removed. Instead direct url calls are made.
import os
import CloudFlare
from datetime import datetime,timezone
import urllib3
import requests
def listzones(cf):
allzones = list()
page_number = 0
results_per_page = 50 # max allowed
while True:
page_number += 1
#raw_results = cf.zones.get(params={'per_page':20,'page':page_number})
url = f"https://api.cloudflare.com/client/v4/zones?per_page={results_per_page}&page={page_number}"
raw_results = requests.get(
url=url,
headers={
'Authorization': 'Bearer {}'.format(os.environ['CLOUDFLARE_API_TOKEN']),
'Content-Type': 'application/json'
},
)
print('results')
#print(raw_results)
print(raw_results.json())
zones = raw_results.json().get('result')
for zone in zones:
zone_id = zone['id']
zone_name = zone['name']
print("zone_id=%s zone_name=%s" % (zone_id, zone_name))
allzones.append([zone_id,zone_name])
total_count = raw_results.json()['result_info']['total_count']
total_pages = int(total_count/results_per_page)
# fraction of a page is the last page, if applicable
if total_count % results_per_page > 0: total_pages += 1
if page_number == total_pages:
break
#print('allzones',allzones)
return allzones
# main program
urllib3.disable_warnings()
#today = datetime.utcnow() # today's date
today = datetime.now(timezone.utc)
date = today.strftime('%Y%m%d') # formatted date
print('Begin backup of zones on this day:',date)
newdir = 'zones-' + date
os.makedirs(newdir,exist_ok=True)
cf = CloudFlare.CloudFlare(raw=True, warnings=False)
print('Getting list of all zones and zone ids')
allzones = listzones(cf)
print('Total zone count:',len(allzones))
print('Begin export of the zone data')
for zone in allzones:
zone_id,zone_name = zone
print('Doing dns export of',zone_id,zone_name)
# call to do a BIND-style export of the zone, specified by zoneid
#res = cf.zones.dns_records.export.get(zone_id)
raw_results = requests.get(
url = 'https://api.cloudflare.com/client/v4/zones/{}/dns_records/export'.format(zone_id),
headers={
'Authorization': 'Bearer {}'.format(os.environ['CLOUDFLARE_API_TOKEN']),
'Content-Type': 'application/json'
},
)
dns_records = raw_results.text
print('dns records: first few lines, skip a few, then a few more entries')
print(dns_records[:128],dns_records[840:968])
with open(f'{newdir}/{zone_name}','w') as f:
f.write(dns_records)
# create compressed tar file and delete temp directory
print('Create compressed tar file')
os.system(f'tar czf backups/{newdir}.tar.gz {newdir}')
print(f'Remove directory {newdir} and all its contents')
os.system(f'rm -rf {newdir}')
The environment
Just to note it, you install the package with a pip3 install cloudflare. Then I set up an environment variable CLOUDFLARE_API_TOKEN before running these programs.
Conclusion
I’ve shown a corrected python script which uses the Cloudflare api. I’ve also shown another one which can do a backup of all Cloudflare zones.
I’m looking to test my old Raspberry Pi model 3 to see if it can play mp4 videos I recorded on my Samsung Galaxy A51 smartphone. I had assumed it would get overwhelmed and give up, but I haven’t tried in many years, so… The first couple videos did play, sort of. I was using vlc. Now if you’ve seen any of my posts you know I’ve written a zillion posts on running a dynamic slideshow based on RPi. Though the most important of these posts was written years ago, it honestly still runs and runs well to this day, amazingly enough. Usually technology changes or hardware breaks. But that didn’t happen. Every day I enjoy a brand new slideshow in my kitchen.
In most of my posts I use the old stalwart program fbi. In fact I don’t even have XWindows installed – it’s not a requirement if you know what you’re doing. But as far as I can see, good ‘ole fbi doesn’t do streaming media such as videos in mp4 format. As far as I know, vlc is more modern and most importantly, better supported. So after a FAIL trying with mplayer (still haven’t diagnose that one), I switched to trials with vlc.
I haven’t gotten very far, and that’s why I wanted to share my learnings. There’s just so much you can do with vlc, that even what you may think are the most common things anyone would want are very hard to find working examples for. So that’s where I plan to contribute to the community. As I figure out an ‘easy” thing, I will add it here. And if I’m the only one who ever refers to this post, so be it. I love my own My favorite python tips, post, for instance. it has everything I use on a regular basis. So I’m thinking this will be similar.
After its customary overnight charging my A51 simply showed me a black screen in the morning. Yet I felt something was there because when I plugged it into the computer’s USB port the device was recognized. I was very concerned. But I did manage to completely fix it!
The symptoms
So various sites address this problem and give somewhat different advice. I sort of needed to combine them. So let’s review.
Black screen
Holding power button down for any length of time does nothing
plugging in to USB port of computer shows A51 device
What kind of works, but not really
Yes it’s true that holding the power button and volume down button simultaneously for a few seconds (about three or four) will bring up a menu. The choices presented are
Restart
Power off
Emergency Call
There’s no point to try Emergency Call. But when you try Restart you are asked to Restart a second time. Then the screen goes black again and you are back to where you started. If you choose Power off the screen goes black and you are back to where you started.
What actually works
Continue to hold the power button and volume down button simultaneously – ignore the screen you get mentioned above. Then after another 15 seconds or so it displays a lightning bolt inside a cricle. And if you keep holding that will disappear and you have a black screen. Keep holding and the lightning bolt appears, etc. So let them go. I don’t think it matters at which stage.
Now hopefully you have realy powered off the phone. So then hold the power button for a few seconds like you do to start the phone after it’s been powered off. It should start normally now.
As the other posts say, when you see Samsung on your screen you know you are golden.
Conclusion
I have shared what worked for me recover my Samsugn Galaxy A51 from its Black Screen of Death.
I give to lots of non-profits and realized there are some common elements and that perhaps others could learn from what I have observed. I also give to some political organizations.
Define terms
By non-profit I mean a 501(c)(3) as per IRS regulations. These can have a local focus or a national focus. They will be chartered in a particular state which will have its own rules for incorporation. For my purposes that doesn’t matter too much. I believe they all will have a board of directors. They all have to abide by certain rules such as spending most of what they take in (I think).
Common to all
Engage, engage, engage
They want to send you frequent correspondences, sometimes under various pretenses, to keep you engaged. You will receive correspondence under the following pretenses: the “annual renewal”, the “quarterly report”, the xmas update, the thankyou for contributing letter, the “for tax purposes” letter, the emergency appeal or rapid reaction to something in the news, the special donor multiplying your gift by 3x or even 10x, the estate giving solicitation, and, worst of all, the fake survey. I’m talking about you, ACLU. I have never once seen a published result of these fake surveys, which have zero scientific value and consist of one-sided questions. I used to fill them out the opposite way they expected out of spite, but to no avail as they kept coming with self-addressed stamped envelopes no less. All these correspondences have in common that they will always solicit you to give even more money as though what you’ve already given isn’t good enough.
But by all means read the newsletters on occasion to make sure they are doing the things you expect of them based on their mission. And ignore the extra pleas for money unless you are truly sympathetic. Emergencies do occur, after all.
Snail mail? No problem
You would naively think that by creating a known, non-trivial cost to these non-profits, namely forcing them to contact you by postal mail that they would send you fewer requests for money. Not so! I only contribute online when it seems to be the only practical way to do so (I’m thinking of you, Wikimedia), yet still, I get, no exaggeration, about a letter every two weeks from my organizations.
Phone etiquette
First off, you don’t need to give out your phone number even though they ask for it. It’s asked for in a purposefully ambiguous way, near the billing, as though it is needed to process your credit card. It isn’t. I happily omit my phone number. I figure if they really need it they can just wrote me a letter asking for it – and that’s never happened.
But if you’ve made the mistake of having given out your number, perhaps in the past, you may get called periodically. They do have the right to call you. But you can ask them to put you on a do not call list. What I do once I learn what organization is calling, is to sometime during their long opening sentence – which may come after you confirmed your identity – is to hold the phone away from my ear a little and just calmly say I’m not going to give any more money and hang up.
Universities have a special way of asking for money. I knew classmates who did this for their campus job. They call alumni, especially recent alumni who are more naive, and engage them with a scripted conversation that starts innocently enough, :I’m from your college and I wanted to update you update recent happenings on campus.” pretty soon they’re being asked to donate at the $250 level, then after an uncomfortable No, they’re relieve to learn they can still contribute at the $125 level, and so on down until the hapless alumnus/alumna is guilted into contributing something, anything at all.
Local giving
Fortunately, local giving where they haven’t signed on to use professional fund-raising organizations is more pleasant because you are normally not solicited very often, often just once a year.
Track it
I keep a spreadsheet with my gifts and a summed column at the bottom. I create a new worksheet named with the year for each new year. I have a separate section at the bottom with my non-deductible contributions.
I try to give all my gifts in the first one or two months of the year.
Come tax time, I print out the non-profit giving and include it with my paperwork for my accountant, but more on that below.
Deductions – forget about it
I pat a lot of taxes and still these days (from about 2018 onwards) I don’t get any tax credit for my contributions. Why? The reason is that the standard deduction is so high that it applies instead. This is the case ever since the tax changes of 2017. So if that’s true for me I imagine that’s true for most people. But each year I try…
Non-deductible organizations
Some organziations you would think are non-profits, but they are actually structured differently and so they are not. I’m thinking of you, The Sierra Club. The Sierra Club is using much of your donation to lobby politicians to their point of view about environmental issues and therefore by the rules cannot be a non-profit in the sense of a 501(c)(3).
Privacy
I’m not sure what privacy rules apply around your giving. In my personal experience, there are few constraints. This means expect your name to be sold as part of a giant list of donors. You are data and worth $$ to the organization selling your name to, usually, like-minded organizations who will hope to extract more money out of you. To be concrete, let’s say you donated one time to a senator in a tight senate race. Before six months is up every senator in a competitive race from that party will be soliciting you for funds. And not just one time but often on that bi-weekly basis! Once again using snail mail seems to be no obstacle. maybe it is even worse because with email you can in theory unsubscribe. I haven’t tried it but perhaps I should. I just hate to have my inbox flooded with solicitations. I’m really afraid to contribute to a congressional race for this reason. Or a governor’s race.
But this privacy issue is not just restricted to PACs sharing your data. Let’s say a relative had congenital heart failure so you decide to contribute to a heart association. Eventually you will be solicited by other major organizations with focus on other organs or health in general: lungs, kidneys, cancer, even the same organ but from a different organization, etc. Your data has been sold…
Amazon Smile – Giving while shopping
When I first learned of Amazon Smile from a friend at work I thought there was no way this could be true. Margins are said to be razor thin in retail, yet here was Amazon giving away one half percent of your order to the charity of your choice?? Yet it was true. And Amazon gave away hundreds of millions of dollars. Even my local church got into the program. My original recipient was Habitat for Humanity, which raised well over ten thousand dollars from Amazon Smile.
But Amazon killed this too-good-to-be-true program in March 2023 for reasons unknown. I’m not sure if other merchants have something which can replace it and will update this if I ever find out.
The efficiency of your charity
You want to know if a large portion of your gift to a particular charity is going towards the cause that is its mission, or, to administrative costs such as fund-raising itself. I’ve noticed good charities actually show you a pie chart which breaks down the amount taken by administrative overhead – usually 5 – 10 percent. But another way to learn something about efficiency is to use a third party web site such as Charity Navigator. But don’t get crazy about worrying about their ratings. I have read criticisms of their methods. Still, it’s better than nothing. 5 – 10 % administrative costs is fine. Hey, I used to know people who worked in such administrative positions and they are good people who deserve decent pay. Another drawback of Charity Navigator is that it won’t have ratings for your local charities.
For PACs as far as I know, there is no easy way to get comparable information. You just have to hope your money is well spent. I guess they have quarterly disclosure forms they fill out, but I don’t know how to hunt that down.
Tactics
The national organizations know everything you have ever given and will suggest you give at slightly higher amounts than you have in the past. 25 years ago the American Cancer Society asked if I would solicit my neighbors for contributions, which I did. I pooled all the money and gave them something like $300. I swear for the next 15 years they solicited me suggesting I contribute at that level even though I never gave more than $40 in the following 15 years. So annoying…
Death – an opportunity – for them
Many charities will encourage you to remember them in your estate planning. I suppose this may be a reasonable option if you feel really identified with their cause. I suppose The Nature Conservancy evokes these kinds of emotions, for example, because who doesn’t love nature? So think about your legacy, what you’re leaving behind.
National, with local chapters
Some national charities have local chapters. I’m thinking of you, Red Cross. I’m not really sure how this works out. But I know I have received solicitations from both the local chapter as well as the national chapter. So just be mindful of this. I suppose when you give to the local chapter it has more discretion on spending your donation locally and I guess giving a fraction of it to the national chapter.
Charitable Annuities
I don’t know all the details but if you have for instance appreciated equities instead of paying capital gains taxes you could gift them to a charity and receive a deduction for the gift. They in turn, if they’re a big outfit, usually a university, can set up a charitable annuity which provides you further tax benefits. I will flesh this out if I ever come across it again.
Conclusion
As a reliable contributor I am annoyed by the methods employed to shake even more out of my pockets. But I guess those methods work in bulk and so they continue to be used. As far as I can tell all national non-profits use professional fund-raising methods which closely follow the same patterns.
Although the tenor of this post is terribly cynical, obviously, I think non-profits are doing invaluable work and filling some of the gaping holes left by government. If I didn’t think so I wouldn’t be contributing. Most non-profits do good work and are run efficiently, but the occasional scam happens.
I wrote about my new HP Pavilion Aero laptop previously and how pleased I am with this purchase. And I’m not getting any kickbacks from HP for saying it! Well, this week was a sad story as all of a sudden, the wireless driver could no longer detect the presence of the Mediatek Wireless card. We hadn’t done anything! All the reboots in the world didn’t help. Fortunately it is still under warranty and fortunately HP’s consumer tech support is actually quite good. They helped me fix the problem. I wish to share with the wider community what happened and what fixed it.
The symptoms
No amount of rebooting fixes the issue
WiFi tile no longer appears (so there is no option to simply turn WiFi back on because you accidentally turned it off)
duet.exe file is not found (I don’t think this matters, honestly)
Where you normally see a WiFi icon in the shape of an amphitheater in the system tray, instead you only see:
a globe for the WiFi icon
HP PC Hardware Diagnostics Windows utility shows:
wireless IRQ test (RZ616) 160 MB FAILED
wireless ROM test FAILED
This diagnostics tool can be run in BIOS mode. It restarts the computer and puts you into a special BIOS mode diagnostics. When you run the wireless networking component test:
BIOS level component test of wireless networking PASSES
Yes, that’s right. You really didn’t fry the adapter, but Windows 11 totally messed it up.
On my own I tried…
to run HP PC Hardware Diagnostic Windows utility. It suggested I upgrade the BIOS, which I did. I ran some checks. The wireless IRQ test (RZ616) 160 MB failed, as did the wireless ROM test.
I uninstalled the Mediatek driver and reinstalled it.
Nothing doing. I had the insight to make the laptop useful, i.e., connected to Internet, by inserting an old USB wireless adapter that I used to use for my old Raspberry Pi model 2’s! It worked perfectly except only at 2.4 GHz band, ha, ha. But I knew that wasn’t a long-term solution.
Quickly…
The BIOS diag succeeded.
Hold the power button down for a long time to bring up a new menu. The sequence which results from holding power button down a long time seems to be:
Initial normal boot
Forced shutdown
Boot into a special BIOS submenu
Then you enable something. I don’t remember what. But it should be obvious as there were not a lot of choices.
Another reboot, and voila, the WiFi normal icon appears, though it has forgotten the passwords to the networks.
A word about HP support
Maybe I got a tech support person who was exceptionally knowledgeable, but I have to say tech support was exceptional in its own right. And this is coming from someone who is jaded with regards to tech support. My support person was clearly not simply following a script, but actually creatively thinking in real time. So kudos to them.
Conclusion
I lost my Mediatek WiFi adapter on my brand new HP Pavilion Aero notebook which I was so enamored with. HP support said it was due to a deficiency in the way Microsoft does Windows 11 upgrades. But they did not dance around the issue and helped me to resolve it. Although I don’t exactly what we did, I have tried to provide enough clues that someone else could benefit from my misfortune. Or perhaps I will be the beneficiary should this happen again.
I sometimes find myself in need to blur images to avoid giving away details. I once blurred just a section of an image using a labor-intensive method involving MS Paint. Here I provide a python program to blur an entire image.
The program
I call it blur.py. It uses the Pillow package and it takes an image file as its input.
# Dr John - 4/2023
# This will import Image and ImageChops modules
import optparse
from PIL import Image, ImageEnhance
import sys,re
p = optparse.OptionParser()
p.add_option('-b','--brushWidth',dest='brushWidth',type='float')
p.set_defaults(brushWidth=3.0)
opt, args = p.parse_args()
brushWidth = opt.brushWidth
print('brushWidth',brushWidth)
# Open Image
image = args[0]
print('image file',image)
base = re.sub(r'\.\S+$','',image)
file_type = re.sub(r'^.+\.','',image)
canvas = Image.open(image)
width,height = canvas.size
print('Original image size',width,height)
widthn = int(width/brushWidth)
heightn = int(height/brushWidth)
smallerCanvas = canvas.resize((widthn, heightn), resample=Image.Resampling.LANCZOS)
# Creating object of Sharpness class
im3 = ImageEnhance.Sharpness(smallerCanvas)
# no of blurring passes to make. 5 seems to be a minimum required
iterations = 5
# showing resultant image
# 0,1,2: blurred,original,sharpened
for i in range(iterations):
canvas_fuzzed = im3.enhance(0.0)
im3 = ImageEnhance.Sharpness(canvas_fuzzed)
# resize back to original size
canvas = canvas_fuzzed.resize((width,height), resample=Image.Resampling.LANCZOS)
canvas.save(base + '-blurred.' + file_type)
So there would be nothing to write about if the the Pillow ImageEnhance worked as expected. But it doesn’t. As far as I can tell on its own it will only do a little blurring. My insight was to realize that by making several passes you can enhance the blurring effect. My second insight is that Image Enhance is probably only working within a 1 pixel radius. I have intruduced the concept of a brush size where the default width is 3.0 (pixels). I effectuate a brush size by reduing the image by the factor equal to the brush size! Then I do the blurring passes, then finally resize back to the original size! Brilliant, if I say so myself.
So in general it is called as
$ python3 blur.py -b 5 image.png
That example would be to use a brush size of 5 pixels. But that is optional so you can use my default value of 3 and call it simply as:
$ python3 blur.py image.png
Example output
Blur a select section of an image
You can easily find the coordinates of a rectangular section of an image by using, e.g., MS Paint and doing a mouseover in the corners of the rectangular section you wish to blur. Note the coordinates in the upper left corner and then again in the lower right corner. Mark them down in that order. My program even allows more than one section to be included. In this example I have three sections. The resulting image with its blurred sections is shown below.
Three rectangular setions of this image were blurred
Here is the code, which I call DrJblur.py for lack of a better name.
# blur one or more sections of an image. Section coordinates can easiily be picked up using e.g., MS Paint
# partially inspired by this tutorial: https://auth0.com/blog/image-processing-in-python-with-pillow/
# This will import Image and ImageChops modules
from PIL import Image, ImageEnhance
import sys,re
def blur_this_rectangle(image,x1,y1,x2,y2):
box = (x1,y1,x2,y2)
cropped_image = image.crop(box)
# Creating object of Sharpness class
im3 = ImageEnhance.Sharpness(cropped_image)
# no of blurring passes to make. 10 seems to be a minimum required
iterations = 10
# showing resultant image
# 0,1,2: blurred,original,sharpened
for i in range(iterations):
cropped_fuzzed = im3.enhance(-.75)
im3 = ImageEnhance.Sharpness(cropped_fuzzed)
# paste this blurred section back onto original image
image.paste(cropped_fuzzed,(x1,y1)) # this modified the original image
# Open Image
image = sys.argv[1]
base = re.sub(r'\.\S+$','',image)
file_type = re.sub(r'^.+\.','',image)
canvas = Image.open(image)
argNo = len(sys.argv)
boxNo = int(argNo/4) # number of box sections to blur
# (x1,y1) (x2,y2) of rectangle to blur is the next argument
for i in range(boxNo):
j = i*4 + 2
x1 = int(sys.argv[j])
y1 = int(sys.argv[j+1])
x2 = int(sys.argv[j+2])
y2 = int(sys.argv[j+3])
blur_this_rectangle(canvas,x1,y1,x2,y2)
canvas.save(base + '-blurred.' + file_type)
Since it can be a little hard to find an a simple and easy-to-use blurring program, I have written my own and provided it here for general use. Actually I have provided two programs. One blurs an entire picture, the other blurs rectangular sections within a picture. Although I hardcoded 10 passes, that number may need to be increased depending on the amount of blurriness desired. To blur a larger font I changed it to 50 passes, for example!
Obviously, obviously, if you have a decent image editing program like an Adobe Photoshop, you would just use that. There are also probably some online tools available. I myself am leery of using “free” online tools – there is always a hidden cost. And if you all you want to do is to erase in that rectangle and not blur, even lowly MS Paint can do that quite nicely all on its own. But as for me, I will continue to use my blurring program – I like it!
References and related
The need for the ability to blur an image arose when I wanted to share something concrete resulting from my network diagram as code effort.
Since they took away our Visio license to save licensing fees, some of us have wondered where to turn to. I once used the venerable old MS Paint after learning one of my colleagues used it. Some have turned to Powerpoint. Since I had some time and some previous familiarity with the components – for instance when I create CAD designs for 3D printing I am basically also doing CAD as code using openSCAD – I wondered if I could generate my network diagram using code? It turns out I can, at least the basic stuff I was looking to do.
Pillow
I’m sure there are much better libraries out there but I picked something that was very common although also very limited for my purposes. That is the python Pillow package. I created a few auxiliary functions to ease my life by factoring out common calls. I call the auxiliary modules aux_modules.py. Here they are.
from PIL import Image, ImageDraw, ImageFont
serverWidth = 100
serverHeight = 40
small = 5
fnt = ImageFont.truetype('/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf', 12)
fntBold = ImageFont.truetype('/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf', 11)
def drawServer(img_draw,xCorner,yCorner,text,color='white'):
# known good colors for visibility of text: lightgreen, lightblue, tomato, pink and of course white
# draw the server
img_draw.rectangle((xCorner,yCorner,xCorner+serverWidth,yCorner+serverHeight), outline='black', fill=color)
img_draw.text((xCorner+small,yCorner+small),text,font=fntBold,fill='black')
def drawServerPipe(img_draw,xCorner,yCorner,len,source,color='black'):
# draw the connecting line for this server. We permit len to be negative!
# known good colors if added text is in same color as pipe: orange, purple, gold, green and of course black
lenAbs = abs(len)
xhalf = xCorner + int(serverWidth/2)
if source == 'top':
coords = [(xhalf,yCorner),(xhalf,yCorner-lenAbs)]
if source == 'bottom':
coords = [(xhalf,yCorner+serverHeight),(xhalf,yCorner+serverHeight+lenAbs)]
img_draw.line(coords,color,2)
def drawArrow(img_draw,xStart,yStart,len,direction,color='black'):
# draw using several lines
if direction == 'down':
x2,y2 = xStart,yStart+len
x3,y3 = xStart-small,y2-small
x4,y4 = x2,y2
x5,y5 = xStart+small,y3
x6,y6 = x2,y2
coords = [(xStart,yStart),(x2,y2),(x3,y3),(x4,y4),(x5,y5),(x6,y6)]
if direction == 'right':
x2,y2 = xStart+len,yStart
x3,y3 = x2-small,y2-small
x4,y4 = x2,y2
x5,y5 = x3,yStart+small
x6,y6 = x2,y2
coords = [(xStart,yStart),(x2,y2),(x3,y3),(x4,y4),(x5,y5),(x6,y6)]
img_draw.line(coords,color,2)
img_draw.line(coords,color,2)
def drawText(img_draw,x,y,text,fnt,placement,color):
# draw appropriately spaced text
xy = (x,y)
bb = img_draw.textbbox(xy, text, font=fnt, anchor=None, spacing=4, align='left', direction=None, features=None, language=None, stroke_width=0, embedded_color=False)
# honestly, the y results from the bounding box are terrible, or maybe I don't understand how to use it
if placement == 'lowerRight':
x1,y1 = (bb[0]+small,bb[1])
if placement == 'upperRight':
x1,y1 = (bb[0]+small,bb[1]-(bb[3]-bb[1])-2*small)
if placement == 'upperLeft':
x1,y1 = (bb[0]-(bb[2]-bb[0])-small,bb[1]-(bb[3]-bb[1])-2*small)
if placement == 'lowerLeft':
x1,y1 = (bb[0]-(bb[2]-bb[0])-small,bb[1])
xy = (x1,y1)
img_draw.text(xy,text,font=fntBold,fill=color)
How to use
I can’t exactly show my eample due to proprietary elements. So I can just mention I write a main program making lots of calls tto these auxiliary functions.
Tip
Don’t forget that in this environment, the x axis behaves like you learned in geometry class with positive x values to the right of the y axis, but the y axis is inverted! So positive y values are below the x axis. That’s just how it is in a lot of these programs. get used to it.
What I am lacking is a good idea to do element groupings, or an obvious way to do transformations or rotations. So I just have to keep track of where I am, basically. But even still I enjoy creating a network diagram this way because there is so much control. And boy was it easy to replicate a diagram for another one which had a similar layout.
It only required the Pillow package. I am able to develop my diagrams on my local PC in my WSL environment. It’s nice and fast as well.
Example Output
This is an example output from this diagram as code approach which I produced over the last couple days, sufficiently blurred for sharing.
Network diagram (blurred) resulting from use of this code-first approach
Conclusion
I provide my auxiliary functions which permit creating “network diagrams as code.” The results are not pretty, but networking people will understand them.
I am very pleased with my online purchase of an HP lsptop. So I am sharing my experience here. Believe it or not, I did not, unfortunately, receive anything for this endorsement! I simply am thrilled with the product. I heartily recommend this laptop to others if it is similarly configured.
Requirements
Requirements are never made in the abstract, but represent a combination of what is possible and what others offer.
laptop
13″ diagonal screen
lightweight
“fast,” whatever that means
future-proof, if at all possible
distinctive (you’ll see what that means in a second)
durable
no touch-screen!! (hate them)
Windows 11 Home Edition
under $1200
1 TB of storage space
SSD
HP brand
What I got
I used to be a fan of Dell until I got one a few years back in which the left half of the keyboard went dead. It seems that problem was not so uncommon when you would do a search. Also my company seems to much more on the HP bandwagon than the Dell one, and they generally know what they are doing.
I remember buying an HP Pavilion laptop in November 2017. It was an advertised model which had the features I sought at the time, including Windows 7, 512 GB SSD disk. Surely, with the inexorable improvements in everything, wouldn’t you have thought that in the intervening five years, 1 TB would be commonplace, even on relatively low-end laptop models? For whatever reason, that upgrade didn’t happen and even five years later, 1 TB is all but unheard of on sub $1000 laptops. I guess everyone trusts the cloud for their storage. I work with cloud computing every day. But I want the assurance of having my photos on my drive, and not exclusively owned by some corporation. And we have lots of photos. So our Google Drive is about 400 GB. So with regards to storage, future-proof for me means room to grow for years, hence, 1 TB.
My company uses HP Elitebooks. They have touchscreens which I never use and are more geared towards business uses. Not only do I dislike touchscreens (you’re often touching them unintentionally), but they add weight and draw power. So not having one – that’s a win-win.
So since so few cheap laptops offer 1 TB standard, I imagined, correctly, that HP would have a configurator. The model which supports this is the HP Pavilion Aero. I configured a few key upgrades, all of which are worthwhile.
The screen size and the fact of running Windows 11 are not upgrades, everything else on the above list is. Some, like the cpu, a bit pricey. But my five-year-old laptop, which runs fine, by the way, is EOL because Microsoft refuses to support its cpu for Windows 11 upgrade. I’m hoping when I write my five year lookback in 2028 the same does not happen to this laptop!
I especially like the pale rose gold trim. Why? When you go to a public place such as an airport, your laptop does not look like everyone else’s.
We also want to carry this laptop around. So another benefit is that it’s one of the lightest laptops around, for its size. Again, a touchscreen would have been heavier.
Of course the Aero contains microphone, built-in speakers, but no ethernet port (I’m a little leery about that). Only two USB ports, plus a USB-C port and full-sized hdmi port.
One usage beef I have is that it supposedly has a back-lit keyboard, but I’ve never seen it turn on.
My company has a coupon code for a roughly four percent discount – not huge, but every bit helps. Shipping is free. But to get the discount I had to talk to a human being to place the order, which is a good idea anyway for a purchase of this magnitude. She carefully reviewed the order with me multiple times. She commended me on my choice to upgrade to the OLED display, which gave me a good feeling.
Unexpected features
I wasn’t really looking for it, but there it is, a fingerprint scanner(!) in order to do a Windows Hello verification. I did not set it up. I guess it could also do a facial recognition as well (that’s what I use at work for Windows Hello for Business), but I also didn’t try that.
I think there’s a mini stereo output but maybe no microphone input? Of course get a USB microphone and you’re all good…
Price
Price as configured above and with my company coupon code applied was $1080. I think that’s much better than a similarly equipped Surface tablet though I honestly didn’t do any real comparisons since I wanted to go HP from the get-go.
Conclusion
I bought a new HP Pavilion Aero laptop. It’s only been a month but I am very pleased with it so far. I configured it the with upgrades important to me since no off-the-shelf model has adequate storage capacity at the sub $1000 price point where I am.
I recommend this configuration for others. I think it’s really a winning combo. I have – I know this is hard to believe – not been compensated in any way for this glowing review! See my site – no ads? That shows you this is a different kind of web site, the kind that reflects the ideals of the Internet when it was conceived decades ago as an altruistic exchange of ideas, not an overly commercialized hellscape.
Since I saw this laptop was a winner I decided to give it away to a loved one, and now I’m back on that five-year-old HP Pavilion laptop!
