Categories
Consumer Tech Web Site Technologies

Starlink Internet service: a first look

Intro

Many of us were quite enthusiastically awaiting the availability of SpaceX’s Starlink Internet service. On paper it sounded promising. Now first results are in and the reality is far less impressive.

The details

I do not have this service but spoke with someone who does. He lives in Puerto Rico where the broadband option are limited. There’s the local cable company, then maybe some boutique services where you use microwave dishes, and this year, finally, Starlink. He had just a couple users on it. I think the net results are that it basically works, but with a big caveat. It sucks for real-time communication. And that’s precisely what he needed it for.

So you know when you’re streaming a movie, that downloads the movie in six(?) second chunks, so it’s a bit robust in the face of brief outages. But when you’re doing web conferencing an outage is very noticeable. And that’s what they experienced, time and again. Brief outages that interrupted their real-time applications. Perhaps lasting for a few seconds, but enough to spoil the broadcast.

Then one night, knowing their cable provider, Liberty, was out, they tested it again. It seemed fine at night. But during the day next day it failed in the same way – brief, disruptive outages.

Maybe some of it is due to holes in the satellite coverage and will get better as the fleet fills out. We’re not sure at this point.

And, yes, the dish was placed in a place where the app showed something like 98% visibility to the satellites in the sky.

Some interesting screenshots of what a Starlink IP looks like in Puerto Rico
ipinfo.io
ipchicken.com
speedtest results

That speedtest looks quite good to me!

Results of Starlink app for this actual user in Puerto Rico
Running PING to google.com shows a single dropped packet
A few words from the actual user

Liberty Cable is not working even after a change of the cable modem. So this past two weeks his household has been exclusively using Starlink. In his own words:

“The single most important thing to consider when using Starlink is how obstructed your northern facing view of the sky is. I am attaching a screenshot from my Starlink app. The red shows the obstructed area. My placement is 2.5% obstructed but I still get an interruption every 4 minutes the app says. In reality it might be every 20 minutes for a few seconds. 

“While my Liberty service has been out Starlink has been a life saver for us. A second user can still do her video calls but it will freeze during those 20-ish minute intervals for a few seconds. It’s not the end of the world for her but not totally idea.


“For me, my VPN will disconnect for those same few seconds and then will reconnect. If I’m entering a trade that can be a crucial few seconds while my vpn and software reconnects, but it’s workable. 


“The Starlink app is free and available for everyone to use. I would suggest that anyone who is interested in the service to first download the app and scan the sky where they think they have the freest point of view north. They will only get purely uninterrupted service if the app registers 100% obstacle-free view. The properties that are most suited for Starlink are the ones at the top of a hill, with a field, or a roof taller than the surrounding trees, especially the trees to the north. An obstructed view like mine is perfectly suitable for streaming movies as they tend to buffer a few minutes in advance, downloading files, and surfing the web. Without a completely obstructed view of the sky, video calls, VPN connection, Remote desktop connection, and online gaming will be interrupted in a frustrating manner. 


“I am also attaching a text file of my results from running a  ping -t to google.com from my Starlink connection. This test ran for about 25 minutes. The request time outs are the times when the Starlink satellite was not able to connect due to my obstructions. However, also notice that during these times it only lost 1 ping and was immediately able to reconnect. Again, somewhat frustrating but it’s a usable product. 

Conclusion

Don’t throw away your cable modem*. In general as of this writing in June 2022, Starlink is a good solution for those working from home, but be prepared to be bumped every 20 minutes or so from your video conferencing or other real time uses. And of course it’s good for surfing the web or on-demand streaming.

I don’t cnosider this the final word however. There’s still hope. I’ll update this post if the quality of service ever improves.

*Unless you’re one of the many whose cable modem service isn’t all that great to begin with.

References and related

https://en.wikipedia.org/wiki/Starlink

Categories
Consumer Tech Web Site Technologies

Consumer tech: Edge new tab in Chinese

Intro

If you’ve ever had the misfortune to access a web site in China in your Edge browser, you may find that from that point onwards all your new tab pages display in Chinese despite of your best efforts to eradicate it.

The details

I was in that same boar until today. There are many bad leads out there on the Internet. In fact I never did find the solution on the Internet. I got it from a colleague.

You click on the three dots, go to Settings and search for reset.

Do the Reset. It is a little disruptive, as i have found. It does not delete everything, but it certainly resets some things. As soon as that’s done you will no longer have new tab pages be in Chinese.

Categories
Consumer Interest Consumer Tech

Consumer tech: Android phone tip of the day

Intro

My wife was stuck while using the WW app on her Samsung A51 smartphone. She needed to lookup a nearby “studio.” We’ve all seen these forms – you enter a zipcode and up pops their nearby locations. But in this case there was a problem. No keyboard was popping up! Instead the bottom of the screen below the search field was filled with some blather which we could not get rid of to reveal the presumably hidden software keyboard.

The (kludge) solution

Please note that I am a specialist in doing things the wrong way that manage to get it done. I noticed the field still permitted long touch, and hence paste (from the clipboard). So I told her to enter the zipcode into another app such as Evernote, copy that text into the clipboard, return to the WW app and paste it into that field.

And do you know – that actually worked!

Conclusion

So if you’re in a jam and just need to fill out a field on your Android phone but your software keyboard isn’t appearing, a way out is to paste the desired content from another app such as Evernote or Onenote.

References and related

I use this one all the time: find my phone – no BS apps, just the straight-up Google URL for this built-in service.

Categories
Admin Linux Network Technologies Web Site Technologies

The IT Detective Agency: This site can’t be reached

Intro

It’s been awhile since I’ve had the opportunity to relatean IT mystery. After awhile they are repates of what’s already happened in the past, or it’s too complex to relate, or I was only peripherally involved. But today I came across a good one. It falls into the never been seen before category.

The details

A web server behind my web application firewall became unreachable. In the browser they get a message This site can’t be reached. The app owners came to me looking for input. I checked the WAF and it was fine. The virtual server was looking healthy. So I took a packet trace, something to this effect:

$ tcpdump -nni 0.0 host 192.168.2.124

14:00:45.180349 IP 192.68.1.13.42045 > 192.68.2.124.443: Flags [S], seq 1106553901, win 23360, options [mss 1460,sackOK,TS val 3715803515 ecr 0], length 0 out slot1/tmm3 lis=/Common/was90extqa.drjohn.com.app/was90extqa.drjohn.com_vs port=0.53 trunk=
14:00:45.181081 IP 192.68.2.124 > 192.68.1.13: ICMP host 192.68.2.124 unreachable - admin prohibited filter, length 64 in slot1/tmm2 lis= port=0.47 trunk=
14:00:45.181239 IP 192.68.1.13.42045 > 192.68.2.124.443: Flags [R.], seq 1106553902, ack 0, win 0, length 0 out slot1/tmm3 lis=/Common/was90extqa.drjohn.com.app/was9
0extqa.drjohn.com port=0.53 trunk=

I’ve never seen that before, ICMP host 192.68.2.124 unreachable – admin prohibited filter. But I know ICMP can be used to relay out-of-band routing information on occasion, though I do not see it often. I suspect it is a BAD THING and forces the connection to be shut down. Question is, where was it coming from?

The communication is via a firewall so I check the firewall. I see a little more traffic so I narrow the filter down:

$ tcpdump -nni 0.0 host 192.168.2.124 host 443

And then I only see the initial SYN packet followed by the RST – from the same source IP! So since I didn’t see the bad ICMP packet on the firewall, but I do see it on the WAF, I preliminarily conclude the problem exists on the WAF.

Rookie mistake! Did you fall for it? So very, very often, in the heat of debugging, we invent some unit test which we’ve never done before, and we have to be satisified with the uncertainty in the testing method and hope to find a control test somehow, somewhere to validate our new unit test.

Although I very commonly do compound filters, in this case it makes no sense, as I realized a few minutes later. My port 443 filter would of course exclude logging the bad ICMP packets because ICMP does not use tcp port 443! So I took that out and re-run it. Yup. bad ICMP packet still present on the firewall, even on the interface of the firewall directly connected to the server.

So at this point I have proven to my satisfaction that this packet, which is ruining the communication, really comes frmo the server.

What the server guys say

Server support is outsourced. The vendor replies

As far as the patching activities go , there is nothing changed to the server except distro upgrading from 15.2 to 15.3. no other configs were changed. This is a regular procedure executed on almost all 15.2 servers in your environment. No other complains received so far…

So, the usual It’s not us, look somewhere else. So the app owner asks me for further guidance. I find it’s helpful to create a test that will convince the other party of the error with their service. And what is one test I would have liked to have seen but didn’t cnoduct? A packet trace on the server itself. So I write

I would suggest they (or you) do a packet trace on the server itself to prove to themselves that this server is not behaving ini an acceptable way, network-wise, if they see that same ICMP packet which I see.

The resolution

This kind of thing can often come to a stand-off, or many days can be wasted as an issue gets escalated to sufficiently competent technicians. In this case it wasn’t so bad. A few hours later the app owners write and mention that the home-grown local firewall seemed suspect to them. They dsabled it and this traffic began to work.

They are reaching out to the vendor to understand what may have happened.

Case: closed!

Conclusion

An IT mystery was resolved today – something we’ve never seen but were able to diagnose and overcome. We learned it’s sometimes a good thing to throw a wider net when seeing unexpected reset packets because maybe just maybe there is an ICMP host unreachable packet somewhere in the mix.

Most firewalls would just drop packets and you wait for a timeout. But this was a homegrown firewall running on SLES 15. So it abides by its own ways of working, I guess. So because of the RST, your connection closes quickly, not timing out as with a normal network firewall.

As always, one has to maintain an open mind as to the true source of an issue. What was working yesterday does not today. No one admits to changing anything. Finding clever ad hoc unit tests is the way forward, and don’t forget to validate the ad hoc test. We use curl a lot for these kinds of tests. A browser is a complex beast and too much of a black box.

Categories
Network Technologies

How to force snmpwalk to convert strings to numeric OIDs

Intro

It’s a little hard to find this information on the Internet, so I’m amplifying the correct answer here by using my blog.

The details

I’m not super-competent with MIBs and such, but I manage for my purposes with my basic understanding. I have access to an F5 bigip with various IPSEC tunnels on it. I want to use Zabbix to check the status of those tunnels. So I do an SMPwalk like this:

snmpwalk -v3 … -c public 127.0.0.1 F5-BIGIP-SYSTEM-MIB::sysIpsecSpdStatTunnelState

which produces output like this line:

F5-BIGIP-SYSTEM-MIB::sysIpsecSpdStatTunnelState.”/Common/tunnel-01″.58401 = STRING: up

But I cannot take that as it is and use it in an snmpget like this:

snmpget -v3 … -c public 127.0.0.1 F5-BIGIP-SYSTEM-MIB::sysIpsecSpdStatTunnelState.”/Common/tunnel-01″.58401

That produces an error like this:

Unknown Object Identifier (Index out of range: /Common/tunnel-01 (sysIpsecSpdStatTrafficSelectorName))

So we need to convert the string into a numeric OID. But how?

The answer

Use the -On switch as an additional argument in your snmpwalk.

You will get a scary long OID, but it will at least be numeric.

Gonig further

You can then deconstruct the response and reconstitute the section at the beginning with a nice name. For my F5 example

.1.3.6.1.4.1.3375.2.1.2.17.1.3.1.14

becomes

F5-BIGIP-SYSTEM-MIB::sysIpsecSpdStatBytes

I think. Then preserve the following digits as is.

Conclusion

We have shown how to output a numeric OID from an snmpwalk. This, specifically, is sueful in converting a string embedded in the output into a numeric OID, which may then be used by other SNMP applications such as Zabbix which may or may not have the MIB file loaded. The secret is simply to use the -On switch in snmpwalk.

References and related

My Zabbix FAQ – questions you wish they had answered, can be very helpful

Categories
Admin Network Technologies TCP/IP

Verizon Airspeed Hotspot uses ipv6 and interferes with VPN client Global Protect

Intro

The headline says it all. I got my shiny brand new Verizon hotspot from Walmart. I managed to activate it and add it to my Verizon account (not super easy, but after a few stumbles it did work.) I tried it out my home PC – works fine. I tried it out on my work PC. No good. My Global Protect connection was unstable. It connects for about a minute, then disconnects, then connects, etc. Basically unusable.

The details

I have heard of possible problem with the GP client (version 5.2.11) and IPv6. So I looked to see if this hotspot could be handing out IPv6 info. Yes. It is. But is that really making a difference? I concocted a simple test. I disabled IPv6 on my Wi-Fi adapter, then re-tested the GP client. The connection was smooth as glass! No disconnects!

Disable ipv6 on your Wi-Fi adapter

Bring up a powershell as administrator. Then:

get-netadapterbinding -componentid ms_tcpip6

will show you the current state of ipv6 on your adapters.

disable-netadapterbinding -Name “Wi-Fi” -ComponentID ms_tcpip6

will disable ipv6 on your Wi-Fi. And

enable-netadapterbinding -Name “Wi-Fi” -ComponentID ms_tcpip6

will re-enable it.

ipconfig /all output

For the record, here are some interesting bits from running ipconfig /all:

Wireless LAN adapter Wi-Fi:

Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) Dual Band Wireless-AC 8265
Physical Address. . . . . . . . . : 0C-BD-94-98-11-5B
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
Temporary IPv6 Address. . . . . . : 2600:1001:b004:2b78:8ab:145c:d014:2edd(Deprecated)
IPv6 Address. . . . . . . . . . . : 2600:1001:b004:2b78:2cc0:71b0:7f1e:a973(Deprecated)
Link-local IPv6 Address . . . . . : fe80::2cc0:71b0:7f1e:a973%30(Preferred)
IPv4 Address. . . . . . . . . . . : 192.168.1.103(Preferred)

Subnet Mask . . . . . . . . . . . : 255.255.255.0
Lease Obtained. . . . . . . . . . : Thursday, April 21, 2022 4:54:04 PM
Lease Expires . . . . . . . . . . : Friday, April 22, 2022 4:54:04 AM
Default Gateway . . . . . . . . . : 192.168.1.1
DHCP Server . . . . . . . . . . . : 192.168.1.1
DHCPv6 IAID . . . . . . . . . . . : 302832932
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-28-89-F6-8E-B0-5C-DA-E6-09-0A
DNS Servers . . . . . . . . . . . : fe80::50ae:caff:fea8:1dbc%30
192.168.1.1
NetBIOS over Tcpip. . . . . . . . : Enabled

But, having done all that, I can only occasionally connect to GP. It seems to work slightly better at night. ipv6 does not seem to be the sole hiccup. No idea what the recipe for reliable success is. If I ever learn it I will publish it. Meanwhile, my phone’s hotspot, also VErizon, also handing out ipv6 info, usually permits me to connect to GP. It’s hard to see the difference.

Conclusion

The Verizon Airspeed Hotspot sends out a mix of IPv6 and IPv4 info to dhcp clients. Palo Alto Networks’ Global Protect client does not play well with that setup and wil not have a stable connection.

I do not think there is a way to disable IPv6 on the hotspot. However, for those with admin access it can be disabled on a Windows PC. And then GP will work just fine. Or not.

Oh, and by the way, otherwise the Airspeed works well and is an adequate solution where you need a good reliable hotspot. Well, in fact, don’t expect reliability like you have from a wired connection. After a couple hours, all users just got dropped for no apparent reason whatsoever.

Categories
Consumer Tech

Consumer Tech: Setting up an Echo Dot while staying at a hotel

Intro

I’m sure many people have faced this. You like to query your Echo Dot for basic information and you’d like to take it with you and work in your hotel room so you can get the local weather, etc. But then you are foiled by the setup process until you finally give up after trying to discover it as a new device numerous times.

I was in that exact situation last week. Finally I thought of something that broke me out of the discover device loop, so I wanted to share that.

The details

A hotel will typically offer Internet, but through a “captive portal.” This means, technically, the password they provide you at the front desk is not the SSID password, but a password to their portal’s web site. So it is technically not part of the WiFi information. That’s what makes it so hard for dumb devices like the Echo Dot to get past that page.

Normally you go to your room after checking in, and you perhaps set your phone to use their WiFi after dutifully logging in to the portal page. Then you get around to setting up your Echo device. I could not manage to finish the Echo Dot configuration under those conditions last week, though I tried many times over, including manual setup.

What I realized, though, is that it might help to start clean. So I “forgot” the hotel’s SSID (WiFi) on my phone. Then I was only using 4G communication. Then I went through the Device Setup in the Alexa App on my phone. At some point you get asked which WiFi to connect to, and you get redirected to the captive portal, where you put in the information. In other words I stumbled my way through it. But then it did work.

So I’d say the upshot is to configure the Echo Dot first before setting up your phone to use the hotel’s WiFi. I’m not sure it will always work, but at least it did work once! But if you did things in the wrong order go to your WiFi settings and “forget” the hotel’s WiFi – it may have the same effect.

2nd test – successful

Since I’m supposed to be the expert I had the opportunity to try my own technique at another hotel. It was about 10 minutes of stumbling, but then it did work in the end. Maybe my tip helped, maybe not.

As I do not have the straight recipe, you have to be willing to try different things. Pay special attention to whether or not your setup device is connected to the Dot’s WiFi (amazon-dkd or something like that) or not. That’s generally a good thing. Then you have it (the Dot) search for available WiFi networks. I guess. It’s confusing. You’ll probably need to put your Dot into setup mode mulitple times. Esimated time to stumble your way through it: 10 minutes. You can decide if it’s worth it or not.

Other ways

I’m thinking about mac address emulation, i.e., temporarily emulate the MAC address of the Echo device with a more amenable device, and do the poral login. But I haven’t had time to research it. I will post it here if I ever figure that out.

Categories
Linux

Install WSL2 for Windows 10 Home Edition: not as easy as they say, but not impossible either, and definitely worth it

Intro

I installed WSL2 on my work laptop a couple weeks ago. It didn’t go terribly smoothly but now that I have it, I love it. I had been using a Cygwin environment, but I fear that is looking a little long in the tooth. WSL2 is fast to start up. But the main contrast is that while Cygwin is an emulator, WSL2 is a true hypervisor so you get a full-fledged linux VM, right on your PC. Of course this was always possible with products like VirtualBox or whatnot, but Microsoft has sort of built in this capability with newer versions of Windows 10, so there’s no mussing with external software any longer.

But at work I have Windows 10 Professional, of course. What about at home where I have Windows 10 Home Edition like most of us? My understanding is that you could not run a hypervisor with Windows 10 Home Edition. And I was probably right, until recently. But now you can. I know because I just managed it tonight.

None of the tutorials out there were exactly right, but they all contained pieces of the truth. So my contribution is to add weight to the correct steps you’ll need to take. Unfortunately I only get to do it once so my notes aren’t the best. Still, I may be able to spare you some pitfalls.

Why you should want WSL2

If you love linux command-line, then I would say this is a must-have.

What doesn’t work

You’ll see suggestions to fire up powershell and simply run

wsl –install

Chances are about 95% that that won’t work if you are reading this article – would that it would be so simple.

Instead, do this

Open a powershell window as administrator. To do that type powershell in the start menu, and look around at all the options. Pick out the one that mentions Run as administrator.

Running Powershell as Administrator

Then enter this command into the PS window.

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart

Then this.

dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Then this.

wsl --set-default-version 2

You need to update your kernel. Download this WSL2 kernel update file and install it: https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_x64.msi

A reboot at this point is probably a good idea.

Now you need to get yourself a linux distro to install.

There are certain wsl commands you can issue which will helpfully give you the URL to the Debian distro: You put the URL into the browser and it redirects you to the MS Store. But I forget what that is. perhaps wsl -d Debian. But I suppose you can simply go to the MS Store directly and search for Debian and install it.

Enhance your experience with Windows Terminal

One good suggestion out of Windows Central is to use Windows Terminal. At least it looks good. I haven’t had time to try it myself. I normally just fire up a CMD window and type wsl. My Debian starts immediately and I have a satisfactory command-line environment. But working with multiple windows will be nice so I have to check it out.

Just look for Windows Terminal in the MS Store.

Windows Central suggestions

A web site called Windows Central has a pretty good stepwise guide. But their advertising is so obnoxious, I’m afraid to accidentally touch any part of the page for fear of getting sent to one of their many advertisers. Even still it probably happened about five times. So I won’t make the link to them too prominent. And, anyway, their guide is a little oversimplified.

My equipment

I have a four-year old HP Pavilion laptop running Windows 10 2021 H2 if I remember correctly. It has solid state drives so it’s not too slow, and it boots pretty quickly.

BIOS – basically impossible to get into these days

I’m sure people who do this for a living will disagree, but for ordinary people it’s basically impossible to disrupt the boot process to modify the BIOS settings. And you may need to do that. In fact that was the hardest thing of all for me. Pressing F10 or delete or Escape or F2 – and does that mean hold the FN key down first?? No one explains that, and I don’t have patience to watch a YouTube video. But after trying a bunch of combinations and booting a bunch of times, and never getting into the BIOS settings, I was really glad to learn Windows 10 offers an alternate way! And it works…

Access BIOS settings from Windows 10

Very briefly, the steps are:

Windows Settings > Update & Security > Recovery > Restart Now > Advanced Startup -> Restart Now > Reboot > select Troubleshoot > Advanced Options > UEFI Firmware Settings > (BIOS menu) enable virtualization > Save.

To see the details, go to this HP article: How to Enter BIOS Setup on Windows PCs | HP® Tech Takes

Why you may need to alter the BIOS settings

Well, on my laptop my installation of Debian kept failing with this error. Error: 0x80370102 The virtual machine could not be started because a required feature is not installed. I read on a Microsoft site that could be because the ability to run virtual servers was not enabled in the BIOS. And, yes, that turned out to be absolutely true. It was disabled. So I enabled it and bam, the Debian install started asking me for a username and password, and I was running a Debian VM!

To be fleshed out as my time permits…

But, I love my Debian linux. It’s just like Raspberry Pi OS Lite. I just install packages as I need them: python, pip, curl, bind9-dnsutils, ssh, etc.

Operating inbound TCP services

After the initial thrill wears off, you realize you may need practical things that you have on your Raspberry Pi such as an ssh server or a web server. I believe this will be possible. Still working on it. After installing ssh you can fire it up:

$ sudo service ssh start

This post describes some of those service commands which you have under a WSL linux install: [3 Fixes] System Has Not Been Booted With Systemd as Init System (partitionwizard.com)

If you ignore that article you may see this error! System has not been booted with systemd as init system (PID 1). Can’t operate.

Back to your ssh server. Now you can already connect to it from the Windows system itself, e.g., from a CMD window:

C:\Users\me> ssh user@localhost

user is the Debian user you set during initial setup. So, anyway, that works and that’s cool. But you’re still locked out from the outside.

This helpful Microsoft article discusses networking for WSL2. Apparently it is still evolving and so it’s a bit primitive right now: Accessing network applications with WSL | Microsoft Docs

From a CMD Window launched as administrator:

netsh interface portproxy add v4tov4 listenport=22 listenaddress=0.0.0.0 connectport=22 connectaddress=172.22.167.12

But this does not work in my case. Firewall thing, I’m sure. Yes! for me, where I also run Mcafee, I needed to go to their firewall settings > Ports and system services. Then I had to add a service for TCP port 22 – the ssh default port. Then it began to work and my RPi could ssh and sftp to my Debian VM! sftp kind of hanged a bit. Have to see how bad that is.

References and related

That obnoxious Windows Central article I mentioned above with a lot of the WSL2 installation information. It’s a veritable minefield of links to irrelevant stuff, so you’ve been warned: How to install Linux WSL2 on Windows 10 and Windows 11 | Windows Central

WSL2 kernel update.

Seeing Error: 0x80370102? Try Troubleshooting Windows Subsystem for Linux | Microsoft Docs A whole host of other WSL2 errors are addressed in this article as well.

This article purports to be for servers, but I think it’s applicable to PCs as well. It gets pretty technical. System requirements for Hyper-V on Windows Server | Microsoft Docs

About starting system services such as the ssh daemon: [3 Fixes] System Has Not Been Booted With Systemd as Init System (partitionwizard.com)

A good overview of WSL2 networking: Accessing network applications with WSL | Microsoft Docs

How to Enter BIOS Setup on Windows PCs | HP® Tech Takes

Categories
Python

My favorite Python tips

Intro

I mostly transitioned from perl to python programming. I resisted for the longest time, but now I would never go back. I realized something. I was never really good at Perl. What I was good at were the regular expressions. So Perl for me was just a framework to write RegExes. But Perl code looks ugly with all those semicolons. Python is neater and therefore more legible at a glance. Python also has more libraries to pick from. Data structures in Perl were just plain ugly. I never mastered the syntax. I think I got it now in Python, which is a huge timesaver – I don’t have to hit the books every time I have a complex data structure.

I will probably find these tips useful and will improve upon them as I find better ways to do things. It’s mostly for my own reference, but maybe someone else will find them handy. I use maybe 5% of python? As I need additional things I’ll throw thm in here.

What is this object?

Say you have an object <obj> and want to know what type it is because you’re a little lost. Do:

print(<obj>.__class__)

Check if this key exists in this dict

if “model” in thisdict:

Remove key from dict

if “model” in thisdict: del thisdict[“model”]

Copy (assign) one dict to another

Do not use dict2 = dict1! That is accepted, syntactically, but won’t work as you expect. Instead do this:

dict2 = dict1.copy()

Multiple assignments in on line

a,b,c = “hi”,23,”there”

Key and value from a single line

for itemid,val in itemvals.items():

Formatting

I guess it is pretty common to use a space-based (not tab) indent of four spaces for each subsequent code block.

Initializing lists and dicts

alist = []

adict = {}

Test for an empty list

if not alist: print(“empty list”)

Merge two lists together

for elmnt in list2: list1.append(elmnt)

Conditional operators

if a == b: print(“equals”)

Ternary operator

I don’t think is well-developed in Python and shouldn’t be used (my opinion).

++ operator? Doesn’t exist. += and its ilk does, however.

Boolean variables + multiple assignment example

a, b=True, False

if a==b: print(“equals”)

if a: print(“a is true”)

Reduce number of lines in the program

for n in range(12): colors[n] = ‘red’

if not mykey in mydict: mydict[mykey] = []

Printing stuff while developing

print(“mydict”,mydict,flush=True)

Python figures it out how to print your object, whatever type it is, which is cool. That flush=True is needed if you want to see our output which you’ve redirected to a file right away! Otherwise it gets buffered.

Reading and writing files – prettyify.py
                    

import requests, json, sys, os
import sys,json

from pathlib import Path
aql_file = sys.argv[1]
aql_path = Path(aql_file)
json_file = str(aql_path.with_suffix('.json'))

# Script path
dir_path = os.path.dirname(os.path.realpath(__file__))
dir_path_files = dir_path + "/files/"

# make ugly json file prettier    
# this is kind of a different example, mixed in there
file = sys.argv[1]
f = open(file)
# return json obj as dict
fjson = json.load(f)
nicer = json.dumps(fjson,indent=4)
print(nicer,flush=True)
# back to original example
f = open(dir_path_files + json_file,'w+')
f.write(body)
f.close()

Reading in command-line arguments

See Reading and Writing files example.

Command line tips

The command line is your friend and should be used for little tests. Moreover, you can print an object without even a print statement.

>>>a =[1,’hi’,3]

>>>a

Going from byte object to string

s_b = b’string’

s = s_b.decode(‘utf-8’)

Test if object is a string

if type(thisobject) == str: print(“It is a string”)

Python as a calculator

I always used python command line as a calculator, even ebfore I knew the language syntax! It’s very handy.

>>> 5 + 6/23

Breaking out of a for loop

Use the continue statement after testing a condition.

Executing shell commands

import os

os.system(“ls -l”)

But, to capture the output, you can use the subprocess package.

Accessing environment variables

os.environ.get(‘ENV_TOKEN’)

Handling glob (wildcards) in your shell command

import glob

for query_results_file in glob.glob(os.path.join(dir_path_files,OSpattern)): print(“query_results_file”,query_results_file)

But, if you want the results in the same order as the shell gives, put a sorted() around that. Otherwise the results come out randomly.

JSON tips

Python is great for reading and writing JSON files.

# Load inventory file

with open(dir_path_files + inventory_file) as inventory_file:
inventory_json = json.load(inventory_file)

sitenoted={'gmtOffset':jdict["gmtOffset"],'timezoneId':jdict["timezoneId"]}

# update inventory with custom field Site Notes – put GMT

sitenote=json.dumps(sitenoted)
print("sitenote",sitenote)

Test for null in JSON value

You may see “mykey”:null in your json values. How to test for that?

if my_dict[mykey] == None: continue

RegExes

Although supported in Python, seems kind of ugly. Many RegExes will need to prefaced with r (raw), or else you’ll get yourself into trouble, as in

import re

r'[a-z]{4}.\s*\w(abc|def)’

if re.search(‘EGW-‘,locale): continue

b = re.sub(‘ ‘,’-‘,locale)

Function arguments: are they passed by reference or by value?

By reference for complex objects like a dict (not sure about a list), but by value for a simple object like a Boolean! I got burned by this. I wanted a Boolean to persist across a function call. In the end I simple stuffed it into a dict! And that worked. But python doesn’t use that terminology. But it means you can pass your complex data structure, say a list of dicts of dicts, start appending to the list in your function, and the calling program has access to the additional members.

Excel files

I’ve been using the package openpyxl quite successfully to read and write Excel files.

Math

numpy seems to be the go-to package.

Python and self-signed certificates, or certificates issued by private CAs

I updated this blog article to help address that: Adding private root CAs in Redhat or SLES or Debian.

Write it with style

Use flake8 to see if your python program conforms to the best practice style. In CentOS I installed flake8 using pip while in Debian linux I installed it using apt-get install flake8.

Can I modify a Python script while its running?

Sure. No worries. It is safe to do so.

Create virtual environment for portability

I like to call my virtual environment venv.

virtualenv venv

Use this virtual environment

source ./venv/bin/activate

List all the packages in this virtual environment

Good portable development style would have you install the minimal set of packages in your virtual environment and then build a requirements.txt file:

pip3 freeze > requirements.txt

Leave this virtual environment

deactivate

Conclusion

I’ve written down some of my favorite tips for using python effectively.

References and related

Adding private root CAs in Redhat or SLES or Debian.

Categories
Consumer Tech

Book Review: extraterrestrial The First Sign of Intelligent Life Beyond Earth

Intro

I don’t normally do book reviews but since someone wanted to get my thoughts on this one, I thought I would share with a broader community.

Source material

Extraterrestrial: The First Sign of Intelligent Life Beyond Earth: Loeb, Avi: 9780358278146: Amazon.com: Books

Let’s get into it

This book is mainly about what to make of the very unusual sighting of an object which came from outside our solar system in 2017, dubbed ‘Oumuamua.

This book is written at a very simple level – perhaps fifth grade or sixth grade? So I guess it’d be a great addition to a middle school library. I was aching for some more details.

I’m actually ready and willing to ‘believe.” For me the main thing was the paucity of facts presented.

The biggest failing of the author is to fail to be so guarded! Professor Avi, you are uniquely qualified to spin any plausible story about this object. Surely you have thought of many origin stories for this thing. Why not share them? This is a book, after all, not a scientific publication. We won’t peer review you for daring to create a plausible backstory for this object.

For me to make my points I have to give some background so everyone sees what I am seeing.

Professor Avi has this super cool lightsail project. Something like 10 GigaWatt lasers are required to send a super slender light sail up to 1/10th the speed of light.

An aside. Wouldn’t a multi Gigawatt blast create a plasma out of the atmosphere, thereby transferring its power to the air rather than the target lightsail? Maybe you overcome that by spreading the laser over a wide area. or using micosecond blasts. Not sure. And what about the reflecting rays? Can they be adequately disbursed to avoid singeing the earth?

But I digress. Sending any macro object to any fraction of the speed of light is wondrous. And dangerous. By conservation of energy I have to assume the thing at that speed would have 10? 100? gigajoules of energy (I will do the math later). Imagine the consequences of an earth-like planet being “visited” by one of those things. Either cataclysmic, or at least terrifying to those lifeforms. Yes, I know the odds of collision are infinitesimal. But they are not zero. And no way can these things be aimed with such precision to avoid that scenario altogether. Not to mention the idea is to send thousands or millions of them out. Welcome to mankind, extraterrestrials, we like to announce our entrance with a bang!

For instance, ‘Oumuamua, though it comes from outside our solar system, is at the local system of rest of the nearby stars. So that is pretty remarkable, and it means its speed is not anywhere close to a fraction of the speed of light, unlike Prof Avi’s lightsails. Why is that? This particular civilization felt they could wait around thousands of years?? Or did it start out at a fraction of the speed of light and then get decelerated as it neared its target? And if so, by what force?

And when I learned it sped up as it zoomed around the sun, I immediately thought of the analogy of our space missions which sometimes use giant planets like Jupiter to pick up speed and slingshot away faster than they had been going. Was ‘Oumuamua purposely aimed near to our sun for its boomerang effect? But it was tumbling like every eight hours. Should lightsails do that? Or does that show if it was a lightsail, it was no longer fit for purpose – inactive space junk? ‘Oumuamua trajectory deviated in the manner a lightsail might. But if it was slowly tumbling, how is that compatible with that statement? Aren’t lightsails only good for catching rays from one orientation? I actually don’t know but that’s what I would naively assume.

And it came from somewhere. if you reverse its trajectory, where did it come from? Was it an area with an inhabitable exoplanet?? Is that area receiving heightened scrutiny from SETI and company??

Going back to these lightsail things travelling at a good fraction of the speed of light, if one were to whip past us, would we even have a chance to see it? I believe it would be effectively invisible to us. How would they fare when colliding with space dust?

A manufactured lightsail would have great symmetry. The brightness profile was not a nice sine function, though close. What is the lightsail shape we can assume given the observed brightness profile fluctuation? A partially destroyed lightsail, perhaps? Where is the artist’s rendering of that?

So you see my point now? Don’t make me speculate. You’re the expert. Your speculations will be grounded in better science than anything I can dream up. So I guess Prof Avi, despite being a maverick in many ways and bucking current scientific thinking in promoting this as a thing created by alien life, reverted to the usual scientist’s conservatism in not making unprovable statements. And we are worse off because of it.

The great filter

And this term bothers me. It reads poorly. The first I heard it was from a fellow reader, and, even though I was familiar with the concept, I had to ask for a definition. A good term of art is self-explanatory. This is not a good term. Advanced civilizations probably last for only a few hundred or at best a few thousand years before they self-destruct. Looking at ourselves, we’re probably only going to get a few hundred the way we’re going. And this is supposed to be the great filter or something? I don’t have a better term, but far more clever people could come up with one I am sure. Like inevitable self-destruction, except something with more of a ring to it.

So I was asked, if this comes from an advanced civilization, is this a cause for hope, or a cause for despair? To argue the despair first, we got space junk from an advanced civilization. Probably they died out and we are left to do astroarchaeology on their junk. Not so great. But I am more hopeful. It’s incredibly difficult to target another star, they managed to do it. Maybe their lightsail had an accident or something. No worries because they sent out millions more like Prof Avi proposes to do. And, the main thing, we overlapped with them! We were advanced enough to detect another’s technology. Mostly because of the self-destruction tendencies, and the randomness of when advanced life forms, we’re not going to have any overlap with the vast majority of our fellow aliens. Their time in the sun was either way in the distant past, or will occur way in the future. That we overlapped in any way at all, probably means there are very, very many advanced civilizations, even in our stellar neighborhood, such that we had decent odds of intercepting and overlapping with one. And that gives me awe and excitement to learn about this advanced life. The hope comes from the viewpoint that these beings aren’t threatening to us. I have a naive belief that they would be trained in cultural sensitivities a la Star Trek The Next Generation or something as opposed to Independence Day, and decide not to wipe us out, nor to alter our technology (much), but more to observe us from afar. So on balance this encounter makes me hopeful.

Any insight if this was a civilization which reached the singularity? I.e., where it transferred its organic intelligence to a program in silicon or some other infinitely long-lasting, purpose-built medium??

So yes I am convinced the simplest explanation is the best one, and prof Avi’s hypothesis is by far the simplest. It raises a few questions which I would have preferred answered. And I was dying for more speculations. The speculations of an insider is worth 100 times the speculations of an outsider such as myself who doesn’t know what they don’t know!!

Philosophy

Prof Avi devotes a lot of time to philosophy. That’s all good. I didn’t learn too much from it, but I suppose others could find it useful. I don’t have an issue with it.

Resources

If I were really responsible, I’d do research, or at least read the darn book reviews on Amazon or get an answer to some of my questions on Quora or something. Probably a lot of my questions are addressed. But my time is not infinite and I’m not trying to impress anyone. As I learn more in the course of my ad hoc reading I will revise this blog with better information. And one more thing about my personal philosophy, I am writing this based solely on self-reflection and readings I’ve absorbed from years ago. I consider active research “cheating” in this regard, and I will inevitably be swayed and biased by the first educated opinion I come across.

Before reading this book, I was aware of this object from 2017 and that it was special, just based on my general reading of science news. It was only from this book that I realized how compelling the extraterrestrial case was.

Prof Avi took a few facts and made a book out of it. He should probably create a fictional but plausible back story for this object and make another book that addresses some of these basic questions.

February 2022 update

They have just discovered signs of a third exoplanet around Proxima Centauri, our nearest neighbor in the galaxy! That’s exciting. The place is only four light years away. Maybe a lightsail craft will visit it within our lifetimes.

Reference and related

Extraterrestrial: The First Sign of Intelligent Life Beyond Earth: Loeb, Avi: 9780358278146: Amazon.com: Books

My cheese grater image of the 2017 eclipse.