Categories
Raspberry Pi SLES Web Site Technologies

Pi-hole: it’s as easy as pi to get rid of your advertisements

Intro
I learned about pi-hole from Bloomberg Businessweek of all places. Seems right up my alley – uses Raspberry Pi in your home to get rid of advertisements. Turns out it was too easy and I don’t have much to contribute except my own experiences with it!

The details
When I read about it I got to thinking big picture and wondered what would prevent us from running an enterprise version of this same thing? Well, large enerprises don’t normally run production critical applications like DNS servers (which this is, by the way) on Raspberry Pis, which is not the world’s most stable hardware! But first I had to try it at home just to learn more about the technology.

pi-hole admin screen

I was surprised just how optimized it was for the Raspberry Pi, to the neglect of other systems. So the idea of using an old SLES server is out the window.

But I think I got the essence of the idea. It replaces your DNS server with a custom one that resolves normal queries for web sites the usual way, but for DNS queries that would resolve to an Ad server, it clobbers the DNS and returns its own IP address. Why? So that it can send you a harmless blank image or whatever in place of an Internet ad.

You know those sites that obnoxiously throw up those auto-playing videos? That ain’t gonna happen any more when you run pi-hole.

You have to be a little adept at modifying your home router, but they even have a rough tutorial for that.

Installation
For the record on my Rspberry Pi I only did this:
$ sudo su ‐
$ curl ‐sSL https://install.pi‐hole.net | bash

It prompted me for a few configuration details, but the answers were obvious. I chose Google DNS servers because I have a long and positive history using them.

You can see that it installs a bunch of packages – surprisingly many considering how simple in theory the thing is.

Test it
On your Raspberry Pi do a few test resolutions:

$ dig google.com @localhost # should look like it normally does
$ dig pi.hole # should return the IP of your Raspberry Pi
$ dig adservices.google.com # I gotta check this one. Should return IP address of your Pi

It runs a little web server on your Pi so the Pi acts as adservices.google.com and just serves out some white space instead of the ad you would have gotten.

Linksys router
Another word about the home router DHCP settings. You have the option to enter DNS server. So I put the IP address of my raspberry pi, 192.168.1.119. What I expected is that this is the DNS server that would be directly handed out to the DHCP clients on my home network. But that is not the case. Instead it still hands out itself, 192.168.1.1 as DNS server. But in turn it uses the raspberry PI for its resolution. This through me when I did an ipconfig /all on my Windows 10 and didn’t see the DNS server I expected. But it wa all working. About 10% of my DNS queries were pi-holed (see picture of my admin screen above).

I guess pi-hole is run by fanatics, because it works surprisingly well. Those complex sites still worked, like cnn.com, cnet.com. But they probably load faster without the ads.

Two months check up

I checked back with pihole. I know a DNS server is running. The dashboard is broken – the sections just have spinning circle instead of data. It’s already asking me to upgrade to v 3.3.1. I run pihole -up to do the upgrade.

Another little advantage
I can now ssh to my pi by specifying the host as pi.hole – which I can actually remember!

Idea for enterprise
finally, the essence of the idea probably could be ported over to an enterprise. In my opinion the secret sauce are the lists of domain names to clobber. There are five or six of them. Some have 50,000 entries. So you’d probably need a specialized DNS server rather than the default ISC BIND. I remember running a specialized DNS server like that when I ran Puremessage by Sophos. It was optimized to suck in real-time blacklists and the like. I have to dig through my notes to see what we ran. I’m sure it wasn’t dnsmasq, which is what pi-hole runs on the Raspberry Pi! But with these lists and some string manipulation and a simple web server I’d think it’d be possible to replicate in enterprise environment. I may never get the opportunity, more for lack of time than for lack of ability…

Conclusion
Looking for a rewarding project for your Raspberry Pi? Spare yourself Internet advertisements at home by putting it to work.

References and related
The pi-hole web site: https://pi-hole.net/
Another Raspberry Pi project idea: monitor your cable modem and restart it when it goes south.

Categories
DNS Linux Network Technologies Raspberry Pi Security

Whois information without the pushy hard sell tactics

Intro
Did you ever want to learn about a domain registration but were put off by the hard sell tactics that basically all web-based whois searches subject you to? Me, too. Here’s what you can do.

The details
Linux – so that includes you, Raspberry Pi owners – has a little utility called whois which you can use to get the registrant information of a domain, e.g.,

$ whois johnstechtalk.com

   Domain Name: JOHNSTECHTALK.COM
   Registry Domain ID: 1795918838_DOMAIN_COM-VRSN
   Registrar WHOIS Server: whois.godaddy.com
   Registrar URL: http://www.godaddy.com
   Updated Date: 2017-03-27T00:52:51Z
   Creation Date: 2013-04-23T00:54:17Z
   Registry Expiry Date: 2019-04-23T00:54:17Z
   Registrar: GoDaddy.com, LLC
   Registrar IANA ID: 146
   Registrar Abuse Contact Email: [email protected]
   Registrar Abuse Contact Phone: 480-624-2505
   Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited
   Domain Status: clientRenewProhibited https://icann.org/epp#clientRenewProhibited
   Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
   Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited
   Name Server: NS45.DOMAINCONTROL.COM
   Name Server: NS46.DOMAINCONTROL.COM
   DNSSEC: unsigned
   URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/
>>> Last update of whois database: 2018-04-19T19:59:35Z <<<
...

Admittedly that did not tell us much, but it points us to another whois server we can try, whois.godaddy.com. So try that:

$ whois ‐h whois.godaddy.com johnstechtalk.com

Domain Name: JOHNSTECHTALK.COM
Registry Domain ID: 1795918838_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.godaddy.com
Registrar URL: http://www.godaddy.com
Updated Date: 2017-03-27T00:52:50Z
Creation Date: 2013-04-23T00:54:17Z
Registrar Registration Expiration Date: 2019-04-23T00:54:17Z
Registrar: GoDaddy.com, LLC
Registrar IANA ID: 146
Registrar Abuse Contact Email: [email protected]
Registrar Abuse Contact Phone: +1.4806242505
Domain Status: clientTransferProhibited http://www.icann.org/epp#clientTransferProhibited
Domain Status: clientUpdateProhibited http://www.icann.org/epp#clientUpdateProhibited
Domain Status: clientRenewProhibited http://www.icann.org/epp#clientRenewProhibited
Domain Status: clientDeleteProhibited http://www.icann.org/epp#clientDeleteProhibited
Registry Registrant ID: Not Available From Registry
Registrant Name: ******** ******** (see Notes section below on how to view unmasked data)
Registrant Organization:
Registrant Street: ***** ****
Registrant City: Newton
Registrant State/Province: New Jersey
Registrant Postal Code: 078**
Registrant Country: US
Registrant Phone: +*.**********
Registrant Phone Ext:
Registrant Fax:
Registrant Fax Ext:
Registrant Email: ********@*****.***
Registry Admin ID: Not Available From Registry
Admin Name: ******** ******** (see Notes section below on how to view unmasked data)
...

So now we’re getting somewhere. So GoDaddy tries to force you to their web page an sell you stuff in any case. Not at all surprising for anyone who’s ever been a GoDaddy customer (includes yours truly). Because that’s what they do. But not all registrars do that.

Here’s a real-life example which made me decide this technique should be more broadly disseminated. I searched for information on a domain in Argentina:

$ whois buenosaires.com.ar

This TLD has no whois server, but you can access the whois database at
http://www.nic.ar/

Now if you actually try their suggested whois server, it doesn’t even work:

$ whois ‐h www.nic.ar buenosaires.com.ar

Timeout.

What you can do to find the correct whois server is use iana – Internet Assigned Numbers Authority – namely, this page:

https://www.iana.org/domains/root/db

So for Argentina I clicked on .ar (I expected to find a separate listing for .com.ar but that was not the case), leading to the page:

See it? At the bottom it shows Whois server: whois.nic.ar. So I try that and voila, meaningful information is returned, no ads accompanying:

$ whois ‐h whois.nic.ar buenosaires.com.ar

% La información a la que estás accediendo se provee exclusivamente para
% fines relacionados con operaciones sobre nombres de dominios y DNS,
% quedando absolutamente prohibido su uso para otros fines.
%
% La DIRECCIÓN NACIONAL DEL REGISTRO DE DOMINIOS DE INTERNET es depositaria
% de la información que los usuarios declaran con la sola finalidad de
% registrar nombres de dominio en ‘.ar’, para ser publicada en el sitio web
% de NIC Argentina.
%
% La información personal que consta en la base de datos generada a partir
% del sistema de registro de nombres de dominios se encuentra amparada por
% la Ley N° 25326 “Protección de Datos Personales” y el Decreto
% Reglamentario 1558/01.
 
domain:         buenosaires.com.ar
registrant:     50030338720
registrar:      nicar
registered:     2012-07-05 00:00:00
changed:        2017-06-27 17:42:45.944889
expire:         2018-07-05 00:00:00
 
contact:        50030338720
name:           TRAVEL RESERVATIONS SRL
registrar:      nicar
created:        2013-09-05 00:00:00
changed:        2018-04-17 13:14:55.331068
 
nserver:        ns-1588.awsdns-06.co.uk ()
nserver:        ns-925.awsdns-51.net ()
nserver:        ns-1385.awsdns-45.org ()
nserver:        ns-239.awsdns-29.com ()
registrar:      nicar
created:        2016-07-01 00:02:28.608837

2nd example: goto.jobs
I actually needed this one! So I learned of a domain goto.jobs and I wanted to get some background. So here goes…
$ whois goto.jobs

getaddrinfo(jobswhois.verisign-grs.com): Name or service not known

So off to a bad start, right? So we hit up the .jobs link on iana, https://www.iana.org/domains/root/db/jobs.html, and we spy a reference to their whois server:

Registry Information
This domain is managed under ICANN's registrar system. You may register domains in .JOBS through an ICANN accredited registrar. The official list of ICANN accredited registrars is available on ICANN's website.
URL for registration services: http://www.goto.jobs
WHOIS Server: whois.nic.jobs

So we try that:
$ whois ‐h whois.nic.jobs goto.jobs

   Domain Name: GOTO.JOBS
   Registry Domain ID: 91478530_DOMAIN_JOBS-VRSN
   Registrar WHOIS Server: whois-all.nameshare.com
   Registrar URL: http://www.nameshare.com
   Updated Date: 2018-03-29T20:08:46Z
   Creation Date: 2010-02-04T23:54:33Z
   Registry Expiry Date: 2019-02-04T23:54:33Z
   Registrar: Name Share, Inc
   Registrar IANA ID: 667
   Registrar Abuse Contact Email:
   Registrar Abuse Contact Phone:
   Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
   Name Server: KATE.NS.CLOUDFLARE.COM
   Name Server: MARK.NS.CLOUDFLARE.COM
   Name Server: NS1.REGISTRY.JOBS
   Name Server: NS2.REGISTRY.JOBS
   DNSSEC: unsigned
   URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/
>>> Last update of WHOIS database: 2018-04-23T18:54:31Z <<<

Better, but it seems to merely point to a registrar and its whois server:

Registrar WHOIS Server: whois-all.nameshare.com

So let’s try that:

$ whois ‐h whois-all.nameshare.com goto.jobs

Domain Name: GOTO.JOBS
Registry Domain ID: 91478530_DOMAIN_JOBS-VRSN
Registrar WHOIS Server: whois-jobs.nameshare.com
Registrar URL: http://www.nameshare.com
Updated Date: 2018-03-29T20:08:46Z
Creation Date: 2010-02-04T23:54:33Z
Registrar Registration Expiration Date: 2017-02-04T23:54:33Z
Registrar: NameShare, Inc.
Registrar IANA ID: 667
Registrar Abuse Contact Email: [email protected]
Registrar Abuse Contact Phone: +1.7809429975
Domain Status: clientTransferProhibited http://www.icann.org/epp#clientTransferProhibited
Registry Registrant ID:
Registrant Name: DNS Administrator
Registrant Organization: Employ Media LLC
Registrant Street: 3029 Prospect Avenue
Registrant City: Cleveland
Registrant State/Province: OH
Registrant Postal Code: 44115
Registrant Country: United States
Registrant Phone: +1.2064261500
Registrant Phone Ext:
Registrant Fax: +1.1111111111
Registrant Fax Ext:
Registrant Email: [email protected]
Registry Admin ID:
Admin Name: DNS Administrator
Admin Organization: Employ Media LLC
Admin Street: 3029 Prospect Avenue
...

Bingo! We have hit pay dirt. We have meaningful information about the registrant – an address, phone number and email address – and received no obnoxious ads in return. For me it’s worth the extra steps.

ICANN: another alternative
Most registrar’s whois sites are rate-limited. ICANN’s is not. And they also do not sic ads on you. It is

https://whois.icann.org/en/lookup?name=

ICANN, for the record, it the body that decides what goes on in DNS namespace, for instance, what new gTLDS should be added. You can use its whois tool for all gTLDs, but not in general for ccTLDs.

whois is undergoing changes due to GDPR. Especially the “social” information of the contacts: registrant, admin and technical contacts will be masked, except for perhaps state and country, in the future. But whois is slowly dying and a new standard called RDAP will take its place.

References and related
This page has some great tips. Wish I had seen it first! https://superuser.com/questions/758647/how-to-whois-new-tlds

Here’s that iana root zone database link again: https://www.iana.org/domains/root/db

ICANN’s whois: https://whois.icann.org/en/lookup?name=

Categories
Linux Raspberry Pi

Raspberry Pi as Retro Arcade Games emulator

Intro
I am not going to attempt to provide a guide as there are much better guides out there than anything I can produce.

In addition to the arcade function, we wanted to display a slidedeck when not being used for gaming.

Two main approaches I see are

1) install RetroPie, then add X packages
2) install Raspbian, then install RetroPie on top of that

The reason we want X is to run a presentation software such as pipresents, which we are already familiar with.

For approach 1) I roughly followed this installation order.

Notes
Install lightdm and lxde
This takes a long time, maybe 30 minutes:
sudo apt install lxde lxde-core lxterminal lxappearance
sudo apt install lightdm
sudo apt-get install xutils
sudo apt-get install xserver-xorg

But one of my games didn’t run properly afterwards, so I am focused on method 2) for now.

I’m having trouble running startx from a non-console terminal. One thing I’m trying is:
sudo usermod -a -G tty pi
sudo apt-get install xserver-xorg-legacy
These two commands still didn’t do the trick, so I edited this file

/etc/X11/Xwrapper.config

and replaced allowed_user=console with allowed_users=anybody, and that worked! Once.

Then I installed RetroPie, turned it off so it does not autostart, and tried startx from a non-console terminal and I see this error:

(EE) xf86OpenConsole: Cannot open virtual console 2 (Permission denied)

then I re-installed xserver-xorg-legacy and startx once again worked. Hmm.

The instructions for installing RetroPie on top of an existing Raspbian installation are here:

https://retropie.org.uk/docs/Manual-Installation/

You should be comfortable with the linux command line. In the end I like this method of installation the best. I’ve done it several times now.

Equipment ideas
These $15 speakers https://www.amazon.com/gp/product/B003JTHO3U/ref=oh_aui_detailpage_o01_s01?ie=UTF8&psc=1 only use the USB port for power. They have a standard mini-stereo jack that is compatible with the Pi. I bought them. The Pi has enough juice to power them, which is convenient.
I went with NES (Nintendo Entertainment System) games. This pair of USB controllers I am told are a good approximation of the real thing: https://www.amazon.com/gp/product/B075ZN1GXK/ref=oh_aui_detailpage_o02_s00?ie=UTF8&psc=1. they’re only about $14.
Two player arcade quality controller from Recroommasters. About $349.

How to configure two player setup when you have an arcade-style console with only one USB connection
I find the documentation available on the Internet on this particular topic is terrible. In fact I never did find it. This YouTube video was just created. Although it’s specific to their Xtension console it looks to me applicable to any similar console:

https://www.youtube.com/watch?v=E8jHfhM5t_A&feature=youtu.be

Configuration
It takes a little getting used to. There are two main places where you do some configuration. There’s the RetroPie Configuration. Then there’s the emulationstation menu. The main thing to do from the emulationstation menu, which is launched by clicking Start from the main emulationstation screen, is to map the controller keys. For instance I program for an NES controller at home, and bring it to school where there is a cool two-player arcade-style controller which will have to be re-mapped.
The RetroPie configuration shows up from the main screen when you hit the down arrow key or something like that, then A. From here you can launch traditional raspi-config. I also used it to go into RetroPie setup, then into configuration and have emulationstation autolaunch at boot-up. You can also do a reboot from RetroPie setup.

Sound
To force sound out of the 3.5 mm stereo jack, go to RetroPie Configuration|RetroPie Setup|Configuration/tools|801 – audio settings|Headphones – 3.5 mm jack.

To get volume to 100% which you will need with the speakers I list below, go to emulation station menu|sound settings|system volume. By default it seems to be 77% which just isn’t enough juice.

References and related
Good discussion on X windows, display managers and desktop environments: https://raspberrypi.stackexchange.com/questions/26836/possible-to-reinstall-x-server-and-use-graphical-after-having-removed-it
Speakers for about $15: https://www.amazon.com/gp/product/B003JTHO3U/ref=oh_aui_detailpage_o01_s01?ie=UTF8&psc=1
Nintendo style USB controllers, $14: https://www.amazon.com/gp/product/B075ZN1GXK/ref=oh_aui_detailpage_o02_s00?ie=UTF8&psc=1
Two player setup with an arcade controller that has only one USB connection.
Arcade style two player console. Very cool. https://www.recroommasters.com/Xtension-Two-Player-Control-Board-Emulator-Edition-p/rm-xt-sd-board-ee.htm

Categories
Linux Network Technologies Raspberry Pi

Multiple IPs on the Raspberry Pi

Intro
In my previous post I showed how to turn a Raspberry Pi plus USB camera into something like an IP camera. In the course of that work I found it wasn’t so easy as it was in the past to assign static IPs upon boot. So I came up with my own unique method, which combines a modicum of Linux knowledge with a dash of networking knowledge.

The requirements
I sort of invented these requirements for myself, putting myself in the pickle I found myself in. I am working with a friend’s Pi 3 and didn’t want to mess it up too badly. Yet I wanted to easily work with it at home, and for the Robotics team. How to do it all?

I decided to permit the DHCP client, now called dhcpcd, running. So it will assign an IP address and appropriate gateway if there is a DHCP server present on the network. When I test at home I sometimes don’t use DHCP. When I bring my test setup to Robotics, more often than not I have my own little isolated LAN and no DHCP server. So, knowing that a single interface can have two or even more than two IP adresses, I created the following list of requirements for myself.

Act as DHCP client if there is DHCP server.
Additionally,
Assign static IP of 192.168.1.161/24 so it works in my home.
Assign another static IP of 10.31.42.15 so it works with a predictable IP in the robotics environment.
Let the two above IP assignments work even in the absence of a DHCP server!

Sounds kind of simple, but it’s not so easy.

I’m running a Raspberry Pi 3 with Raspbian Stretch (the release after Jessie).

Initial approach
With this version you’re supposed to use the file

/etc/dhcpcd.conf

to create a static IP.

But it works like c**p, at least when you want to push it and have it meet all the requirements above. It’s got a bug and doesn’t allow you to meet all the above requirements. I experimented. But my method does work.

The final solution
So in the end I leave /etc/dhcpcd.conf alone!

I use this new (to me) feature that crontab has an @reboot feature that calls its argument at boot time – just what we need.

Then I combined some old school use of ifconfig plus newer school command ip.

Here’s the script, which I call ip-assign.sh.

#!/bin/bash
sleep 2
# see if there is a dhcp-assigned IP already. If so 'scope global' appears in the listing
#  ip add show eth0 sample output:
addflag=""
ip add show eth0|grep -q 'scope global'
if [ $? == 0 ]; then
  addflag="add"
fi
# first IP
ifconfig eth0 $addflag 10.31.42.15 netmask 255.255.255.0 broadcast 10.31.42.255
# next IP
ifconfig eth0 add 192.168.1.161 netmask 255.255.255.0 broadcast 192.168.1.255

What I observed is that eth0 already has an IP assigned to it (for instance from a DHCP server), then the string “scope global” appears when you run ip add, otherwise it doesn’t. Furthermore, ifconfig has an optional argument I noticed call add, which seems to exist in order to add additional virtual interfaces – precisely what we want. But if there is no IP yet assigned we should call ifconfig the first time with the add argument. If I had had additional virtual IPs I could have just kept on going…

So to call this at boot time I use my lazy method. I edit the crontab file and insert a line like this:

@reboot sudo ~/ip‐assign.sh > /tmp/ip‐assign.log 2>&1

So without a DHCP server I have after booting:
$ ip add show eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:e3:02:74 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.161/24 brd 10.31.42.255 scope global eth0:0
       valid_lft forever preferred_lft forever
    inet 10.31.42.15/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 169.254.159.115/16 brd 169.254.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::e923:3131:224c:ecd/64 scope link
       valid_lft forever preferred_lft forever

If you’re lazy like me just type
$ ip add
and you’ll get the other interfaces as well. It’s very easy to type, too!

Note the broadcast (brd) addresses are reversed from how you’d expect them. I decided that it doesn’t matter as long as they’re both present somewhere with the correct value. It’s all using the one physical interface so the interface doesn’t really care. And from all my testing I am right I believe on this point.

Disable WiFi – wlan0
To disable WiFi entirely, which you may want to do if using in a FIRST FRC competition, add this to /boot/config.txt and reboot:
dtoverlay=pi3‐disable‐wifi
After doing that wlan0 does not even show up when you do an ip add.

References and related
Raspberry Pi plus USB camera: brought together like an IP camera.

Categories
Linux Raspberry Pi Web Site Technologies

Raspberry Pi USB webcam turned into IP camera

Intro
Why would you even want to do this when you can buy a native IP webcam for less? I’m not sure, but i found myself in this situation so it could happen to others, and I found some things that worked and some that required quite some effort.

In my previous post I spoke about using opencv on Raspberry Pi.

This post is more about getting at an image with a minimum of lag time and relatively low bandwidth.

The setup
The specific camera I am working with is an ELP mini USB camera for $20.

What I did not do
I considered bolting on an add-on to opencv to convert the video stream into mjpeg. But the process looked relatively obscure so I did not feel that was a good way to go.

I skimmed through the mjpeg (motion jpeg) standard. Looks pretty straightforward. i even considered writing my own streamer. It’s probably not too hard to write a bad one! But I feared it would be unreliable so I didn’t go that route. It’s just jpeg, separator, jpeg, separator, jpeg, etc. Here’s the Wikipedia link: https://en.wikipedia.org/wiki/Motion_JPEG.

I think the best software for is mjpg_streamer. It is not available as a simple package. So you have to compile it and patch it.

Follow his recipe
This guy’s recipe worked for me:
https://jacobsalmela.com/2014/05/31/raspberry-pi-webcam-using-mjpg-streamer-over-internet/

Mostly! I needed the patch as well (which he also mentions). his instructions for the patch aren’t accurate.

He provides a link. You need to save the contents by launching the downloaded file and saving it as input_uvc_patch.txt after opening it in Windows Notepad (if you’re doing this download through Windows).

On the Pi, you would do these steps:

cd ~/mjpg-streamer
patch -p0 < input_uvc_patch.txt
make USE_LIBV4L2=true clean all
sudo make DESTDIR=/usr/local install

That is, assuming you had copied the patch file into that ~/mjpg-streamer directory.

Before we get too far, I wished to mention that the command fswebcam proved somewhat useful for debugging.

Here’s a weird thing about that camera
We had one, then I got another one. The two cameras do not behave the same way!

Device files
I guess Raspberry Pi has its own version of plug-and-play. So what it means is that when you plug in the camera a device file is dynamically created called /dev/video0. Now if you happen to plug in a second USB camera, that one becomes device /dev/video1. Some utilities are designed to work with /dev/video0 and require extra arguments to deal with a camera with a different device number, e.g., fswebcam -d /dev/video1 image.jpg.

But actually running two cameras did not work out too well for me. It seemed to crash and I don’t have time to investigate that.

The working command is…
My livestream.sh file looks like this right now. It will change but this is a good document point.

#!/bin/bash
/usr/local/bin/mjpg_streamer -i "/usr/local/lib/input_uvc.so -yuv -f 12 -q 50 \
 -r 352x288" -o "/usr/local/lib/output_http.so -w /usr/local/www"

The main point is that I found this additional -yuv argument seemed to get the one webcam to work, whereas the other USB camera didn’t need that! If you don’t include it launcher.sh may appear to work, but all you see when you connect to the direct video stream looks like this image:

One time when I ran it it crashed and suggested that -yuv argument be added, so I tried it and it actually worked! That’s how i discovered that oddity.

Bandwidth with those settings
About 2 mbps. How do I measure that? simple. I bring up the web page and tool around the networking stuff until i find Change Adapter Settings (always difficult to find). Then I double-click on my active adapter and stare at the received bytes to get a feel for how much it’s incrementing by each second. Multiply by 10, and voila, you have a crude measure, perhaps +/- 30%, of your bandwidth consumed!

Latency
This is so important it needs its own section.

Latency is pretty good. We’ve measured it to be 0.26 seconds.

fswebcam errors
What happens if you run fswebcam while livestream is running?
$ fswebcam /tmp/image.jpg

--- Opening /dev/video0...
Trying source module v4l2...
/dev/video0 opened.
No input was specified, using the first.
Error selecting input 0
VIDIOC_S_INPUT: Device or resource busy

Makes sense. Only one program on the Pi can capture the output form the camera.

Does the simple command fswebcam image.jpg work all the time? No it does not! Sometimes it simply fails, which is scary.

Here is an example of two consecutive calls to fswebcam about a second apart which illustrates the problem:

$ fswebcam /tmp/image.jpg

--- Opening /dev/video0...
Trying source module v4l2...
/dev/video0 opened.
No input was specified, using the first.
Adjusting resolution from 384x288 to 352x288.
--- Capturing frame...
Timed out waiting for frame!
No frames captured.

$ fswebcam /tmp/image.jpg

--- Opening /dev/video0...
Trying source module v4l2...
/dev/video0 opened.
No input was specified, using the first.
Adjusting resolution from 384x288 to 352x288.
--- Capturing frame...
Captured frame in 0.00 seconds.
--- Processing captured image...
Writing JPEG image to '/tmp/image.jpg'.

Running two USB cameras wih a single Ras Pi
This initially did not work in my first attempts but now it does!

It probably helps to be running a Raspebrry Pi 3 with Raspbian Stretch OS.

Maybe this wasn’t needed but we made a directory /usr/local/www2 and copied all the files from /usr/local/www to /usr/local/www2. A 2nd USB camera when plugged in creates /dev/video1 as I mentioned. You have to pick a different port, so we chose port 8090. Putting it all together we have the script below, livestream2.sh:

#!/bin/bash
/usr/local/bin/mjpg_streamer -i "/usr/local/lib/input_uvc.so -d /dev/video1 -yuv -q 50 -r 352x288 -f 12" -o "/usr/local/lib/output_http.so -w /usr/local/www2 -p 8090"

If a 2nd camera isn’t plugged in then the script errors out and doesn’t run, which is pretty much what we want. Running it by hand we get this:

$ ./livestream2.sh

MJPG Streamer Version: svn rev: 3:172M
 i: Using V4L2 device.: /dev/video1
 i: Desired Resolution: 352 x 288
 i: Frames Per Second.: 12
 i: Format............: YUV
 i: JPEG Quality......: 80
ERROR opening V4L interface: No such file or directory
 Init v4L2 failed !! exit fatal
 i: init_VideoIn failed

Reining in the bandwidth
We found that by lowering the jpeg quality with the -q option we could reduce the bandwidth and the quality, but the quality was still good enough for our purposes. Now the video streams from both cameras comes in around 4.5 mbps, even in bright lighting. So we settled on -q 50 for a 50% quality. Even a quality of 10 (10%) is not all that bad! I believe the default is 80%.

Bandwidth monitor on the Pi
Some of this was written by the student so apologies for the misspellings! Probably will be refined in the future. We can tease out how much bandwidth we’re actually using on the Pi by measuring the transmitted (TX) bytes periodically. We’ll record that during a matcgh so we can prove to ourselves and others that we have our bandwidth under control – far less than 7 mbps despite using two cameras.

banwidthmonitor.pl Perl program

#!/usr/bin/perl
#monitor banwidth
$DEBUG = 1;
$sleep = 5;
$| = 1;
$date = `date`;
print $date;
for (;;) {
  $tx = `ip -s link show eth0 | tail -1| awk \'{print \$1}\'`;
  print $tx if $DEBUG;
  $txbitstotal = 8 * $tx;
  $timetotal = time;
  $txbits = $txbitstotal - $txbitstotalold if $txbitstotalold;
  $time = $timetotal - $timetotalold;
  $txbitstotalold = $txbitstotal;
  $banwidth = $txbits / $time if $timetotalold;
  print "banwidth $banwidth\n";
  $timetotalold = $timetotal;
# TX: bytes  packets  errors  dropped carrier collsns
#    833844072  626341   0       0       0       0
  sleep $sleep;
}

Output from program
Watch as our bandwidth usage grows to around 700 kbps as we turn on one of our video cameras.
$ ./banwidthmonitor.pl

Tue Jan 30 21:09:32 EST 2018
9894771
banwidth
9895095
banwidth 518.4
10252073
banwidth 571164.8
10697648
banwidth 712920
11151985
banwidth 726939.2
11597595
banwidth 712976
12043230
banwidth 713016
^C

Unreliable video stream startup
Sometimes one video stream does not come on correctly after first power-up. This is most perplexing as with computer gear one expects consistent, reproducible behaviour, yet that is not at all what we’ve observed.
This makes no sense, but in one environment we had our two streams running successfully six times in a row. Then I take the equipment home and find only one of the two streams starts up. It seems more likely to fail after sitting powered off for a few hours! I know it doesn’t make sense but that’s how it is.

In any case we have built a monitor which looks for and corrects this situation. It’s pretty clever and effective if I say so myself! And necessary! We created one monitor each for the two video devices. Here’s videomonitor.sh:

#!/bin/bash
# DrJ make sure video stream is not stuck. Restart it if it is
sleep 8
while /bin/true; do
  chars=`curl -s -m1 localhost:80/?action=stream|wc -c`
  if [ $chars -lt 100 ]; then
# we are stuck!
    date
    echo Video stuck so we will restart it
    pid=`ps -ef|grep mjpg|grep 'p 80'|grep -v sudo|awk '{print $2}'`
    sudo kill $pid
    sleep 1
    ~/livestream.sh &
# restart...
  else
# we have a good stream
    touch /tmp/stream80
  fi
  sleep 5
done

and videomonitor2.sh

#!/bin/bash
# DrJ make sure video stream is not stuck. Restart it if it is
sleep 8
while /bin/true; do
  chars=`curl -s -m1 localhost:443/?action=stream|wc -c`
  if [ $chars -lt 100 ]; then
# we are stuck!
    date
    echo Video stuck so we will restart it
    pid=`ps -ef|grep mjpg|grep 'p 443'|grep -v sudo|awk '{print $2}'`
    sudo kill $pid
    sleep 1
    ~/livestream2.sh &
# restart...
  else
# we have a good stream
    touch /tmp/stream443
  fi
  sleep 5
done

And we’ll start these at boot time like the long and growing list of things we are starting at boot time.

Allowed ports
From rule 66…

R66. Communication between the ROBOT and the OPERATOR CONSOLE is restricted as follows:
A. Network Ports:
HTTP 80: Camera connected via switch on the ROBOT, bi-directional
HTTP 443: Camera connected via switch on the ROBOT, bi-directional
...

So…to be safe we are switching from use of ports 8080 and 8090 to ports 80 and 443. But this means we have to preface certain commands – such as mjpg_streamer – with sudo since tcp ports < 1024 are privileged.

Flashing an led when we have a good video stream
Our led is soldered to a gruond pin and GPIO pin 18.

We call this program ledflash.sh

#!/bin/bash
#flashes the led
while /bin/true; do
if [ -f /tmp/stream80 ] && [ -f /tmp/stream443 ]; then
  pin=18
  cd /sys/class/gpio
  echo $pin > export
  cd gpio$pin
  echo out > direction
  while /bin/true; do
#make 5 quick flashes
    for i in `seq 1 5`; do
      echo 1 > value
      sleep 0.1
      echo 0 > value
      sleep 0.1
    done
#now lets make the long flash
    echo 1 > value
    sleep 0.6
  done
fi
sleep 2
done

We start it at boot time as well. It tells us when both video streams are ready for viewing because only then do the files get created and then the led starts flashing.

It takes about 62 seconds from the time power is supplied to the Raspberry Pi to the time the LED starts flashing (indicating the two video streams are ready).


Picture of setup

This picture goes a long way to convey the ideas.

2 USB cameras, 1 Ras Pi, flashing LED

References and related
Multiple IP addresses
We needed an IP for testing in the lab, another when we brought it home and a third for competitions. This blog post showed how we gave it all needed IP addresses for our purposes!

FIRST FRC provides this guide for use of IP addresses at their events.

Amazon seemed to run out of the original USB camera we worked with. The ELP pinhole USB camera seems to work just as well and is just as cheap, around $20: https://smile.amazon.com/gp/product/B00K7ZWVVO/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1

Raspberry Pi model 2 and 3 GPIO pins are documented here: https://www.raspberrypi.org/documentation/usage/gpio-plus-and-raspi2/.

The lazy person’s way to start any script at during boot-up is presented here: Linux tip: easy way to automatically start a script after a reboot

Categories
Linux Python Raspberry Pi

A first taste of OpenCV on a Raspberry Pi 3

Intro
I’ve done a few things to do some vision processing with OpenCV on a Raspberry Pi 3. I am a rank amateur so my meager efforts will not be of much help to anyone else. My idea is that maybe this could be used on an FRC First Robotics team’s robot. Hence I will be getting into some tangential areas where I am more comfortable.

Even though this is a work in progress I wanted to get some of it down before I forget what I’ve done so far!

Tangential Stuff

Disable WiFi
You shouldn’t have peripheral devices with WiFi enabled. Raspeberry Pi 3 comes with built-in WiFi. Here’s how to turn it off.

Add the following line to your /boot/config.txt file:

dtoverlay=pi3‐disable‐wifi

Reboot.

If it worked you should only see the loopback and eth0 interefaces in response to the ip link command, something like this:

$ ip link
1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether b8:27:eb:3f:92:f3 brd ff:ff:ff:ff:ff:ff

Hardcode an IP address the simple-minded way
On a lark I decided to try the old-fashioned method I first used on Sun Solaris, or was it even Dec Ultrix? That is, ifconfig. I thought it was to be deprecated but it works well enough for my purpose.

So something like

$ sudo ifconfig eth0 192.168.1.160

does the job, as long as the network interface is up and connected.

Autolaunch a VNC Server so we can haul the camera image back to the driver station
$ vncserver &hypher;geometry 640×480 ‐Authentication=VncAuth :1

Launch our python-based opencv program and send output to VNC virtual display

$ export DISPLAY=:1
$ /home/pi/.virtualenvs/cv/bin/python green.py > /tmp/green.log 2>&1 &

The above was just illustrative. What I actually have is a single script, launcher.sh which puts it all together. Here it is.

#!/bin/sh
# DrJ
sleep 2
# set a hard-wired IP - this will have to change!!!
sudo ifconfig eth0 192.168.1.160
# launch small virtual vncserver on DISPLAY 1
vncserver -Authentication=VncAuth :1
# launch UDP server
$HOME/server.py > /tmp/server.log 2>&1 &
# run virtual env
cd $HOME
# don't need virtualenv if we use this version of python...
#. /home/pi/.profile
#workon cv
#
# now launch our python video capture program
#
export DISPLAY=:1
/home/pi/.virtualenvs/cv/bin/python green.py > /tmp/green.log 2>&1 &

OpenCV (open computer Vision)
opencv is a bear and you have to really work to get it onto a Pi 3. There is no apt-get install opencv. You have to download and compile the thing. There are many steps and few accurate documentation sources on the Internet as of this writing (January 2018).

I think this guide by Adrian is the best guide:

Install guide: Raspberry Pi 3 + Raspbian Jessie + OpenCV 3

However I believe I still ran into trouble and I needed this cmake command in stead of the one he provides:

cmake -D CMAKE_BUILD_TYPE=RELEASE \
        -D CMAKE_INSTALL_PREFIX=/usr/local \
        -D INSTALL_C_EXAMPLES=OFF \
        -D ENABLE_PRECOMPILED_HEADERS=OFF \
        -D INSTALL_PYTHON_EXAMPLES=ON \
        -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib-3.1.0/modules \
        -D BUILD_EXAMPLES=ON ..

I also replaced opencv references to version 3.0.0 with 3.1.0.

I also don’t think I got make -j4 to work. Just plain make.

An interesting getting started tutorial on images, opencv, and python:

http://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_gui/py_image_display/py_image_display.html#display-image

Simplifying launch of VNC Viewer
I wrote a simple-minded DOS script which launches UltraVNC with a password. So with a double-click it should work).

Here’s a Dos .bat file to launch ultravnc viewer by double-clicking on it.

if not "%minimized%"=="" goto :minimized
set minimized=true
start /min cmd /C "%~dpnx0"
goto :EOF
:minimized
c:\apps\ultravnc\vncviewer -password raspberry 192.168.1.160:1

I’m sure there’s a better way but I don’t know it.

The setup
We have a USB camera plugged into the Pi.
A green disc LED light.
A green filter over the camera lens.
A target with two parallel strips of retro-reflective tape we are trying to suss out from everything else.
Some sliders to control the sensitivity of our color matching.
The request to analyze the video in opencv as well as display it on the driver station.
Have opencv calculate the pixel distance (“correction”) from image center of the “target” (the two parallel strips).
Send this correction via a UDP server to any client who wants to know the correction.

Here is our current python program green.py which does these things.

import Tkinter as tk
from threading import Thread,Event
from multiprocessing import Array
from ctypes import c_int32
import cv2
import numpy as np
import sys
#from Tkinter import *
#cap = cv2.VideoCapture(0)
global x
global f
x = 1
y = 1
f = "green.txt"
 
class CaptureController(tk.Frame):
    NSLIDERS = 7
    def __init__(self,parent):
        tk.Frame.__init__(self)
        self.parent = parent
 
        # create a synchronised array that other threads will read from
        self.ar = Array(c_int32,self.NSLIDERS)
 
        # create NSLIDERS Scale widgets
        self.sliders = []
        for ii in range(self.NSLIDERS):
            # through the command parameter we ensure that the widget updates the sync'd array
            s = tk.Scale(self, from_=0, to=255, length=650, orient=tk.HORIZONTAL,
                         command=lambda pos,ii=ii:self.update_slider(ii,pos))
            if ii == 0:
                s.set(0)  #green min
            elif ii == 1:
                s.set(0)
            elif ii == 2:
                s.set(250)
            elif ii == 3:
                s.set(3)  #green max
            elif ii == 4:
                s.set(255)
            elif ii == 5:
                s.set(255)
            elif ii == 6:
                s.set(249)  #way down below
            s.pack()
            self.sliders.append(s)
 
        # Define a quit button and quit event to help gracefully shut down threads
        tk.Button(self,text="Quit",command=self.quit).pack()
        self._quit = Event()
        self.capture_thread = None
 
    # This function is called when each Scale widget is moved
    def update_slider(self,idx,pos):
        self.ar[idx] = c_int32(int(pos))
 
    # This function launches a thread to do video capture
    def start_capture(self):
        self._quit.clear()
        # Create and launch a thread that will run the video_capture function
#        self.capture_thread = Thread(cap = cv2.VideoCapture(0), args=(self.ar,self._quit))
        self.capture_thread = Thread(target=video_capture, args=(self.ar,self._quit))
        self.capture_thread.daemon = True
        self.capture_thread.start()
 
    def quit(self):
        self._quit.set()
        try:
            self.capture_thread.join()
        except TypeError:
            pass
        self.parent.destroy()
 
# This function simply loops over and over, printing the contents of the array to screen
def video_capture(ar,quit):
    print ar[:]
    cap = cv2.VideoCapture(0)
    Xerror = 0
    Yerror = 0
    XerrorStr = '0'
    YerrorStr = '0'
    while not quit.is_set():
        # the slider values are all readily available through the indexes of ar
        # i.e. w1 = ar[0]
        # w2 = ar[1]
        # etc.
        # Take each frame
        _, frame = cap.read()
        # Convert BGR to HSV
        hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
        # define range of blue color in HSV
        lower_green = np.array([ar[0],ar[1],ar[2]])
        upper_green = np.array([ar[3],ar[4],ar[5]])
        # Threshold the HSV image to get only green colors
        mask = cv2.inRange(hsv, lower_green, upper_green)
        # Bitwise-AND mask and original image
        res = cv2.bitwise_and(frame,frame, mask= mask)
        cv2.imshow('frame', frame)
#        cv2.imshow('mask',mask)
#        cv2.imshow('res',res)
        #------------------------------------------------------------------
        img = cv2.blur(mask,(5,5))   #filter (blur) image to reduce errors
        cv2.imshow('img',img)
        ret,thresh = cv2.threshold(img,127,255,0)
        im2,contours,hierarchy = cv2.findContours(thresh, cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
        print 'number of contours==640x480====================  ', len(contours)
        target=0
        if len(contours) > 0:
            numbercontours = len(contours)
            while numbercontours > 0:
                numbercontours = numbercontours -1  # contours start at 0
                cnt = contours[numbercontours]   #this is  getting the first contour found, could look at 1,2,3 etc
                x,y,w,h = cv2.boundingRect(cnt)
#
#---line below has the limits of the area of the target-----------------------
#
                #if w * h > 4200 and w * h < 100000:  #area of capture must exceed  to exit loop
                if h > 30 and w < h/3:  #area of capture must exceed  to exit loop
                    print ' X   Y  W  H  AREA      Xc  Yc      xEr yEr'
                    Xerror = (-1) * (320 - (x+(w/2)))
                    XerrorStr = str(Xerror)
                    Yerror = 240 - (y+(h/2))
                    YerrorStr = str(Yerror)
                    print  x,y,w,h,(w*h),'___',(x+(w/2)),(y+(h/2)),'____',Xerror,Yerror
                    break
 
#-------        draw horizontal and vertical center lines below
                cv2.line(img,(320,0),(320,480),(135,0,0),5)
                cv2.line(img,(0,240),(640,240),(135,0,0),5)
                displaySTR = XerrorStr + '  ' + YerrorStr
                font = cv2.FONT_HERSHEY_SIMPLEX
                cv2.putText(img,displaySTR,(10,30), font, .75,(255,255,255),2,cv2.LINE_AA)
                cv2.imshow('img',img)
# wrtie to file for our server'
                sys.stdout = open(f,"w")
                print 'H,V:',Xerror,Yerror
                sys.stdout = sys.__stdout__
                target=1
                #
                #--------------------------------------------------------------------
        if target==0:
                # no target found. print non-physical values out to a file
                sys.stdout = open(f,"w")
                print 'H,V:',1000,1000
                sys.stdout = sys.__stdout__
        k = cv2.waitKey(1) & 0xFF    #parameter is wait in millseconds
        if k == 27:   # esc key on keboard
            cap.release()
            cv2.destroyAllWindows()
            break
 
if __name__ == "__main__":
    root = tk.Tk()
    selectors = CaptureController(root)
    selectors.pack()
#    q = tk.Label(root, text=str(x))
#    q.pack()
    selectors.start_capture()
    root.mainloop()

Well, that was a big program by my standards.

Here’s the UDP server that goes with it. I call it server.py.

#!/usr/bin/env python
# inspired by https://gist.github.com/Manouchehri/67b53ecdc767919dddf3ec4ea8098b20
# first we get client connection, then we read data frmo file. This order is important so we get the latest, freshest data!
 
 
import socket
import re
 
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
 
server_address = '0.0.0.0'
server_port = 5005
 
server = (server_address, server_port)
sock.bind(server)
print("Listening on " + server_address + ":" + str(server_port))
 
while True:
# read up to 32 bytes from client
        payload, client_address = sock.recvfrom(32)
        print("Request from client: " + payload)
# get correction from file
        while True:
                with open('green.txt','r') as myfile:
                        data=myfile.read()
#H,V:  9 -14
                data = data.split(":")
                if len(data) == 2:
                        break
        sent = sock.sendto(data[1], client_address)

For development testing I wrote a UDP client to go along with that server. I called it recvudp.py.

#!/usr/bin/env python
import socket
UDP_IP = "127.0.0.1"
UDP_PORT = 5005
 
print "UDP target IP:", UDP_IP
print "UDP target port:", UDP_PORT
 
sock = socket.socket(socket.AF_INET, # Internet
                 socket.SOCK_DGRAM) # UDP
# need to send one newline minimum to receive server's message...
MESSAGE = "correction";
sock.sendto(MESSAGE, (UDP_IP, UDP_PORT))
# get data
data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
print "received message:", data

Problems
Lag is bad. Probably 1.5 seconds or so.
Video is green, but then we designed it that way.
Bandwidth consumption of VNC is way too high. We’re supposed to be under 7 mbps and it is closer to 12 mbps right now.
Probably won’t work under the bright lights or an arena or gym.
Sliders should be labelled.
Have to turn a pixel correction into an angle.
Have to suppress initial warning about ssh default password.

To be improved, hopefully…

Categories
Linux Raspberry Pi

Raspberry Pi automates cable modem power cycling task

2021 update

The problem with my cable modem is back. So I’ve revisited my own project and just slightly tweaked things. I am also using an RPi model 4 now. Works great…

Original Intro

I lose my Internet far too often – sometimes once a day. Of course I have lots of network gear in a rat’s nest of cables. I narrowed the problem down to the cable modem, which simply needs to be power cycled and all is good. Most people would call their cable company at this point. I decided to make a little project of it to see if I could get my Raspberry Pi to

– monitor the Internet connection and
– automatically power-cycle the cable modem

Cool, right?

Needless to say, if I can power cycle a modem, I can control power to all kinds of devices with the RPi.

Is there a product already on the market?
Why yes, there is. Normally that would shut me down in my tracks because what’s the point? But the product is relatively expensive – $100, so my DIY solution is considerably less since I already own the RPi. See references for a link to the commercial solution to this problem.


Getting a control cable

This is pathetic, but, in 2017 I originally cut out a cable from an old computer that no longer works. The jumper has more pins than I need, but I could make it work. In 2021 I used proper jumper cables. It’s neater. Thing is, I’ve had them for awhile and I forget where I got them from. perhaps a friend.

Setting up my GPIO, just for testing

The following is only there to show you how easy it is to send signals out the GPIO pin. The script I wrote below, connTest.sh, does all this setup for you if you just want to quickly get down to business.

I am plugged into the end so I need to manipulate GPIO pin 21.
Become root
$ sudo su –
Get to the right directory
$ cd /sys/class/gpio
Create the pin for user manipulation
$ echo 21 > export
Move to that pin’s directory
$ cd gpio21
Set up pin for sending signal OUT
$ echo out > direction
Test what we have so far
$ cat direction

out

$ cat value
0

connTest.sh script
I put this in /usr/local/etc and called it connTest.sh. I’m still tinkering with it a bit. But it shows what we’re basically trying to do.

#!/usr/bin/bash
# DrJ 8/2021
# Test if Internet connection is still good and send signal to relay if it is not
# see https://drjohnstechtalk.com/blog/2017/10/raspberry-pi-automates-cable-modem-power-cycling-task/?preview_id=3121&preview_nonce=9b896f248d&post_format=standard&_thumbnail_id=-1&preview=true
Break=300
Sleep=11
pingpadding=30 # if no response ping takes longer to run
log=/var/log/connTest
pinglog=/tmp/ping.log
#
# one-time setup of our GPIO pin so we can control it
# if the power is on the right, GPIO pin 21 is the lower right pin
pin=21
cd /sys/class/gpio
echo $pin > export
cd gpio$pin
echo out > direction

# divert STDOUT and STDERR to log file
exec 1>$log
exec 2>&1
echo "$0 starting monitoring at "$(date)
# report our external IP
curl -s ipinfo.io|head -2|tail -1
while /bin/true; do
try1=$(curl -is --connect-timeout 6 www.google.com|wc -c)
[[ $try1 -lt 300 ]] && {
  echo google came up short. Trying amazon next. characters: $try1
  sleep 60
  try2=$(curl -is --connect-timeout 6 https://www.amazon.com|wc -c)
  [[ $try2 -lt 300 ]] && {
    echo "#################"
    echo "We have a connection problem at "$(date)
    echo character counts. google: $try1, amazon, $try2
    echo "Power cycling router and waiting for $Break seconds"
# start a ping job
    ping -c $Break 1.1.1.1 > $pinglog 2>&1 &

# this will shut power off
    echo 1 > value
    sleep 4
# and this will turn it back on
    echo 0 > value
# this prevents us from too aggressively power-cycling
    sleep $(($Break+$pingpadding))
# report on ping results
#22 packets transmitted, 22 received, 0% packet loss, time 53ms
#rtt min/avg/max/mdev = 6.536/15.533/24.705/4.510 ms
    echo printing last three lines from ping results log:
    tail -3 $pinglog
    line=`tail -2 $pinglog|head -1`
    t1=`echo -n $line|awk '{print $1}'`
    t2=`echo -n $line|awk '{print $4}'`
#  downtime=$(($t1-$t2))
# test for integer inputs
    [[ "$t1" =~ ^[0-9]+$ ]] && [[ "$t2" =~ ^[0-9]+$ ]] && downtime=$(($t1-$t2))
    echo  DOWNTIME: $downtime seconds
# report our external IP
    curl -s ipinfo.io|head -2|tail -1
    echo "#################"
  }
}
sleep $Sleep
done

Starting on boot

These days I just use my crontab trick – much easier. You edit your crontab by saying sudo crontab -e. Then put in these lines at the bottom:

# DrJ 7/13/21
@reboot sleep 45; /usr/local/etc/connTest.sh > /tmp/connTestRun.log 2>&1
# bring down wireless after awhile – assume we have a wired connection
@reboot sleep 120; /usr/sbin/ifconfig wlan0 down

Only include that last line if you have an ethernet cable connection, which, for monitoring purposes, you should. WiFi is just not as reliable.

In all this I had the most trouble getting the startup script to bend to my will! But I think it’s functioning now. It may not be the most efficient, but it’s workable, meaning, it starts up connTest.sh after a reboot, and sends the log to /var/log/connTest.

My conntest file looks like this after I rebooted a few days ago:

/usr/local/etc/connTest.sh starting monitoring at Sat 18 Sep 08:11:47 EDT 2021
“ip”: “67.83.122.167”,
#################
We have a connection problem at Mon 20 Sep 14:13:03 EDT 2021
Power cycling router and waiting for 300 seconds
printing last three lines from ping results log:
— 1.1.1.1 ping statistics —
300 packets transmitted, 202 received, +9 errors, 32.6667% packet loss, time 563ms
rtt min/avg/max/mdev = 7.581/13.387/35.981/3.622 ms
DOWNTIME: 98 seconds
“ip”: “67.83.122.167”,

So it needs to restart my cable modem about every other day and often during those critical daytime hours when I am working from home.


Substitute below for one thousand words

Raspberry Pi GPIO pins 21 plus ground connected to the power relay

So you can almost make out the different outlets from the power relay: always on; normally on; normally off. Makes perfect sense, right?
See that green plug on the side of the relay? I was such a newbie I was shoving the wires into it, unsure how to make a good connection. Well, with a little effort it simply pulls out, revealing a screws that can be used to secure the wires in the holes.

Some conclusions about my cable modem problems

The problems always occur during the day, i.e., when it is being used more heavily (the monitoring is 24×7 so it doesn’t distinguish). So somehow it’s actual usage which triggers failure. I  wonder if it outputs more heat and overheats when the Internet is used more heavily? Just a hypothesis.

Outage can be reduced to about 90 seconds with this script based on the ping drop testing. Your mileage may vary, as they say.

My ISP does not give me a new IP after I reboot.

A strange error pops up

After running for awhile I noticed this error in the log:

604 Segmentation fault curl -s –connect-timeout 3 www.google.com > /dev/null

I ran curl by hand and confirmed the error. meanwhile, my bandwidth constraints were also lifted, so my librespeed testing was using up all my bandwidth due to that.

I’ve still got to look into that root cause of that issue. A reboot cleared it up however.

Conclusion

It’s fun to actually turn off and on 110V AC power using your Raspberry Pi! Especially when there is a useful purpose behind it such as a cable modem which starts to perform better after being power cycled. At only $30 this is a pretty affordable DIY project. I provide some scripts which shows how to work with GPIO pins using the command line. That turns out to be not so mysterious after all…
If the switching can work fast enough, I’m thinking of a next project with lights set to musical beats…!

References and related
Raspberry Pi models 2 – 4 GPIO pins are documented here: GPIO – Raspberry Pi Documentation

Generic GPIO documentation – how to use it from the operating system – is here: https://www.kernel.org/doc/Documentation/gpio/sysfs.txt
A PERL example of controlling GPIO I personally find too difficult to follow is here: https://raspberrypi.stackexchange.com/questions/41014/gpio-callbacks-in-any-language

This really nice 110 volt AC relay device, controlled by DC signal of anywhere from 3.3 to 48 volt DC, is on Amazon: https://smile.amazon.com/gp/product/B00WV7GMA2/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1. This is a really sweet device. Perfect for hobbyists with either Raspberry Pi or Arduino. And only $29!

Product which does all this monitoring/power cycling for you automatically: https://smile.amazon.com/dp/B015NM0LKI/ref=dp_sp_detail?psc=1. But it’s $100.

Another alternative which would also work is switching over ethernet. This device does that: https://smile.amazon.com/dp/B00KQ4R1RK/ref=dp_sp_detail?psc=1&pd_rd_i=B00KQ4R1RK&pd_rd_wg=dxYdP&pd_rd_r=8TBE6X2F39S2XWX9CGKK&pd_rd_w=bmvYC. There is a simple CGI URL you can use to turn power off/on. But, again, it’s more costly: $75.

Need gpio cables? If you don’t happen to have a desktop computer you can cannibalize, then for only $10 this Rpi expansion kit seems like a good way to go. https://smile.amazon.com/Kuman-Expansion-Raspberry-Solderless-Breadboard/dp/B074DSMPYD/ref=sr_1_1_sspa?ie=UTF8&qid=1509737721&sr=8-1-spons&keywords=GPIO+CABLES&psc=1

Categories
Raspberry Pi

Getting your micro SD card ready to load Raspbian: easier than you thought

Intro
Configuring your own micro SD card in order to install Raspbian on a Raspberry Pi is not so hard. Some of the instructions out there are a bit dated and make it out to be harder than it really is.

2021 update

Nowadays you ought to use the tool they finally created just for this purpose, the Raspberry Pi Imager: Introducing Raspberry Pi Imager, our new imaging utility – Raspberry Pi

Old information from 2017 follows…

The details
For instance this site has some extra steps you don’t need: http://elinux.org/RPi_Easy_SD_Card_Setup.

I’d stick with the simplest possible approach, which turns out to be this set of instructions: https://diyhacking.com/install-raspbian-raspberry-pis-sd-card/

But all these instructions seem to refer to an IMG file which I don’t even see. The main thing is to download NOOBS (new out-of-box software) from https://www.raspberrypi.org/downloads/ .

Then, get the SD card formatter. But the latest version is 5, not 4, and it looks different from before – there are essentially no options!

SD Card Formatter

So go with Quick Format and it works out OK. Unless your SD card is used. Then choose Overwrite format. That also works but takes a lot longer.

Then when it comes to copying the image file, which makes no sense with NOOBS because the image file is hidden, I think. Just extract all the files form the NOOBs zip file and copy them over to your E: drive, of whatever drive your SD card appears as.

Then follow the instructions on your Ras Pi display.

That’s it! I know because I just did it.

(non-)Reliability of SD Card
For the record, I’m in this situation because my old micro SD card just died. This is after running it continuously for a little over two years. Not very impressive in my book. Also for the record the card came as part of a Cana kit.

Symptoms of SD card failure in my case:

– boot paused, then after 120 seconds spits out some warnings about MMC something or other.
– LED status light solid green

A word about NOOBS and Balena Etcher
Note that the Etcher people were a bit lazy, and refuse to support burning NOOBS to an SD card with Etcher! to repeat, Etcher and NOOBS are incompatible. The stated reason is that NOOBS is not a true image.

A word about downloading from https://www.raspberrypi.org/downloads/
Today my PC just was not up to the task of downloading the full NOOBS zip file. It got to about 800 MB and then kept saying Failed. I found I could restart it, and that would download another 10 MB or so before failing again. This was getting pretty boring so I simply went to a Raspberry Pi and downloaded it from the command line using wget. No problems…

I suspect that my PC’s AV software was running amok and interfering with this download. I haven’t messed with disabling it in awhile (the usual prescription), so i used the Ras Pi itself. Then I did an sftp from my PC to the Ras Pi to get the downloaded image. That was also unusually slow, but it did go through, eventually.

References and related

Just go to this page to learn about and download the Raspberry Pi Imager which works for Windows or Mac: Introducing Raspberry Pi Imager, our new imaging utility – Raspberry Pi

All about Micro SD cards, their ratings and how to measure your own micro SD card’s performance: Raspberry Pi SD Card Speed Test – Raspberry Pi

Older references from 2017
This is probably the best guide, but I did not find a guide with 100% correct info; https://diyhacking.com/install-raspbian-raspberry-pis-sd-card/
As mentioned the latest image comes from: https://www.raspberrypi.org/downloads/

Categories
Apache Raspberry Pi

Automating guest wireless access for monitoring purposes with Raspberry Pi

Intro
I decided to monitor guest wireless access to the Internet using a Raspberry Pi. By that I mean a basic, binary, is it working now or not response. The back end is a Cisco wireless LAN Controller (WLC). Like most such systems there is no WiFi password, but your connection is extremely limited until you authenticate to the WLC login page in a browser. Further, this particular system is configured to only permit usage for up to four hours, after which another authentication is required to continue. The system is pretty reliable overall, but there are lots of pieces involved and I decided it would be nice to be the first to know if it isn’t working. And it’d be nice to put one of my spare Raspberry Pi’s to work in this semi-official capacity.

The details
Let’s cut to the chase. This is what my crontab file looks like:

# added for drj4guest WiFi testing - DrJ 4/26/17
# this line should keep us authenticating...
* * * * * curl -d `cat /home/pi/data` https://verify.drj4guests.johnstechtalk.com/login.html > /dev/null 2>&1
# and this is what we actually touch, where we have a separate monitor looking for it...every 2 minutes
*/2 * * * * curl http://johnstechtalk.com/raspberrypidrj4guest?`perl -e 'print time;'` > /dev/null 2>&1

For this to work I need accurate time on the Raspberry Pi. By default it was in the wrong timezone – UTC instead of EDT – and it had anyway drifted by quite a few seconds. I describe how to fix this all up in this post.

Let’s break this down. The WiFi is known as drj4guest, hence some of the naming conventions you see.

Here is the contents of the file data in /home/pi:

buttonClicked=4&redirect_url=johnstechtalk.com%2F&err_flag=0&agree=on&username=john&password=<DRJ4GUEST_PASSWORD>

So I meticulously reverse engineered all the fields the login form sends over and figured out what it is doing.

In the data file I put my assigned WiFi login username, john (replace it with yours) and my password, which also needs to be replaced with an appropriate value for your situation.

Then I decided to run an attempted authentication every one minute, while running the query to my web server every two minutes. That is what the */2 field does in my crontab. That way I will always have authenticated first, even when my four hours has run out.

I like that this also tests the authentication that has been set up, as this could also be the cause of a failure.

Meanwhile my web server log gets entries like this one every two minutes:

50.17.188.196 - - [26/Apr/2017:15:12:02 -0400] "GET /raspberrypidrj4guest?1493233922 HTTP/1.1" 404 219 "-" "curl/7.26.0"

On the webserver
On the webserver being accessed by the Ras Pi I have this Perl script:

#!/usr/bin/perl
# check if Raspberry Pi on the DrJ guest WiFi is phoning home
# - DrJ 4/26/17
#
# to test good to error transition,
# call with a very small maxDiff, such as 0!
use Getopt::Std;
getopts('m:d'); # maximum allowed time difference
$maxDiff = $opt_m;
$DEBUG = 1 if $opt_d;
unless (defined($maxDiff)) {
  usage();
  exit(1);
}
$monitorName = 'Raspberry Pi phone home';
# access line looks like:
# 96.15.212.173 - - [02/Feb/2013:22:00:02 -0500] "GET /raspberrypidrj4guest?136456789 HTTP/1.1" 200 455 "-" "curl/7.26.0"
$magicString = "raspberrypidrj4guest";
$accessLog = "/var/log/apache202/access.log";
#
# pick up timestamp in access file
$piTime = `grep $magicString $accessLog|tail -1|cut -d\? -f2|cut -d' ' -f1`;
$curTime = time();
chomp($time);
$date = `date`;
chomp($date);
# your PID file is somewhere else. It tells us when Apache was started.
# you could comment out these next lines just to get started with the program
$PID = "/var/run/apache202.pid";
($atime,$mtime,$ctime) = (stat($PID))[8,9,10];
$diff = $curTime - $piTime;
if ($curTime - $ctime < $maxDiff) {
  print "Apache hasn't been running long enough yet to look for something in the log file. Maybe next time\n";
  exit(0);
}
print "magicString, accessLog, piTime, curTime, diff: $magicString, $accessLog, $piTime, $curTime, $diff\n" if $DEBUG;
print "accessLog stat. atime, mtime, ctime: $atime,$mtime,$ctime\n" if $DEBUG;
print "Freshness: $diff s\n";
###############################
sub usage {
  print "usage: $0 -m <maxDiff (seconds)> [-d (debug)]\n";
}

It’s designed to be run by SiteScope as a script monitor. You would run it by hand like this:

> ./timecheck.pl ‐m 300

Freshness: 35 s

If that Freshness time grows too large then the Ras Pi hasn’t been phoning home and you – presumably – have a problem somewhere. /var/log/apache202 happens to be where I have my apache access file on that system.

Conclusion
We showed how to set up a Raspberry Pi to monitor Guest WiFi access on a Cisco Wireless LAN Controller, even though the accounts have to re-authenticated every four hours.

References and related
In the consumer space I do something closely related. I use a Ras Pi at home to monitor whether my Internet connection at home is working. The same phone home concept is used.

Categories
Consumer Tech Network Technologies Raspberry Pi

WAN load-balancing routers

Intro
I got an offer for $20/month broadband access from Centurylink. It got me to thinking, could I somehow use that as a backup connection to my current cable ISP? How would that work? Could I use a Raspberry Pi as a WAN load-balancing router?

The details
Well I’m not sure about using Raspberry Pi. It’s not so simple.

But I just wanted to mention there are solutions out there in the marketplace to this very problem. They’re not that easy to find, hence this article. They’re mostly aimed at small businesses where Internet connectivity is very important, like an Internet cafe.

This Cisco dual WAN router for $157 would do the trick:

https://www.amazon.com/Cisco-Dual-Gigabit-Router-RV042G-NA/dp/B008CWW6VY/ref=pd_cp_147_2?_encoding=UTF8&pd_rd_i=B008CWW6VY&pd_rd_r=5XFRCAG9PT7THJW8BMJZ&pd_rd_w=PQrlm&pd_rd_wg=FUaoX&psc=1&refRID=5XFRCAG9PT7THJW8BMJZ

Or for about the same price, this Linksys Dual WAN router:

https://www.amazon.com/Linksys-Business-Gigabit-Router-LRT224/dp/B00GK640D6/ref=pd_sbs_147_6?_encoding=UTF8&pd_rd_i=B00GK640D6&pd_rd_r=5XFRCAG9PT7THJW8BMJZ&pd_rd_w=rmOWr&pd_rd_wg=FUaoX&psc=1&refRID=5XFRCAG9PT7THJW8BMJZ

Want to go consumer grade and save money? This TP-Link model is only about $85:

https://www.amazon.com/TP-LINK-TL-R480T-Balance-Broadband-Configurable/dp/B002T4D3L8/ref=pd_ybh_a_4?_encoding=UTF8&psc=1&refRID=36DXNVKPFB8MN844NVNP

But it’s ports are only 100 mbps, which is kind of surprising in this day and age.

DIY approach for the intrepid

There now appears to be a Raspberry Pi solution since the advent of RPi 4 and OpenWRT support for ARM. This post is amazingly detailed: Raspberry Pi as a home router. The latest generation of Raspberry Pi… | by Vladimír Záhradník | The Startup | Medium

So the idea there would be to get OpenWRT running on an RPi 4, then explore running multi-WAN on OpenWRT: LEDE/OpenWRT — Setting Up Multi-WAN | by CT WiFi | LEDE/OpenWrt & IoT | Medium

Conclusion
We have identified commercial solutions to the question: can I use two ISPs at home to provide high availability and load-balancing. I have my doubts however and I think running OpenWRT may be the best option.