Category: Raspberry Pi

Perl Python Raspberry Pi Web Site Technologies

Raspberry Pi photo frame using your pictures on your Google Drive

Post author By john
Post date December 22, 2020
No Comments on Raspberry Pi photo frame using your pictures on your Google Drive

Editor’s Note

Please note I am putting all my currently active development and latest updates into this newer post: Raspberry Pi photo frame using your pictures on your Google Drive II

Intro

All my spouse’s digital photo frames are either broken or nearly broken – probably she got them from garage sales. Regardless, they spend 99% of the the time black. Now, since I had bought that Raspberry Pi PiDisplay awhile back, and it is underutilized, and I know a thing or two about linux, I felt I could create a custom photo frame with things I already have lying around – a Raspberry Pi 3, a PiDisplay, and my personal Google Drive. We make a point to copy all our cameras’ pictures onto the Google Drive, which we do the old-fashioned, by-hand way. After 17 years of digital photos we have about 40,000 of them, over 200 GB.

So I also felt obliged to create features you will never have in a commercial product, to make the effort worthwhile. I thought, what about randomly picking a few for display from amongst all the pictures, displaying that subset for a few days, and then moving on to a new randomly selected sample of images, etc? That should produce a nice review of all of them over time, eventually. You need an approach like that because you will never get to the end if you just try to display 40000 images in order!

Equipment

This work was done on a Raspberry Pi 3 running Raspbian Lite (more on that later). I used a display custom-built for the RPi, Amazon.com: Raspberry Pi 7″ Touch Screen Display: Electronics), though I believe any HDMI display would do.

The scripts
Here is the master file which I call master.sh.

Copy Code



#!/bin/sh

# DrJ 8/2019

# call this from cron once a day to refesh random slideshow once a day

RANFILE=”random.list”

NUMFOLDERS=20

DISPLAYFOLDER=”/home/pi/Pictures”

DISPLAYFOLDERTMP=”/home/pi/Picturestmp”

SLEEPINTERVAL=3

DEBUG=1

STARTFOLDER=”MaryDocs/Pictures and videos”
echo “Starting master process at “`date`
rm -rf $DISPLAYFOLDERTMP

mkdir $DISPLAYFOLDERTMP
#listing of all Google drive files starting from the picture root

if [ $DEBUG -eq 1 ]; then echo Listing all files from Google drive; fi

rclone ls remote:”$STARTFOLDER” > files
# filter down to only jpegs, lose the docs folders

if [ $DEBUG -eq 1 ]; then echo Picking out the JPEGs; fi

egrep ‘\.[jJ][pP][eE]?[gG]$’ files |awk ‘$1 > 11000 {$1=””; print substr($0,2)}’|grep -i -v /docs/ > jpegs.list
# throw NUMFOLDERS or so random numbers for picture selection, select triplets of photos by putting

# names into a file

if [ $DEBUG -eq 1 ]; then echo Generate random filename triplets; fi

./random-files.pl -f $NUMFOLDERS -j jpegs.list -r $RANFILE
# copy over these 60 jpegs

if [ $DEBUG -eq 1 ]; then echo Copy over these random files; fi

cat $RANFILE|while read line; do

  rclone copy remote:”${STARTFOLDER}/$line” $DISPLAYFOLDERTMP

  sleep $SLEEPINTERVAL

done
# rotate pics as needed

if [ $DEBUG -eq 1 ]; then echo Rotate the pics which need it; fi

cd $DISPLAYFOLDERTMP; ~/rotate-as-needed.sh

cd ~
# kill any qiv slideshow

if [ $DEBUG -eq 1 ]; then echo Killing old qiv and fbi slideshow; fi

pkill -9 -f qiv

sudo pkill -9 -f fbi

pkill -9 -f m2.pl
# remove old pics

if [ $DEBUG -eq 1 ]; then echo Removing old pictures; fi

rm -rf $DISPLAYFOLDER
mv $DISPLAYFOLDERTMP $DISPLAYFOLDER
#run looping fbi slideshow on these pictures

if [ $DEBUG -eq 1 ]; then echo Start fbi slideshow in background; fi

cd $DISPLAYFOLDER ; nohup ~/m2.pl  >> ~/m2.log 2>&1 &
if [ $DEBUG -eq 1 ]; then echo “And now it is “`date`; fi

I call the following script random-files.pl:

Copy Code


#!/usr/bin/perl
use Getopt::Std;
my %opt=();
getopts("c:df:j:r:",\%opt);
$nofolders = $opt{f} ? $opt{f} : 20;
$DEBUG = $opt{d} ? 1 : 0;
$cutoff = $opt{c} ? $opt{c} : 5;
$cutoffS = 60*$cutoff;
$jpegs = $opt{j} ? $opt{j} : "jpegs.list";
$ranpicfile = $opt{r} ? $opt{r} : "jpegs-random.list";
print "d,f,j,r: $opt{d}, $opt{f}, $opt{j}, $opt{r}\n" if $DEBUG;
open(JPEGS,$jpegs) || die "Cannot open jpegs listing file $jpegs!!\n";
@jpegs = ;
# remove newline character
$nopics = chomp @jpegs;
open(RAN,"> $ranpicfile") || die "Cannot open random picture file $ranpicfile!!\n";
for($i=0;$i<$nofolders;$i++) {
  $t = int(rand($nopics-2));
  print "random number is: $t\n" if $DEBUG;
# a lot of our pics follow this naming convention
# 20160831_090658.jpg
  ($date,$time) = $jpegs[$t] =~ /(\d{8})_(\d{6})/;
  if ($date) {
    print "date, time: $date $time\n" if $DEBUG;
# ensure neighboring picture is at least five minutes different in time
    $iPO = $iP = $diff = 0;
    ($hr,$min,$sec) = $time =~ /(\d\d)(\d\d)(\d\d)/;
    $secs = 3600*$hr + 60*$min + $sec;
    print "Pre-pic logic\n";
    while ($diff < $cutoffS) {
      $iP++;
      $priorPic = $jpegs[$t-$iP];
      $Pdate = $Ptime = 0;
      ($Pdate,$Ptime) = $priorPic =~ /(\d{8})_(\d{6})/;
      ($Phr,$Pmin,$Psec) = $Ptime =~ /(\d\d)(\d\d)(\d\d)/;
      $Psecs = 3600*$Phr + 60*$Pmin + $Psec;
      print "hr,min,sec,Phr,Pmin,Psec: $hr,$min,$sec,$Phr,$Pmin,$Psec\n" if $DEBUG;
      $diff = abs($secs - $Psecs);
      print "diff: $diff\n" if $DEBUG;
# end our search if we happened upon different dates
      $diff = 99999 if $Pdate ne $date;
    }
# post-picture logic - same as pre-picture
    print "Post-pic logic\n";
    $diff = 0;
    while ($diff < $cutoffS) {
      $iPO++;
      $postPic = $jpegs[$t+$iPO];
      $Pdate = $Ptime = 0;
      ($Pdate,$Ptime) = $postPic =~ /(\d{8})_(\d{6})/;
      ($Phr,$Pmin,$Psec) = $Ptime =~ /(\d\d)(\d\d)(\d\d)/;
      $Psecs = 3600*$Phr + 60*$Pmin + $Psec;
      print "hr,min,sec,Phr,Pmin,Psec: $hr,$min,$sec,$Phr,$Pmin,$Psec\n" if $DEBUG;
      $diff = abs($Psecs - $secs);
      print "diff: $diff\n" if $DEBUG;
# end our search if we happened upon different dates
      $diff = 99999 if $Pdate ne $date;
    }
  } else {
    $iP = $iPO = 2;
  }
  $priorPic = $jpegs[$t-$iP];
  $Pic = $jpegs[$t];
  $postPic = $jpegs[$t+$iPO];
  print RAN qq($priorPic
$Pic
$postPic
);
}
close(RAN);

Bunch of simple python scripts

I call this one getinfo.py:

Copy Code



#!/usr/bin/python3

import os,sys

from PIL import Image

from PIL.ExifTags import TAGS
for (tag,value) in Image.open(sys.argv[1])._getexif().items():

        print (‘%s = %s’ % (TAGS.get(tag), value))
print (‘%s = %s’ % (TAGS.get(tag), value))

And here’s rotate.py:

Copy Code



#!/usr/bin/python3

import PIL, os

import sys

from PIL import Image
picture= Image.open(sys.argv[1])
# if orientation is 6, rotate clockwise 90 degrees

picture.rotate(-90,expand=True).save(“rot_” + sys.argv[1])

While here is rotatecc.py:

Copy Code



#!/usr/bin/python3

import PIL, os

import sys

from PIL import Image
picture= Image.open(sys.argv[1])
# if orientation is 8, rotate counterclockwise 90 degrees

picture.rotate(90,expand=True).save(“rot_” + sys.argv[1])

And rotate-as-needed.sh:

Copy Code



#!/bin/sh

# DrJ 12/2020

# some of our downloaded files will be sideways, and fbi doesn’t auto-rotate them as far as I know

# assumption is that are current directory is the one where we want to alter files

ls -1|while read line; do

  echo fileis “$line”

  o=`~/getinfo.py “$line”|grep -ai orientation|awk ‘{print $NF}’`

  echo orientation is $o

  if [ “$o” -eq “6” ]; then

    echo “90 clockwise is needed, o is $o”

# rotate and move it

    ~/rotate.py “$line”

    mv rot_”$line” “$line”

  elif [ “$o” -eq “8” ]; then

    echo “90 counterclock is needed, o is $o”

# rotate and move it

    ~/rotatecc.py “$line”

    mv rot_”$line” “$line”

  fi

don

And finally, m2.pl:

Copy Code


#!/usr/bin/perl
# show the pics ; rotate the screen as needed
# for now, assume the display is in a neutral
# orientation at the start
use Time::HiRes qw(usleep);
$DEBUG = 1;
$delay = 6; # seconds between pics
$mdelay = 200; # milliseconds
$mshow = "$ENV{HOME}/mediashow";
$pNames = "$ENV{HOME}/pNames";
# pics are here
$picsDir = "$ENV{HOME}/Pictures";

chdir($picsDir);
system("ls -1 > $pNames");
# forther massage names
open(TMP,"$pNames");
@lines = ;
foreach (@lines) {
  chomp;
  $filesNullSeparated .= $_ . "\0";
}
open(MS,">$mshow") || die "Cannot open mediashow file $mshow!!\n";
print MS $filesNullSeparated;
close(MS);
print "filesNullSeparated: $filesNullSeparated\n" if $DEBUG;
$cn = @lines;
print "$cn files\n" if $DEBUG;
# throw up a first picture - all black. Trick to make black bckgrd permanent
system("sudo fbi -a --noverbose -T 1 $ENV{HOME}/black.jpg");
system("sudo fbi -a --noverbose -T 1 $ENV{HOME}/black.jpg");
sleep(1);
system("sleep 2; sudo killall fbi");
# start infinitely looping fbi slideshow
for (;;) {
# then start slide show
# shell echo cannot work with null character so we need to use a file to store it
    #system("cat $picNames|xargs -0 qiv -DfRsmi -d $delay \&");
    system("sudo xargs -a $mshow -0 fbi -a --noverbose -1 -T 1  -t $delay ");
# fbi runs in background, then exits, so we need to monitor if it's still alive
# wait appropriate estimated amount of time, then look aggressively for fbi
    sleep($delay*($cn - 2));
    for(;;) {
      open(MON,"ps -ef|grep fbi|grep -v grep|") || die "Cannot launch ps -ef!!\n";
      $match = ;
      if ($match) {
        print "got fbi match\n" if $DEBUG > 1;
        } else {
        print "no fbi match\n" if $DEBUG;
# fbi not found
          last;
      }
      close(MON);
      print "usleeping, noexist is $noexit\n" if $DEBUG > 1;
      usleep($mdelay);
    } # end loop testing if fbi has exited
} # close of infinite loop

You’ll need to make these files executable. Something like this should work:

$ chmod +x *.py *.pl *.sh

My crontab file looks like this (you edit crontab using the crontab -e command):

@reboot sleep 25; cd ~ ; ./m2.pl >> ./m2.log 2>&1
24 16 * * * ./master.sh >> ./master.log 2>&1

This invokes master.sh once a day at 4:24 PM to refresh the 60 photos. My refresh took about 13 minutes the other day, but the old slideshow keeps playing until almost the last second, so it’s OK.

The nice thing about this approach is that fbi works with a lightweight OS – Raspbian Lite is fine, you’ll just need to install a few packages. My SD card is unstable or something, so I have to re-install the OS periodically. An install of Raspberry Pi Lite on my RPi 4 took 11 minutes. Anyway, fbi is installed via:

$ sudo apt-get install fbi

But if your RPi is freshly installed, you may first need to do a

$ sudo apt-get update && sudo apt-get upgrade

python image manipulation

The drawback of this approach, i.e., not using qiv, is that we gotta do some image manipulation, for which python is the best candidate. I’m going by memory. I believe I installed python3, perhaps as sudo apt-get install python3. Then I needed pip3: sudo apt-get install python3-pip. Then I needed to install Pillow using pip3: sudo pip3 install Pillow.

m2.pl refers to a black.jpg file. It’s not a disaster to not have that, but under some circumstances it may help. There it is!

Many of my photos do not have EXIF information, yet they can still be displayed. So for those photos running getinfo.py will produce an error (but the processing of the other photos will continue.)

I was originally rotating the display 90 degrees as needed to display the photos with the using the maximum amount of display real estate. But that all broke when I tried to revive it. And the cheap servo motor was noisy. But folks were pretty impressed when I demoed it, because I did it get it the point where it was indeed working correctly.

Picture selection methodology

There are 20 “folders” (random numbers) of three triplets each. The idea is to give you additional context to help jog your memory. The triplets, with some luck, will often be from the same time period.

I observed how many similar pictures are adjacent to each other amongst our total collection. To avoid identical pictures, I require the pictures to be five minutes apart in time. Well, I cheated. I don’t pull out the timestamp from the EXIF data as I should (at least not yet – future enhancement, perhaps). But I rely on a file-naming convention I notice is common – 20201227_134508.jpg, which basically is a timestamp-encoded name. The last six digits are HHMMSS in case it isn’t clear.

Rclone

You must install the rclone package, sudo apt-get install rclone.

Can you configure rclone on a headless Raspberry Pi?

Indeed you can. I know because I just did it. You enable your Pi for ssh access. Do the rclone config using putty from a Windows 10 system. You’ll get a long Google URL in the course of configuring that you can paste into your browser. You verify it’s you, log into your Google account. Then you get back a url like http://127.0.0.1:5462/another-long-url-string. Well, put that url into your clipboard and in another login window, enter curl clipboard_contents

That’s what I did, not certain it would work, but I saw it go through in my rclone-config window, and that was that!

Don’t want to deal with rclone?

So you want to use a traditional flash drive you plug in to a USB port, just like you have for the commerical photo frames, but you otherwise like my approach of randomizing the picture selection each day? I’m sure that is possible. A mid-level linux person could rip out the rclone stuff I have embedded and replace as needed with filesystem commands. I’m imagining a colossal flash drive with all your tens of thousands of pictures on it where my random selection still adds value. If this post becomes popular enough perhapsI will post exactly how to do it.

Getting started with this

After you’ve done all that, and want to try it out. you can run

$ ./master.sh

First you should see a file called files growing in size – that’s rclone doing its listing. That takes a few minutes. Then it generates random numbers for photo selection – that’s very fast, maybe a second. Then it slowly copies over the selected images to a temporary folder called Picturestmp. That’s the slowest part. If you do a directory listing you should see the number of images in that directory growing slowly, adding maybe three per minute until it reaches 60 of them. Finally the rotation are applied. But even if you didn’t set up your python environment correctly, it doesn’t crash. It effectively skips the rotations. A rotation takes a couple seconds per image. Finally all the images are copied over to the production area, the directory called Pictures; the old slideshow program is “killed,” and the new slideshow starts up. Whole process takes around 15 minutes.

I highly recommend running master.sh by hand as just described to make sure it all works. Probably some of it won’t. I don’t specialize in making recipes, more just guidance. But if you’re feeling really bold you can just power it up and wait a day (because initially you won’t have any pictures in your slideshow) and pray that it all works.

Tip: Undervoltage thunderbolt suppression

This is one of those topics where you’ll find a lot on the Internet, but little about what we need to do: How do we stop that thunderbolt that appears in the upper right corner from appearing?? First, the boilerplate warning. That thingy appears when you’re not delivering enough voltage. That condition can harm your SD Card, blah, blah. I’ve blown up a few SD cards myself. But, in practice, with my RPi 3, I’ve been running it with the Pi Display for 18 months with no mishaps. So, some on, let’s get crazy and suppress the darn thing. So… here goes. To suppress that yellow stroke of lightning, add these lines to your /boot/config.txt:

Copy Code



# suppress undervoltage thunderbolt – DrJ 8/21

# see http://rpf.io/configtxt

avoid_warnings=1

For good measure, if you are not using the HDMI port, you can save some energy by disabling HDMI:

$ tvservice -o

Still missing

I’d like to display a transition image when switching from the current set of photos to the new ones.

Suppressing boot up messages might be nice for some. Personally I think they’re kind of cool – makes it look like you’ve done a lot more techie work than you actually have!

You’re going to get some junk images. I’ve seen where an image is a thumbnail (I guess) and gets blown up full screen so that you see these giant blocks of pixels. I could perhaps magnify those kind of images less.

Movies are going to be tricky so let’s not even go there…

I was thinking about making it a navigation-enabled photo frame, such as integration with a Gameboy controller. You could do some really awesome stuff: Pause this picture; display the location (town or city) where this photo was taken; refresh the slideshow. It sounds fantastical, but I don’t think it’s beyond the capability of even modestly capable hobbyist programmers such as myself.

I may still spin the frame 90 degrees this way an that. I have the servo mounted and ready. Just got to revive the control commands for it.

Appendix 1: rclone configuration

This is my actual rclone configuration session from January 2022.

rclone config
2022/01/17 19:45:36 NOTICE: Config file "/home/pi/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / 1Fichier
\ "fichier"
2 / Alias for an existing remote
\ "alias"
3 / Amazon Drive
\ "amazon cloud drive"
4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, Tencent COS, etc)
\ "s3"
5 / Backblaze B2
\ "b2"
6 / Box
\ "box"
7 / Cache a remote
\ "cache"
8 / Citrix Sharefile
\ "sharefile"
9 / Dropbox
\ "dropbox"
10 / Encrypt/Decrypt a remote
\ "crypt"
11 / FTP Connection
\ "ftp"
12 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage"
13 / Google Drive
\ "drive"
14 / Google Photos
\ "google photos"
15 / Hubic
\ "hubic"
16 / In memory object storage system.
\ "memory"
17 / Jottacloud
\ "jottacloud"
18 / Koofr
\ "koofr"
19 / Local Disk
\ "local"
20 / Mail.ru Cloud
\ "mailru"
21 / Microsoft Azure Blob Storage
\ "azureblob"
22 / Microsoft OneDrive
\ "onedrive"
23 / OpenDrive
\ "opendrive"
24 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
25 / Pcloud
\ "pcloud"
26 / Put.io
\ "putio"
27 / SSH/SFTP Connection
\ "sftp"
28 / Sugarsync
\ "sugarsync"
29 / Transparently chunk/split large files
\ "chunker"
30 / Union merges the contents of several upstream fs
\ "union"
31 / Webdav
\ "webdav"
32 / Yandex Disk
\ "yandex"
33 / http Connection
\ "http"
34 / premiumize.me
\ "premiumizeme"
35 / seafile
\ "seafile"
Storage> 13
** See help for drive backend at: https://rclone.org/drive/ **
Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a string value. Press Enter for the default ("").
client_id>
OAuth Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Full access all files, excluding Application Data Folder.
\ "drive"
2 / Read-only access to file metadata and file contents.
\ "drive.readonly"
/ Access to files created by rclone only.
3 | These are visible in the drive website.
| File authorization is revoked when the user deauthorizes the app.
\ "drive.file"
/ Allows read and write access to the Application Data folder.
4 | This is not visible in the drive website.
\ "drive.appfolder"
/ Allows read-only access to file metadata but
5 | does not allow any access to read or download file content.
\ "drive.metadata.readonly"
scope> 2
ID of the root folder
Leave blank normally.
Fill in to access "Computers" folders (see docs), or for rclone to use
a non root folder as its starting point.
Enter a string value. Press Enter for the default ("").
root_folder_id>
Service Account Credentials JSON file path
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Leading ~ will be expanded in the file name as will environment variables such as ${RCLONE_CONFIG_DIR}.
Enter a string value. Press Enter for the default ("").
service_account_file>
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n>
Remote config
Use auto config?
Say Y if not sure
Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n> N
Please go to the following link: https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=202264815644.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.readonly&state=2K-WjadN98dzSlx3rYOvUA
Log in and authorize rclone for access
Enter verification code> 4/1AX4XfWirusA-gk55nbbEJb8ZU9d_CKx6aPrGQvDJzybeVR9LOWOKtw_c73U
Configure this as a team drive?
y) Yes
n) No (default)
y/n>
[remote]
scope = drive.readonly
token = {"access_token":"ALTEREDARrdaM_TjUIeoKHuEMWCz_llH0DXafWh92qhGy4cYdVZtUv6KcwZYkn4Wmu8g_9hPLNnF1Kg9xoioY4F1ms7i6ZkyFnMxvBcZDaEwEs2CMxjRXpOq2UXtWmqArv2hmfM9VbgtD2myUGTfLkIRlMIIpiovH9d","token_type":"Bearer","refresh_token":"1//0dKDqFMvn3um4CgYIARAAGA0SNwF-L9Iro_UU5LfADTn0K5B61daPaZeDT2gu_0GO4DPP50QoxE65lUi4p7fgQUAbz8P5l_Rcc8I","expiry":"2022-01-17T20:50:38.944524945Z"}
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
remote drive
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
pi@raspberrypi:~ $

References and related

This 7″ display is a little small, but it’s great to get you started. It’s $64 at Amazon: Amazon.com: Raspberry Pi 7″ Touch Screen Display: Electronics

Is your Pi Display mentioned above blanking out after a few seconds? I have just the solution in this post.

I have an older approach using qiv which I lost the files for, and my blog post got corrupted. Hence this new approach.

In this slightly more sophisticated approach, I make a greater effort to separate the photos in time. But I also make a whole bunch of other improvements as well. But it’s a lot more files so it may only be appropriate for a more seasoned RPi command-line user.

My advanced slideshow treatment is beginning to take shape. I just add to it while I develop it, so check it periodically if that is of interest. Raspberry Pi advanced photo frame.

Tags pi display, Raspberry Pi Lite, rclone, slideshow, undervoltage

Consumer Interest Consumer Tech Network Technologies Raspberry Pi

Consumer Tech: Home Internet stopped working

Post author By john
Post date November 26, 2020
No Comments on Consumer Tech: Home Internet stopped working

Intro

We woke up yesterday to no Internet. The usual remedies consumers go through did nothing to resolve the issue. What to do?

The details – November 25, 2020

The usual restarts or my router and the cable modem did not work. I plugged in my work laptop directly to the cable modem for some quick tests but that did not work.

I plugged my work-issued VPN router directly to the cable modem and it did not pick up an IP and re-establish the tunnel.

When I logged into my router I saw that its WAN IP was listed as 0.0.0.0, which means none at all.

I called the ISP twice. Both time they said they could “see” my modem, and they tried to restart it on their end, but that did not seem to do anything at all, based on the constant status LEDs (see picture below). I got my service visit moved up from Dec 11th to Dec 2nd, but still that would mean a week without Internet – not so great when three people are relying on it for their work.

I rebooted the cable modem a couple times at least. Nothing changed.

Then I started some research on quickie alternatives. Ask a friend from work for a spare Cradlepoint air card? They’re already out on vacation. Get a Chinese-made unlocked hotspot with pre-purchased data? Seems fishy, and ultimately expensive. Verizon brand hotspot? We had a borrowed one. Very finicky. And no ethernet ports.

Raspberry Pi + DIY approach?

At one point in the evening, convinced I would have to wait days for for a visit from the cable guy, I rigged up a spare Raspberry Pi to act as a router between a mobile hotspot (a companion tablet to a Verizon phone) and my Linksys router. Why bother? Why not just use the hotspot directly? Mostly because it’s a pain in the rear to reprogram all those Internet of Things devices one has in ones home these days, notably the several Echo Dots, but as well, a wireless printer, a few laptops, Firesticks, tablets, etc. With this approach I keep the WiFi SSID as it was for all those devices. And, it sort of worked! At least I got one Echo Dot to work. I didn’t push my luck. This stuff consumes a lot of data, even when “idle.”

To be continued…

Linksys WRT1200AC status lights – when healthy!

Cable Modem tatus lights – when operating normally

But I am pretty good at troubleshooting. What I know that less experienced people may not is that all the testing I’ve done to that point was not ironclad proof of failure of the cable modem. I know the traditional advice of old is to hook up a laptop directly to the ethernet port and work with it that way. Furthermore the cable company support said that my status lights were reading normally. So, when I tested my work laptop? Are you kidding? That thing has so many problems when I switch between SSIDs due to some new security software – it loves to display the Globe in the system tray, and the only recourse is to reboot. That’s what I was seeing, but notice I said a quickie test? I did not have time to do that reboot and all that. And that work-issued VPN router? I don’t know how that thing really works either. Never having set it up that way I did not trust reading too much into its results (which was essentially an orange status light instead of the usual white).

So when I had more time in the evening, I hooked up a home laptop which I know should work. After a cable modem reboot in fact I did get an IP and could surf the Internet. That was a glimmer of hope. So I put my router back in place. Still it did not pick up an WAN IP address. Still reading 0.0.0.0 for its IP.

Then I put the laptop back, writing down the IP, subnet mask and default gateway. Then I put my router back, switched its WAN mode from DHCP to fixed IP, putting on the exact IP address the laptop had picked up, with correct subnet mask and default gateway. Still it was not working. When the router is not working the WAN status light is sort of orange-ish. It’s white (pictured above) when the WAN link is communicating.

I decided the fault should lie more with my router than anywhere else, and since it wasn’t working and no number of power cycles was changing that situation, I decided that a factory reset is the thing to try. The last thing I could try. I noted the exact name and passwords of my SSIDs, held the reset button for 15 seconds until the status lights flicked out, and let it start up. It went through a start-up process, which i saw after connecting to its default IP of 192.168.1.1. It was clear it was not seeing the cable modem at the point where it should, but it had some very specific advice to try: power off cable modem, wait two minutes, power it back on, and then it would try again. And that did work! Yeah!

What may have precipitated this

My local cable company was recently bought by a much bigger company. I know for a fact what my WAN IP used to be, and I see it has changed. They now draw from a giant pool of IPs – a /14 in CIDR notation – that’s 262,000 addresses – that belongs to the new owner. So I believe the problem occurred due to a poor implementation of the dhcp protocol within my router, or a poor interplay between my router’s DHCP client and the ISP’s DHCP server. But I can’t research that line of troubleshooting because the ISP’s DHCP policies would require a lot of time-consuming experimentation on my part to reverse engineer based on observed behaviour under different conditions. And I would need an open source DHCP client – but I have the Raspberry Pi running dnsmasq for that, so that end could gather all the needed client information.

Prior to this acquisition I would tend to keep the same WAN IP for years – that’s how stable it was.

Another approach

Very germane to this topic is the fact that my neighbor down the street experienced his own Internet outage the day after I did! His solution was to buy a better cable modem. I did not know you could do that – I thought they were proprietary. He also saw his router with the 0.0.0.0 WAN address. And his approach also worked. This makes me less sure my router was really at fault – maybe Altice screwed up their DHCP service for half a day.

Conclusion

Unusual for me, I’m going to write the conclusion before writing the tedious part which is the full explanation in the middle.

By the end of the day I got the Internet working. After isolating the problem to my home router, the Linksys WRT1200AC, and determining that any amount of power cycling was not clearing things up, a factory reset did the trick! The cable modem and my cable Internet service was fine all along.

References and related

How to turn your Raspberry Pi into a router which shares your hotspot with your home router.

The Linksys WRT1200AC is no longer sold. It looks like the newer version is the WRT1900AC – it even looks identical. It’s a good router. I know there are fancier solutions out there, but there are also worse ones as well, so I can only give my qualified endorsement: https://www.amazon.com/Linksys-AC1900-Source-Wireless-WRT1900AC/dp/B014MIBLSA/ref=sr_1_1?dchild=1&keywords=linksys+wrt1200ac&qid=1606519765&sr=8-1

DHCP and CIDR notation are both described in great detail in their respective Wikipedia articles.

Tags Altice, CIDR notation, Cradlepoint, DHCP, factory reset, WRT1200AC

DNS Perl Raspberry Pi

Domain Services: does Backorder work?

Post author By john
Post date October 29, 2020
No Comments on Domain Services: does Backorder work?

Intro

This is a memoir of my personal experience with trying to obtain a DNS domain that was registered by another person and about to expire. Plus some technical discussion of how whois on linux probably works.

The details

I’ve been watching a particular domain for years now. It’s always been registered at auction sites, and has changed hands at least once, maybe even twice. So i was excited this year when it was about to expire at the end of September. I kept checking via linux whois – figuring, or really more like hoping, that a direct query to the authoritative whois server would not tip off the owner if it were done outside of a web page. The linux command is whois -h whois.epik.com drjohnss.com (ok, that is not the real domain, just using it for the sake of preserving anonymitiy).

So about 10 days after it “expired” – at which point I believe it is very easy for the owner to still renew it – I wanted to increase my chances so I decided to make a bid for it, figuring, the owner would face either my offer or the prospect of getting nothing for the domain or shelling out for the renewal. So I offered $150 which is what it’s worth to me.

To my surprise I got a return email:

Hello John,
Thanks for the inquiry.
This seller will not sell for less than $10K. What is your budget?

Christina

Wow, right? Then I thought for a few minutes? I’ve seen this before – at work. There was this no-name domain which matched something the marketing folks were planning, so we made an offer through a third-party service. The response was to the effect, The seller is not interested in selling, but for $47,000 you could buy it. WTF. You can’t make this stuff up. I don’t have a lot of respect for domainers because frankly, almost all my interactions have been negative. Consider the evidence. At work I constantly get unsolicited offers for company_name.nz. The emails always come from different email addresses to avoid spam filters. That is cyber-squatting. Deplorable. I once got an unsolicited offer for a domain similar to one we owned (without the “s”). I checked it and found it wasn’t even registered! So that con artist was trying to take advantage of our naivete. Scum. Then a month ago I was offered some $ for any GoDaddy account which had been registered years ago and so had access to its API auction service, which you apparently cannot get any longer. Sounds like an invitation to violate the terms of service to me – another dodgy tactic.

So I thought about that statement and decided, that’s just a negotiating tactic to make me cower and think unless I raised my offer to, say, $1000, I wouldn’t stand a chance. I decided not to cave. I am the world’s worst negotiator but here I felt I had somewhat a position of strength given my tepid feelings about the domain and the fact that it had officially expired. My – somewhat flip – response:

Hi Christina,
Thanks for the response. Well, I am content to see it expire so the seller gets $0. I know it’s been doing nothing for years now. I am a private person with no commercial interest in development of the domain. My budget is $200.

Christina’s response:

Thanks John.
I hear you.
I advise you to get the refundable exclusive backorder.
Just buy it and then don’t check it.
Regards, Christina

So now this Christina lady sounds like she’s on my side seeing I wasn’t a big bucks buyer. At some point it’s a matter of trust. So I plunk down $200 for their backorder service and wait and don’t check.

Christina sends me this encouraging note:

John,
If you cancel the backorder, the fee is refunded.
And checking WHOIS is data we collect and which the registrant can see.
So, best to wait patiently.
Regards, Christina

She encourages me to be super patient and asks what my plans are for it. My response:

Hi Christina,
Bragging rights at family gatherings, etc.
Then I’ll think about more ambitious things like a private social media site, but I doubt I’ll go there.
Thanks,John

So how did it end up?

Not so good. I eventually broke down and did a single whois check after a couple weeks and found the domain had been renewed. Foiled once again, and out the $200 backorder fee.*

*Technically not out since Christina also said it was refundable. I’m just going to sit on it until next year, and the year after that, …

What is that business model?

I had plenty of days to think about it, and I was trying to square two irreconcilable facts. 1) The seller was going to hold out for big money for a worthless domain, thereby losing money. 2) Yet, presumably, the seller is overall making money. Hmm. So I came up with this hypothesis.

Although to an outsider like myself the seller’s approach is irrational, I have a hypothesis for a business model which could justify it.
My hypothesis for a business model that supports such behavior is that some domainers own hundreds or even thousands of seemingly low-value domains – a domain farm – which they patiently cultivate. In the Internet there is commonly seen the long-tail phenomenon. Chris Anderson described it in a book. So instead of following a normal distribution around the nominal value of an unlikely-sounding domain, the actual value distribution has a long tail on the upside. So, if one owns enough domains, although any one may never get the big offer, it only takes a few big ones a year to hit, make up for all the losers and create positive cash flow. After all a domain is really worth what a buyer is willing to pay, not what the algorithms judge them to be worth. Some people will be willing to pay big.

An industry insider I contacted demurred when asked for confirmation or denial of my hypothesis, but insteadpointed me to this link: https://domaingraduate.com/ . If I understand it correctly, chapter 7, The domain Name Aftermarket, addresses this scenario. But it says it basically doesn’t work the way that I hypothesized. And that plus the other chapters in total present a much, much more complex story. There are business models, of course, but, well, just read it for yourself. I don’t care. I still like my domain farm plus long valuation tail concept.

About whois on linux

I need to investigate further what goes on when a simple whois lookup is done. Like everything, there’s a lot of history and it’s not so straightforward. This somewhat outdated article seems to cover it really well: https://securitytrails.com/blog/whois-lookup . I’m still digesting it myself. I’ve done a trace on port 43 for a whois lookup of drjohnstechtalk.com and see somewhat confounding results – it’s talking to two whois servers, a Verisign one (whois.verisign.com or similar), which provides some minimal information, and one which refuses to provide any information – whois.godaddy.com (GoDaddy is the registrar for this domain). My tenuous conclusion is that whois to Verisign does a static lookup and Verisign has a database which covers all of the .com domains with basic information. More detailed information can be provided by the actual registrar for that domain. But GoDaddy refuses to do that. However, it appears other registrars do accept these requests for details! In particular the registrars which are used by domainers to park their domains. Hence it is entirely possible, even from packet analysis, that a registrar gets tipped off by a linux command-line whois lookup (and therefore could provide metrics back to the registrant about these occurrences.)

Double however

I did still more research on whois, i.e., RTFM type stuff. It looks like there are switches which should turn off lookups on other server, like -r or -R, but when you try them they don’t actually work. But, I enabled verbose mode which shows you the whois servers being queried – no need to do a laborious packet trace – and I discovered that if you run the command this way:

$ whois –verbose -h whois.verisign-grs.com <domain_name>

then the query stays with Verisign’s whois server and there is no data leakage or data sharing with the actual registrar! So, mission accomplished. Note that the Verisign whois server probably only covers .com and .net gTLDs. For others like .io, .us, .info you have to figure out the principal whois server for yourself. Or ask for help in the comments section.

drjwhois makes it easier

I decided to write my own wrapper for whois to make this easier for anyone going down this path. Just bear in mind its limited applicability. It’s aimed at people interested in a domain, probably one on the after market, where they want to know if it’s about to expire or has actually expired, without tipping off the seller. As I said I call it drjwhois.

#!/usr/bin/perl
# DrJ's wrapper for whois - prevents data leakage
# Drj 11/20
$DEBUG = 0;
$domain = lc $ARGV[0];
# These are just the TLDs I consider the most important. Obviously there are thousands. Many do not have a resale market.
#to find the whois server just run whois --verbose
$BIZ = "whois.nic.biz";
$BR = "whois.registro.br";
$CA = "whois.cira.ca";
$CO = "whois.nic.io";
$DE = "whois.denic.de"; # de but whois server does not reveal anything! Must use their web site.
$ENOM = "whois.enom.com"; # biz
$IE = "whois.iedr.ie";
$IN = "whois.registry.in";
$INFO = "whois.afilias.net";
$IO = "whois.nic.io";
$ME = "whois.nic.me";
$ORG = "whois.pir.org";
$RU = "whois.tcinet.ru";
$US = "whois.nic.us";
$Verisign = "whois.verisign-grs.com"; # com, net, edu
%TLDs = ('biz',$BIZ,'br',$BR,'ca',$CA,'com',$Verisign,'me',$ME,'net',$Verisign,'edu',$Verisign,'ie',$IE,'io',$IO,'co',$CO,
'in',$IN,'info',$INFO,'org',$ORG,'ru',$RU,'tv',$ENOM,'us',$US);
if ($DEBUG) {
  foreach $key (keys %TLDs) {
    print $key . " " . $TLDs{"$key"} . "\n";
  }
}
$_ = $domain;
($tld) = /.([^.]+)$/;
print qq(Domain:\t\t$domain
TLD:\t\t$tld
WHOIS server:\t$TLDs{$tld}\n\n);
#$result = whois -h $TLDs{$tld} $domain;
#print $result;
unless ($TLDs{$tld}) {
  print "drjwhois has no information about this TLD. Instead use whois $domain\n";
  exit;
}
open(WHOIS,"whois -h $TLDs{$tld} $domain|") || die "Cannot launch whois -h $TLDs{$tld} $domain!!\n";
while(<WHOIS>) {
  if (/(whois|expir|paid|renewal)/i) {
    print ;
    $exists = 1;
  }
}
print "Domain $domain appears to be unregistered!\n" unless $exists;
print qq(\n\ndrjwhois is designed to only show information about the expiration
date of a domain, and if it has become unregistered, all without
leaking the query to aftermarket sellers such as Sedo, Epik, enom, etc.
If you want full information just use whois $domain
);

Example usage

$ drjwhois johnstechtalk.com

Domain: johnstechtalk.com
TLD:    com
WHOIS server: whois.verisign-grs.com

Registrar WHOIS Server: whois.godaddy.com
Registry Expiry Date: 2021-04-23T00:54:17Z
NOTICE: The expiration date displayed in this record is the date the
currently set to expire. This date does not necessarily reflect the expiration
view the registrar's reported date of expiration for this registration.

drjwhois is designed to only show information about the expiration
date of a domain, and if it has become unregistered, all without
leaking the query to aftermarket sellers such as Sedo, Epik, enom, etc.
If you want full information just use whois johnstechtalk.com

Anyway, I say the write-up is outdated because it’s a lot harder than it was a few years ago to get the registrant information. ICANN was chastened I believe by GDPR (data privacy) concerns and so most of the registrant’s personal details has been yanked, generally speaking. But there are left a few valuable nuggets of information.

How about all those nice web interfaces to whois?

I would personally avoid all the web interfaces registrars offer to whois – they seem to be run by the sales and marketing departments without exception. They almost guarantee data sharing with the registrant in addition to selling you services you don’t want.

Conclusion

My guess is that backorders rarely work out. Mine certainly didn’t. But if you like gambling it has a certain thrill to it since you never know…

If you want to play with the big boys and girls and make some money from buying and selling domains, my impression is that Epik is an honest broker, and that’s important to have when so many are not above coloring outside the lines in this business.

linux whois does indeed provide a way to avoid having your interest in a domain leak out to the owner. Use whois -h whois.verisign-grs.com <domain_name> and you are not giving yourself away.

References and related

An old blog post of mine which describes writing a program to GoDaddy’s api for buying a domain as soon as it becomes available.

Whois – what goes on behind the scenes during a whois lookup: https://securitytrails.com/blog/whois-lookup

Best resource I am aware of which covers the strange virtual world of buying and selling domains for a living.: https://domaingraduate.com/

If you’re dying to try out whois on linux but don’t have access to linux, you could either get a Raspberry Pi, though there is some set up and cost involved there, or install Cygwin on Windows 10, though there is some setup involved in getting the package setup, but at least there’s no cost.

On Centos linux, Raspbian (used by Raspberry Pi) and Cygwin, whois is its own package. On my Centos 8 server it is whois-5.5.1-2.

Tags backorder, Cygwin, domainer, Epik, gTLD, ICANN, whois

Admin Linux Raspberry Pi

Raspberry Pi Recovery Mode or interrupting the boot process

Post author By john
Post date August 26, 2019
1 Comment on Raspberry Pi Recovery Mode or interrupting the boot process

Intro
If you installed Raspbian from the NOOBS distribution as I do, then you may occasionally “blow up” your installation as I just have! You have an out, sort of, short of re-imaging the disk, though about with the same impact.

To interrupt the boot process and enter recovery mode, attach a USB keyboard and repeatedly hit the Shift key. You should come to the NOOBS OS install selection screen. Just re-install Rasbian again… But if you’re using WiFi first configure your WiFi setup before re-installing Raspbian.

Symptoms
When I powered up, I got the initial multi-color screen. Then a two-line text message popped up – too quickly to be read, then a grayish screen, then it split into a lower and upper part, then both halves faded away and there it stayed… At that point it was not responsive to any keyboard inputs or mouse clicks.

Conclusion
While doing my advanced slide show and rotating display project, I somehow managed to blow up my OS. finding the way to interrupt the boot-up was not so easy so I am amplifying the answer that worked for me on the Internet: repeatedly hit the Shift key during the boot, until you see the NOOBS image selector screen.

Tags NOOBS

Linux Network Technologies Raspberry Pi

OLD: Raspberry Pi photo frame using your pictures on your Google Drive

Post author By john
Post date August 6, 2019
1 Comment on OLD: Raspberry Pi photo frame using your pictures on your Google Drive

Intro

This posting is messed up. I’ll have to re-post. Working on it… Try this post instead.

The scripts
Here is the master file which I call master.sh.

#!/bin/sh
# DrJ 8/2019
# call this from cron once a day to refesh random slideshow once a day
RANFILE="random.list"
NUMFOLDERS=20
DISPLAYFOLDER="/home/pi/Pictures"
DISPLAYFOLDERTMP="/home/pi/Picturestmp"
SLEEPINTERVAL=3
DEBUG=1
STARTFOLDER="MaryDocs/Pictures and videos"
 
echo "Starting master process at "`date`
 
rm -rf $DISPLAYFOLDERTMP
mkdir $DISPLAYFOLDERTMP
 
#listing of all Google drive files starting from the picture root
if [ $DEBUG -eq 1 ]; then echo Listing all files from Google drive; fi
rclone ls remote:"$STARTFOLDER" &gt; files
 
# filter down to only jpegs, lose the docs folders
if [ $DEBUG -eq 1 ]; then echo Picking out the JPEGs; fi
egrep '\.[jJ][pP][eE]?[gG]
 
Needless to say, but I'd better say it, the STARTFOLDER in this script is particular to my own Google drive. Customize it as appropriate for your situation.
 
Then qiv (quick image viewer) is called with a bunch of arguments and some trickery to ensure proper display of files with spaces in the filenames (an anathema for Linux but my spouse doesn't know that so I gotta deal with it). I call this script qiv.sh.

#!/bin/sh
# -f : full-screen; -R : disable deletion; -s : slideshow; -d : delay ; -i : status-bar;
# -m : zoom; [-r : ranomdize]
# this doesn't handle filenames with spaces:
##cd /media; qiv -f -R -s -d 5 -i -m `find /media -regex ".+\.jpe?g$"`
# this one does:
export DISPLAY=:0
if [ "$1" = "l" ]; then
# print out proposed filenames
  find . -regex ".+\.[jJ][pP][eE]?[gG]$"
else
# args: f fullscreen d delay s slideshow l autorotate R readonly I statusbar
# i nostatusbar m maxspect
  find . -regex ".+\.[jJ][pP][eE]?[gG]$" -print0|xargs -0 qiv -fRsmil -d 5
fi

Here is the perl script which generates the random numbers and associates them to the file listing we’ve just made with rclone, random-files.pl.

#!/usr/bin/perl
use Getopt::Std;
my %opt=();
getopts("df:j:r:",\%opt);
$nofolders = $opt{f} ? $opt{f} : 20;
$DEBUG = $opt{d} ? 1 : 0;
$jpegs = $opt{j} ? $opt{j} : "jpegs.list";
$ranpicfile = $opt{r} ? $opt{r} : "jpegs-random.list";
print "d,f,j,r: $opt{d}, $opt{f}, $opt{j}, $opt{r}\n" if $DEBUG;
open(JPEGS,$jpegs) || die "Cannot open jpegs listing file $jpegs!!\n";
@jpegs = ;
# remove newline character
$nopics = chomp @jpegs;
open(RAN,"&gt; $ranpicfile") || die "Cannot open random picture file $ranpicfile!!\n";
for($i=0;$i&lt;$nofolders;$i++) {
  $t = int(rand($nopics-2));
  print "random number is: $t\n" if $DEBUG;
  ($dateTime) = $jpegs[$t] =~ /(\d{8}_\d{6})/;
  if ($dateTime) {
    print "dateTime\n" if $DEBUG;
  }
  $priorPic = $jpegs[$t-2];
  $Pic = $jpegs[$t];
  $postPic = $jpegs[$t+2];
  print RAN qq($priorPic
$Pic
$postPic
);
}
close(RAN);

Note that to display 60 pictures only 20 random numbers are used, and then the picture 2 prior and the picture two after the one selected by the random number are also displayed. This helps to provide, hopefully, some context to what is being shown without showing all those duplicate pictures that everyone takes nowadays.

There is an attempt to favor recently uploaded pictures but I really haven’t perfected that part of master.sh, it’s more of a thought at this point.

My crontab entries take care of starting a slideshow upon first boot as well as a daily pick of 60 new random pictures!

@reboot sleep 25; cd ~ ; ./m2.pl &gt;&gt; ./m2.log 2&gt;&amp;1
24 16 * * * ./master.sh &gt;&gt; ./master.log 2&gt;&amp;1

Use crontab -e to edit your crontab file.

qiv – an easy install
To install qiv

$ sudo apt-get install qiv

Rclone shown in some detail
The real magic is tapping into the Google Drive, which is done with rclone. There are older packages but they are awful by comparison so don’t waste your time on any other package. More recent rclone packages offer more options than what is shown here, but work basically the same way.

$ sudo apt-get install rclone
$ rclone config

2019/08/05 20:22:42 NOTICE: Config file "/home/pi/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q&gt; n
name&gt; remote
Type of storage to configure.
Choose a number from below, or type in your own value
 1 / Amazon Drive
   \ "amazon cloud drive"
 2 / Amazon S3 (also Dreamhost, Ceph, Minio)
   \ "s3"
 3 / Backblaze B2
   \ "b2"
 4 / Dropbox
   \ "dropbox"
 5 / Encrypt/Decrypt a remote
   \ "crypt"
 6 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
 7 / Google Drive
   \ "drive"
 8 / Hubic
   \ "hubic"
 9 / Local Disk
   \ "local"
10 / Microsoft OneDrive
   \ "onedrive"
11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
12 / Yandex Disk
   \ "yandex" 
Storage&gt;7
 
Google Application Client Id
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_id&gt;
Google Application Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret&gt;
Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine or Y didn't work
y) Yes
n) No
y/n&gt; N
If your browser doesn't open automatically go to the following link: https://accounts.google.com/o/oauth2/auth?client_id=202264815644.apps.googleusercontent.com&amp;redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&amp;response_type=code&amp;scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&amp;state=07ab6a457efc9384772f919dca93375
Log in and authorize rclone for access

You sign in to your Google account with a regular browser.

After sign-in you see:

rclone wants to access your Google Account
<your_account>@gmail.com
This will allow rclone
to:

See, edit, create, and delete all of your Google Drive files

Make sure you trust rclone

After clicking Allow you get:

Please copy this code, switch to your application and paste it there:
 
Enter verification code&gt;4/nQEXJZOTdP_asMs6UQZ5ucs6ecvoiLPelQbhI76rnuj4sFjptxbjm7w
--------------------
[remote]
client_id =
client_secret =
token = {"access_token":"ya29.Il-KB3eniEpkdUGhwdi8XyZyfBFIF2ahRVQtrr7kR-E2lIExSh3C1j-PAB-JZucL1j9D801Wbh2_OEDHthV2jk_MsrKCMiLSibX7oa_YtFxts-V9CxRRUirF1_kPHi5u_Q","token_type":"Bearer","refresh_token":"1/MQP8jevISJL1iEXH9gaNc7LIsABC-92TpmqwtRJ3zV8","expiry":"2019-09-21T08:34:19.251821011-04:00"}
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d&gt; y
Current remotes:
 
Name                 Type
====                 ====
remote               drive
 
e) Edit existing remote
n) New remote
d) Delete remote
s) Set configuration password
q) Quit config
e/n/d/r/s/q&gt;q

Note you can very well keep the root folder id blank. In my case we store all our pictures in one top-level folder and the nested folders get pretty deep, plus there’s a busload of other things on the drive, so I wanted to give rclone the best possible shot at running well. Still, listing our 40,000+ pictures takes 90 seconds or so.

Goofed up your config of rclone? No worries. Remove .config/rclone and start over.

Don’t forget to make all these scripts executable (chmod +x <script_name>:)or you will end up seeing messages like this:

./master.sh
-bash: ./master.sh: Permission denied

Some noteworthy rclone commands
rclone ls remote: – lists all files, going recursively, no problem with MORE
rclone lsd remote: lists directories in top level of drive
rclone copy remote:”MaryDocs/Pictures and videos/Shutterfly books collection of photos/JJH birth photos/img2165.jpg” . – copies picture to current directory (does not create directory hierarchy)

Do a complete directory listing, capture the results in a file and see how long it took:
$ time rclone ls remote: > lsf-complete

real    1m12.201s
user    0m15.270s
sys     0m1.816s

My initial thought was to do a remote mount of the Google Drive onto a Raspberry Pi mount point, but it’s just so slow that it really provides no advantage to do it that way.

Some encountered issues
Well, I blew up on crontab, which in all my years working with linux/unix I’ve never done before. But I managed to fix it.

Prior to discovering rclone I made the mistake of using gdrivefs to create a mounted Google Drive – sounds great in principle, right? What a disaster. The files’ binary data were not correctly preserved when accessed through the mount though the size was! I have also never encountered a mounting software that corrupted files, but this piece of garbage does. One way to detect corruption in a binary file is to do a cksum (or md5sum, just be consistent and use one or the other) of source file and destination version of same file. The result should be the same number.

Imagined but avoided issue: JPEG orientation
I had prepared a whole python program to orient my pictures correctly, but lo and behold I “discovered” that the -l switch in qiv does that for you! So I actually ripped that whole unnecessary step out.

Conclusion
Re-purposing equipment I had lying around: Raspberry Pi 3, Pi Display, and 40,000 JPEG images on Google Drive, I put together a novel photoframe slideshow which randomly displays a different set of 60 pictures each day. It’s a nice way for us to be exposed to our collection of 17+ years of digital photos.

The qiv really is a quick image viewer, i.e., the slideshow runs clean, like a real one.

Long Todo list

Improve selection of recent pictures if we’ve just uploaded a bunch of pictures from our smartphones.
Hey, how about also showing some of those short videos we also shot with our camera phones and uploaded to Google Drive? And while we’re at it, re-purposing those cheap USB speakers I bought for RetroPi gaming to get the sound, or play a soundtrack!?
I realize that although the selection of the 20 anchor pictures is initially random, when they plus the 40 additional photos are presented for display additional order is imposed by the shell’s expansion of the regex and this has a tendency to make the pictures more chronologically organized than they would be by chance.

References and related
PiDisplay

RetroPi, the gaming emulation project for which I bought economical USB speakers.

The rclone home page.

A detailed write-up on using pipresents program where we had a Raspberry Pi drive a mixed media display 9pictures and videos) for a kiosk.

files |awk '{$1=""; print substr($0,2)}'|grep -i -v /docs/ &gt; jpegs.list # throw NUMFOLDERS or so random numbers for picture selection, select triplets of photos by putting # names into a file if [ $DEBUG -eq 1 ]; then echo Generate random filename triplets; fi ./random-files.pl -f $NUMFOLDERS -j jpegs.list -r $RANFILE # copy over these 60 jpegs if [ $DEBUG -eq 1 ]; then echo Copy over these random files; fi cat $RANFILE|while read line; do rclone copy remote:"${STARTFOLDER}/$line" $DISPLAYFOLDERTMP sleep $SLEEPINTERVAL done # kill any qiv slideshow if [ $DEBUG -eq 1 ]; then echo Killing old qiv slideshow; fi pkill -9 -f qiv # remove old pics if [ $DEBUG -eq 1 ]; then echo Removing old pictures; fi rm -rf $DISPLAYFOLDER mv $DISPLAYFOLDERTMP $DISPLAYFOLDER #run looping qiv slideshow on these pictures if [ $DEBUG -eq 1 ]; then echo Start qiv slideshow in background; fi cd $DISPLAYFOLDER ; nohup ~/qiv.sh &amp; if [ $DEBUG -eq 1 ]; then echo "And now it is "`date`; fi

Needless to say, but I’d better say it, the STARTFOLDER in this script is particular to my own Google drive. Customize it as appropriate for your situation.

Then qiv (quick image viewer) is called with a bunch of arguments and some trickery to ensure proper display of files with spaces in the filenames (an anathema for Linux but my spouse doesn’t know that so I gotta deal with it). I call this script qiv.sh.

Here is the perl script which generates the random numbers and associates them to the file listing we’ve just made with rclone, random-files.pl.

There is an attempt to favor recently uploaded pictures but I really haven’t perfected that part of master.sh, it’s more of a thought at this point.

My crontab entries take care of starting a slideshow upon first boot as well as a daily pick of 60 new random pictures!

Use crontab -e to edit your crontab file.

qiv – an easy install
To install qiv

$ sudo apt-get install qiv

$ sudo apt-get install rclone
$ rclone config

You sign in to your Google account with a regular browser.

After sign-in you see:

rclone wants to access your Google Account
<your_account>@gmail.com
This will allow rclone
to:

See, edit, create, and delete all of your Google Drive files

Make sure you trust rclone

After clicking Allow you get:

Goofed up your config of rclone? No worries. Remove .config/rclone and start over.

Don’t forget to make all these scripts executable (chmod +x <script_name>:)or you will end up seeing messages like this:

Some noteworthy rclone commands
rclone ls remote: – lists all files, going recursively, no problem with MORE
rclone lsd remote: lists directories in top level of drive
rclone copy remote:”MaryDocs/Pictures and videos/Shutterfly books collection of photos/JJH birth photos/img2165.jpg” . : copies picture to current directory (does not create directory hierarchy)

Do a complete directory listing, capture the results in a file and see how long it took:
$ time rclone ls remote: > lsf-complete

My initial thought was to do a remote mount of the Google Drive onto a Raspberry Pi mount point, but it’s just so slow that it really provides no advantage to do it that way.

Some encountered issues
Well, I blew up on crontab, which in all my years working with linux/unix I’ve never done before. But I managed to fix it.

The qiv really is a quick image viewer, i.e., the slideshow runs clean, like a real one.

Long Todo list

Improve selection of recent pictures if we’ve just uploaded a bunch of pictures from our smartphones.
Hey, how about also showing some of those short videos we also shot with our camera phones and uploaded to Google Drive? And while we’re at it, re-purposing those cheap USB speakers I bought for RetroPi gaming to get the sound, or play a soundtrack!?
I realize that although the selection of the 20 anchor pictures is initially random, when they plus the 40 additional photos are presented for display additional order is imposed by the shell’s expansion of the regex and this has a tendency to make the pictures more chronologically organized than they would be by chance.

References and related
PiDisplay

RetroPi, the gaming emulation project for which I bought economical USB speakers.

The rclone home page.

A detailed write-up on using pipresents program where we had a Raspberry Pi drive a mixed media display 9pictures and videos) for a kiosk.

files |awk '{$1=""; print substr($0,2)}'|grep -i -v /docs/ &gt; jpegs.list # throw NUMFOLDERS or so random numbers for picture selection, select triplets of photos by putting # names into a file if [ $DEBUG -eq 1 ]; then echo Generate random filename triplets; fi ./random-files.pl -f $NUMFOLDERS -j jpegs.list -r $RANFILE # copy over these 60 jpegs if [ $DEBUG -eq 1 ]; then echo Copy over these random files; fi cat $RANFILE|while read line; do rclone copy remote:"${STARTFOLDER}/$line" $DISPLAYFOLDERTMP sleep $SLEEPINTERVAL done # rotate pics as needed if [ $DEBUG -eq 1 ]; then echo Rotate the pics which need it; fi cd $DISPLAYFOLDERTMP; ~/rotate-as-needed.sh cd ~ # kill any qiv slideshow if [ $DEBUG -eq 1 ]; then echo Killing old qiv and fbi slideshow; fi pkill -9 -f qiv sudo pkill -9 -f fbi pkill -9 -f m2.pl # remove old pics if [ $DEBUG -eq 1 ]; then echo Removing old pictures; fi rm -rf $DISPLAYFOLDER mv $DISPLAYFOLDERTMP $DISPLAYFOLDER #run looping fbi slideshow on these pictures if [ $DEBUG -eq 1 ]; then echo Start fbi slideshow in background; fi cd $DISPLAYFOLDER ; nohup ~/m2.pl &gt;&gt; ~/m2.log 2&gt;&amp;1 &amp; if [ $DEBUG -eq 1 ]; then echo "And now it is "`date`; fi

Needless to say, but I’d better say it, the STARTFOLDER in this script is particular to my own Google drive. Customize it as appropriate for your situation.

Here is the perl script which generates the random numbers and associates them to the file listing we’ve just made with rclone, random-files.pl.

There is an attempt to favor recently uploaded pictures but I really haven’t perfected that part of master.sh, it’s more of a thought at this point.

My crontab entries take care of starting a slideshow upon first boot as well as a daily pick of 60 new random pictures!

Use crontab -e to edit your crontab file.

qiv – an easy install
To install qiv

$ sudo apt-get install qiv

$ sudo apt-get install rclone
$ rclone config

You sign in to your Google account with a regular browser.

After sign-in you see:

rclone wants to access your Google Account
<your_account>@gmail.com
This will allow rclone
to:

See, edit, create, and delete all of your Google Drive files

Make sure you trust rclone

After clicking Allow you get:

Goofed up your config of rclone? No worries. Remove .config/rclone and start over.

Don’t forget to make all these scripts executable (chmod +x <script_name>:)or you will end up seeing messages like this:

Do a complete directory listing, capture the results in a file and see how long it took:
$ time rclone ls remote: > lsf-complete

My initial thought was to do a remote mount of the Google Drive onto a Raspberry Pi mount point, but it’s just so slow that it really provides no advantage to do it that way.

Some encountered issues
Well, I blew up on crontab, which in all my years working with linux/unix I’ve never done before. But I managed to fix it.

The qiv really is a quick image viewer, i.e., the slideshow runs clean, like a real one.

Long Todo list

Improve selection of recent pictures if we’ve just uploaded a bunch of pictures from our smartphones.
Hey, how about also showing some of those short videos we also shot with our camera phones and uploaded to Google Drive? And while we’re at it, re-purposing those cheap USB speakers I bought for RetroPi gaming to get the sound, or play a soundtrack!?
I realize that although the selection of the 20 anchor pictures is initially random, when they plus the 40 additional photos are presented for display additional order is imposed by the shell’s expansion of the regex and this has a tendency to make the pictures more chronologically organized than they would be by chance.

References and related
PiDisplay

RetroPi, the gaming emulation project for which I bought economical USB speakers.

The rclone home page.

A detailed write-up on using pipresents program where we had a Raspberry Pi drive a mixed media display 9pictures and videos) for a kiosk.

Needless to say, but I’d better say it, the STARTFOLDER in this script is particular to my own Google drive. Customize it as appropriate for your situation.

Here is the perl script which generates the random numbers and associates them to the file listing we’ve just made with rclone, random-files.pl.

There is an attempt to favor recently uploaded pictures but I really haven’t perfected that part of master.sh, it’s more of a thought at this point.

My crontab entries take care of starting a slideshow upon first boot as well as a daily pick of 60 new random pictures!

Use crontab -e to edit your crontab file.

qiv – an easy install
To install qiv

$ sudo apt-get install qiv

$ sudo apt-get install rclone
$ rclone config

You sign in to your Google account with a regular browser.

After sign-in you see:

rclone wants to access your Google Account
<your_account>@gmail.com
This will allow rclone
to:

See, edit, create, and delete all of your Google Drive files

Make sure you trust rclone

After clicking Allow you get:

Goofed up your config of rclone? No worries. Remove .config/rclone and start over.

Don’t forget to make all these scripts executable (chmod +x <script_name>:)or you will end up seeing messages like this:

Some noteworthy rclone commands
rclone ls remote: – lists all files, going recursively, no problem with MORE
rclone lsd remote: lists directories in top level of drive
rclone copy remote:”MaryDocs/Pictures and videos/Shutterfly books collection of photos/JJH birth photos/img2165.jpg” . : copies picture to current directory (does not create directory hierarchy)

Do a complete directory listing, capture the results in a file and see how long it took:
$ time rclone ls remote: > lsf-complete

My initial thought was to do a remote mount of the Google Drive onto a Raspberry Pi mount point, but it’s just so slow that it really provides no advantage to do it that way.

Some encountered issues
Well, I blew up on crontab, which in all my years working with linux/unix I’ve never done before. But I managed to fix it.

The qiv really is a quick image viewer, i.e., the slideshow runs clean, like a real one.

Long Todo list

Improve selection of recent pictures if we’ve just uploaded a bunch of pictures from our smartphones.
Hey, how about also showing some of those short videos we also shot with our camera phones and uploaded to Google Drive? And while we’re at it, re-purposing those cheap USB speakers I bought for RetroPi gaming to get the sound, or play a soundtrack!?
I realize that although the selection of the 20 anchor pictures is initially random, when they plus the 40 additional photos are presented for display additional order is imposed by the shell’s expansion of the regex and this has a tendency to make the pictures more chronologically organized than they would be by chance.

References and related
PiDisplay

RetroPi, the gaming emulation project for which I bought economical USB speakers.

The rclone home page.

A detailed write-up on using pipresents program where we had a Raspberry Pi drive a mixed media display 9pictures and videos) for a kiosk.
files |awk ‘{$1=””; print substr($0,2)}’|grep -i -v /docs/ > jpegs.list

# throw NUMFOLDERS or so random numbers for picture selection, select triplets of photos by putting
# names into a file
if [ $DEBUG -eq 1 ]; then echo Generate random filename triplets; fi
./random-files.pl -f $NUMFOLDERS -j jpegs.list -r $RANFILE

# copy over these 60 jpegs
if [ $DEBUG -eq 1 ]; then echo Copy over these random files; fi
cat $RANFILE|while read line; do
rclone copy remote:”${STARTFOLDER}/$line” $DISPLAYFOLDERTMP
sleep $SLEEPINTERVAL
done

# rotate pics as needed
if [ $DEBUG -eq 1 ]; then echo Rotate the pics which need it; fi
cd $DISPLAYFOLDERTMP; ~/rotate-as-needed.sh
cd ~

# kill any qiv slideshow
if [ $DEBUG -eq 1 ]; then echo Killing old qiv and fbi slideshow; fi
pkill -9 -f qiv
sudo pkill -9 -f fbi
pkill -9 -f m2.pl

# remove old pics
if [ $DEBUG -eq 1 ]; then echo Removing old pictures; fi
rm -rf $DISPLAYFOLDER

mv $DISPLAYFOLDERTMP $DISPLAYFOLDER

#run looping fbi slideshow on these pictures
if [ $DEBUG -eq 1 ]; then echo Start fbi slideshow in background; fi
cd $DISPLAYFOLDER ; nohup ~/m2.pl >> ~/m2.log 2>&1 &

if [ $DEBUG -eq 1 ]; then echo “And now it is “`date`; fi

Needless to say, but I’d better say it, the STARTFOLDER in this script is particular to my own Google drive. Customize it as appropriate for your situation.

Here is the perl script which generates the random numbers and associates them to the file listing we’ve just made with rclone, random-files.pl.

There is an attempt to favor recently uploaded pictures but I really haven’t perfected that part of master.sh, it’s more of a thought at this point.

My crontab entries take care of starting a slideshow upon first boot as well as a daily pick of 60 new random pictures!

Use crontab -e to edit your crontab file.

qiv – an easy install
To install qiv

$ sudo apt-get install qiv

$ sudo apt-get install rclone
$ rclone config

You sign in to your Google account with a regular browser.

After sign-in you see:

rclone wants to access your Google Account
<your_account>@gmail.com
This will allow rclone
to:

See, edit, create, and delete all of your Google Drive files

Make sure you trust rclone

After clicking Allow you get:

Goofed up your config of rclone? No worries. Remove .config/rclone and start over.

Don’t forget to make all these scripts executable (chmod +x <script_name>:)or you will end up seeing messages like this:

Do a complete directory listing, capture the results in a file and see how long it took:
$ time rclone ls remote: > lsf-complete

My initial thought was to do a remote mount of the Google Drive onto a Raspberry Pi mount point, but it’s just so slow that it really provides no advantage to do it that way.

Some encountered issues
Well, I blew up on crontab, which in all my years working with linux/unix I’ve never done before. But I managed to fix it.

The qiv really is a quick image viewer, i.e., the slideshow runs clean, like a real one.

Long Todo list

Improve selection of recent pictures if we’ve just uploaded a bunch of pictures from our smartphones.
Hey, how about also showing some of those short videos we also shot with our camera phones and uploaded to Google Drive? And while we’re at it, re-purposing those cheap USB speakers I bought for RetroPi gaming to get the sound, or play a soundtrack!?
I realize that although the selection of the 20 anchor pictures is initially random, when they plus the 40 additional photos are presented for display additional order is imposed by the shell’s expansion of the regex and this has a tendency to make the pictures more chronologically organized than they would be by chance.

References and related
PiDisplay

RetroPi, the gaming emulation project for which I bought economical USB speakers.

The rclone home page.

A detailed write-up on using pipresents program where we had a Raspberry Pi drive a mixed media display 9pictures and videos) for a kiosk.

Needless to say, but I’d better say it, the STARTFOLDER in this script is particular to my own Google drive. Customize it as appropriate for your situation.

Here is the perl script which generates the random numbers and associates them to the file listing we’ve just made with rclone, random-files.pl.

There is an attempt to favor recently uploaded pictures but I really haven’t perfected that part of master.sh, it’s more of a thought at this point.

My crontab entries take care of starting a slideshow upon first boot as well as a daily pick of 60 new random pictures!

Use crontab -e to edit your crontab file.

qiv – an easy install
To install qiv

$ sudo apt-get install qiv

$ sudo apt-get install rclone
$ rclone config

You sign in to your Google account with a regular browser.

After sign-in you see:

rclone wants to access your Google Account
<your_account>@gmail.com
This will allow rclone
to:

See, edit, create, and delete all of your Google Drive files

Make sure you trust rclone

After clicking Allow you get:

Goofed up your config of rclone? No worries. Remove .config/rclone and start over.

Don’t forget to make all these scripts executable (chmod +x <script_name>:)or you will end up seeing messages like this:

Some noteworthy rclone commands
rclone ls remote: – lists all files, going recursively, no problem with MORE
rclone lsd remote: lists directories in top level of drive
rclone copy remote:”MaryDocs/Pictures and videos/Shutterfly books collection of photos/JJH birth photos/img2165.jpg” . : copies picture to current directory (does not create directory hierarchy)

Do a complete directory listing, capture the results in a file and see how long it took:
$ time rclone ls remote: > lsf-complete

My initial thought was to do a remote mount of the Google Drive onto a Raspberry Pi mount point, but it’s just so slow that it really provides no advantage to do it that way.

Some encountered issues
Well, I blew up on crontab, which in all my years working with linux/unix I’ve never done before. But I managed to fix it.

The qiv really is a quick image viewer, i.e., the slideshow runs clean, like a real one.

Long Todo list

Improve selection of recent pictures if we’ve just uploaded a bunch of pictures from our smartphones.
Hey, how about also showing some of those short videos we also shot with our camera phones and uploaded to Google Drive? And while we’re at it, re-purposing those cheap USB speakers I bought for RetroPi gaming to get the sound, or play a soundtrack!?
I realize that although the selection of the 20 anchor pictures is initially random, when they plus the 40 additional photos are presented for display additional order is imposed by the shell’s expansion of the regex and this has a tendency to make the pictures more chronologically organized than they would be by chance.

References and related
PiDisplay

RetroPi, the gaming emulation project for which I bought economical USB speakers.

The rclone home page.

A detailed write-up on using pipresents program where we had a Raspberry Pi drive a mixed media display 9pictures and videos) for a kiosk.

Needless to say, but I’d better say it, the STARTFOLDER in this script is particular to my own Google drive. Customize it as appropriate for your situation.

Here is the perl script which generates the random numbers and associates them to the file listing we’ve just made with rclone, random-files.pl.

There is an attempt to favor recently uploaded pictures but I really haven’t perfected that part of master.sh, it’s more of a thought at this point.

My crontab entries take care of starting a slideshow upon first boot as well as a daily pick of 60 new random pictures!

Use crontab -e to edit your crontab file.

qiv – an easy install
To install qiv

$ sudo apt-get install qiv

$ sudo apt-get install rclone
$ rclone config

You sign in to your Google account with a regular browser.

After sign-in you see:

rclone wants to access your Google Account
<your_account>@gmail.com
This will allow rclone
to:

See, edit, create, and delete all of your Google Drive files

Make sure you trust rclone

After clicking Allow you get:

Goofed up your config of rclone? No worries. Remove .config/rclone and start over.

Don’t forget to make all these scripts executable (chmod +x <script_name>:)or you will end up seeing messages like this:

Do a complete directory listing, capture the results in a file and see how long it took:
$ time rclone ls remote: > lsf-complete

My initial thought was to do a remote mount of the Google Drive onto a Raspberry Pi mount point, but it’s just so slow that it really provides no advantage to do it that way.

Some encountered issues
Well, I blew up on crontab, which in all my years working with linux/unix I’ve never done before. But I managed to fix it.

The qiv really is a quick image viewer, i.e., the slideshow runs clean, like a real one.

Long Todo list

Improve selection of recent pictures if we’ve just uploaded a bunch of pictures from our smartphones.
Hey, how about also showing some of those short videos we also shot with our camera phones and uploaded to Google Drive? And while we’re at it, re-purposing those cheap USB speakers I bought for RetroPi gaming to get the sound, or play a soundtrack!?
I realize that although the selection of the 20 anchor pictures is initially random, when they plus the 40 additional photos are presented for display additional order is imposed by the shell’s expansion of the regex and this has a tendency to make the pictures more chronologically organized than they would be by chance.

References and related

Current approach and writeup for this photo frame effort.
PiDisplay

RetroPi, the gaming emulation project for which I bought economical USB speakers.

The rclone home page.

A detailed write-up on using pipresents program where we had a Raspberry Pi drive a mixed media display 9pictures and videos) for a kiosk.

Needless to say, but I’d better say it, the STARTFOLDER in this script is particular to my own Google drive. Customize it as appropriate for your situation.

Here is the perl script which generates the random numbers and associates them to the file listing we’ve just made with rclone, random-files.pl.

There is an attempt to favor recently uploaded pictures but I really haven’t perfected that part of master.sh, it’s more of a thought at this point.

My crontab entries take care of starting a slideshow upon first boot as well as a daily pick of 60 new random pictures!

Use crontab -e to edit your crontab file.

qiv – an easy install
To install qiv

$ sudo apt-get install qiv

$ sudo apt-get install rclone
$ rclone config

You sign in to your Google account with a regular browser.

After sign-in you see:

rclone wants to access your Google Account
<your_account>@gmail.com
This will allow rclone
to:

See, edit, create, and delete all of your Google Drive files

Make sure you trust rclone

After clicking Allow you get:

Goofed up your config of rclone? No worries. Remove .config/rclone and start over.

Don’t forget to make all these scripts executable (chmod +x <script_name>:)or you will end up seeing messages like this:

Some noteworthy rclone commands
rclone ls remote: – lists all files, going recursively, no problem with MORE
rclone lsd remote: lists directories in top level of drive
rclone copy remote:”MaryDocs/Pictures and videos/Shutterfly books collection of photos/JJH birth photos/img2165.jpg” . : copies picture to current directory (does not create directory hierarchy)

Do a complete directory listing, capture the results in a file and see how long it took:
$ time rclone ls remote: > lsf-complete

My initial thought was to do a remote mount of the Google Drive onto a Raspberry Pi mount point, but it’s just so slow that it really provides no advantage to do it that way.

Some encountered issues
Well, I blew up on crontab, which in all my years working with linux/unix I’ve never done before. But I managed to fix it.

The qiv really is a quick image viewer, i.e., the slideshow runs clean, like a real one.

Long Todo list

Improve selection of recent pictures if we’ve just uploaded a bunch of pictures from our smartphones.
Hey, how about also showing some of those short videos we also shot with our camera phones and uploaded to Google Drive? And while we’re at it, re-purposing those cheap USB speakers I bought for RetroPi gaming to get the sound, or play a soundtrack!?
I realize that although the selection of the 20 anchor pictures is initially random, when they plus the 40 additional photos are presented for display additional order is imposed by the shell’s expansion of the regex and this has a tendency to make the pictures more chronologically organized than they would be by chance.

References and related
PiDisplay

RetroPi, the gaming emulation project for which I bought economical USB speakers.

The rclone home page.

A detailed write-up on using pipresents program where we had a Raspberry Pi drive a mixed media display 9pictures and videos) for a kiosk.

Tags Exif tags, Google Drive, image orientation, rclone

Admin Linux Network Technologies Raspberry Pi Security Web Site Technologies

How to test if a web site requires a client certificate

Post author By john
Post date June 20, 2019
2 Comments on How to test if a web site requires a client certificate

Intro
I can not find a link on the Internet for this, yet I think some admins would appreciate a relatively simple test to know is this a web site which requires a client certificate to work? The errors generated in a browser may be very generic in these situations. I see many ways to offer help, from a recipe to a tool to some pointers. I’m not yet sure how I want to proceed!

why would a site require a client CERT? Most likely as a form of client authentication.

Pointers for the DIY crowd
Badssl.com plus access to a linux command line – such as using a Raspberry Pi I so often write about – will do it for you guys.

The Client Certificate section of badssl.com has most of what you need. The page is getting big, look for this:

So as a big timesaver badssl.com has created a client certificate for you which you can use to test with. Download it as follows.

Go to your linux prompt and do something like this:
$ wget https://badssl.com/certs/badssl.com‐client.pem

If this link does not work, navigate to it starting from this link: https://badssl.com/download/

badssl.com has a web page you can test with which only shows success if you access it using a client certificate, https://client.badssl.com/

to see how this works, try to access it the usual way, without supplying a client CERT:

$ curl ‐i ‐k https://client.badssl.com/

HTTP/1.1 400 Bad Request
Server: nginx/1.10.3 (Ubuntu)
Date: Thu, 20 Jun 2019 17:53:38 GMT
Content-Type: text/html
Content-Length: 262
Connection: close

400 Bad Request

No required SSL certificate was sent

nginx/1.10.3 (Ubuntu)

Now try the same thing, this time using the client CERT you just downloaded:

$ curl ‐v ‐i ‐k ‐E ./badssl.com‐client.pem:badssl.com https://client.badssl.com/

* About to connect() to client.badssl.com port 443 (#0)
*   Trying 104.154.89.105... connected
* Connected to client.badssl.com (104.154.89.105) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* NSS: client certificate from file
*       subject: CN=BadSSL Client Certificate,O=BadSSL,L=San Francisco,ST=California,C=US
*       start date: Nov 16 05:36:33 2017 GMT
*       expire date: Nov 16 05:36:33 2019 GMT
*       common name: BadSSL Client Certificate
*       issuer: CN=BadSSL Client Root Certificate Authority,O=BadSSL,L=San Francisco,ST=California,C=US
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=*.badssl.com,O=Lucas Garron,L=Walnut Creek,ST=California,C=US
*       start date: Mar 18 00:00:00 2017 GMT
*       expire date: Mar 25 12:00:00 2020 GMT
*       common name: *.badssl.com
*       issuer: CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US
&gt; GET / HTTP/1.1
&gt; User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
&gt; Host: client.badssl.com
&gt; Accept: */*
&gt;
&lt; HTTP/1.1 200 OK
HTTP/1.1 200 OK
&lt; Server: nginx/1.10.3 (Ubuntu)
Server: nginx/1.10.3 (Ubuntu)
&lt; Date: Thu, 20 Jun 2019 17:59:08 GMT
Date: Thu, 20 Jun 2019 17:59:08 GMT
&lt; Content-Type: text/html
Content-Type: text/html
&lt; Content-Length: 662
Content-Length: 662
&lt; Last-Modified: Wed, 12 Jun 2019 15:43:39 GMT
Last-Modified: Wed, 12 Jun 2019 15:43:39 GMT
&lt; Connection: keep-alive
Connection: keep-alive
&lt; ETag: "5d011dab-296"
ETag: "5d011dab-296"
&lt; Cache-Control: no-store
Cache-Control: no-store
&lt; Accept-Ranges: bytes
Accept-Ranges: bytes
 
&lt;
 
 
 
 
  <style>body { background: green; }</style>

client.
badssl.com

 
* Connection #0 to host client.badssl.com left intact
* Closing connection #0

No more 400 error status – that looks like success to me. Note that we had to provide the password for our client CERT, which they kindly provided as badssl.com

Here’s an example of a real site which requires client CERTs:

$ curl ‐v ‐i ‐k ‐E ./badssl.com‐client.pem:badssl.com https://jp.nissan.biz/

* About to connect() to jp.nissan.biz port 443 (#0)
*   Trying 150.63.252.1... connected
* Connected to jp.nissan.biz (150.63.252.1) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* NSS: client certificate from file
*       subject: CN=BadSSL Client Certificate,O=BadSSL,L=San Francisco,ST=California,C=US
*       start date: Nov 16 05:36:33 2017 GMT
*       expire date: Nov 16 05:36:33 2019 GMT
*       common name: BadSSL Client Certificate
*       issuer: CN=BadSSL Client Root Certificate Authority,O=BadSSL,L=San Francisco,ST=California,C=US
* NSS error -12227
* Closing connection #0
* SSL connect error
curl: (35) SSL connect error

OK, so you get an error, but that’s to be expected because our certificate is not one it will accept.

The point is that if you don’t send it a certificate at all, you get a different error:

$ curl ‐v ‐i ‐k https://jp.nissan.biz/

* About to connect() to client.badssl.com port 443 (#0)
*   Trying 104.154.89.105... connected
* Connected to client.badssl.com (104.154.89.105) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* Unable to load client key -8025.
* NSS error -8025
* Closing connection #0
curl: (58) Unable to load client key -8025.

Chrome gives a fairly intelligible error

Possibly to be continued…

Conclusion
We have given a recipe for testing form a linux command line if a web site requires a client certificate or not. thus it could be turned into a program

References and related
My article about ciphers has been popular.

I’ve also used badssl.com for other related tests.

Can you use openssl directly? You’d hope so, but I haven’t had time to explore it… Here are my all-time favorite openssl commands.

https://badssl.com/ – lots of cool tests here. The creators have been really thorough.

Tags client certificate

Linux Network Technologies Raspberry Pi

Live stream to YouTube from a Raspberry Pi + webcam or USB microphone

Post author By john
Post date April 3, 2019
4 Comments on Live stream to YouTube from a Raspberry Pi + webcam or USB microphone

2021 pre-cap

I haven’t run this in two years. I just revived my scripts today, resuscitated an old Raspberry Pi, and voila, it’s pretty much good to go. When I moved to my new server in 2020 some of the scripts got mangled. I’m trying to fix those. For the record, I use the continuousaudio.sh script + ffmpegwireless6.sh mentioned below. That gives me a live audio stream on YouTube with a solid gray video. I’m researching further enhancements and will post those if I ever manage to pull them off.

What I would like to do is to have it interpret and respond to a couple audio commands as in “start recording” and “stop recording”. Now whether or not I can pull that off is an entirely different story. But you’ll never know if you don’t try…

Intro
I’ve been looking at this off and on for awhile now. I finally made a breakthrough this week and started to generate some decent live streams on my YouTube channel, after a lot of misfires.

Note this is applicable for Raspbian Stretch Lite on a Raspberry Pi 3. However, I firmly believe it will work just the same for regular Raspbian Stretch.

There’s a lot of wrong, misleading or outdated information out there on the Internet. Hopefully this will help others to avoid wasting as much time as I had to do.

This project was prompted by my desire to make a more generalized fishcam! Described in this post, my original fishcam implementation – and I realized this form the get-go – has very limited applicability because very few people are in a position to have their own AWS server. And if you don’t know what you’re doing, please don’t run your own server – the security exposure is too great.

So I eventually realized that maybe I could generalize what I had done – essentially remove the dependency on the AWS server – by utilizing Youtube Live Streaming. And, I believe I was right. It’s still a work in progress however.

The command – ffmpeg
I was playing with ffmpeg. The version I am playing with now comes with Raspbian – no need to compile like in the bad old days. ffmpeg -version shows the version to be 3.2.12. I get the impression that its capabilities are version-dependent, so that’s why this information is particularly relevant in this case.

The details
In some of my early attempts I was getting a lot of this (looking at YouTube Live Dashboard)

Dashboard When stream is not quite right

Another attempt
Video works, audio like driving in a car with the windows down. For the record, the command was this:

ffmpeg \
-f alsa -i plughw:CARD=U0x46d0x825,DEV=0 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 2500k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 2 -qscale 3 \
-b:a 96K \
-r 10 \
-s 1280x720 \
-f flv rtmp://a.rtmp.youtube.com/live2/KEY

For the record, the bandwidth required was about 2100 kbps.

List the formats your video device supports

ffmpeg -f video4linux2 -list_formats all -i /dev/video0

Results using my Logitech Webcam

[video4linux2,v4l2 @ 0xcc45c0] Raw       :     yuyv422 :           YUYV 4:2:2 : 640x480 160x120 176x144 320x176 320x240 352x288 432x240 544x288 640x360 752x416 800x448 800x600 864x480 960x544 960x720 1024x576 1184x656 1280x720 1280x960
[video4linux2,v4l2 @ 0xcc45c0] Compressed:       mjpeg :          Motion-JPEG : 640x480 160x120 176x144 320x176 320x240 352x288 432x240 544x288 640x360 752x416 800x448 800x600 864x480 960x544 960x720 1024x576 1184x656 1280x720 1280x960

ffmpeg \
-f alsa -i plughw:CARD=U0x46d0x825,DEV=0 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 1200 \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 2 -qscale 3 \
-b:a 128K \
-r 5 \
-s 640x480 \
-f flv rtmp://a.rtmp.youtube.com/live2/KEY

Audio good, video not working

It is not so pretty to use that hardware address for the Logitech webcam device. Where do you see that hardware address? Either a lsusb or a ls /dev/snd/by-id shows addresses of sound devices. I found a simpler substitute:

ffmpeg \
-f alsa -i plughw:1,0 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 1200k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 2 -qscale 3 \
-b:a 128k \
-r 5 \
-s 640x480 \
-f flv rtmp://a.rtmp.youtube.com/live2/

With this audio's, not too bad, video's a bit choppy. Google reports the stream quality as OK, check resolution.


So I fix the bandwidth (which was a typo in the above, but one with an interesting result). I set video bandwidth to -b:v 1200k. Now the video is OK once again, but the audio is choppy again! Weird. bandwidth is about 1100 kbps.

This version had OK video and OK audio

ffmpeg \
-f alsa -i plughw:CARD=U0x46d0x825,DEV=0 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 1600k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 2 -qscale 3 \
-b:a 128k \
-r 5 \
-s 640x480 \
-f flv rtmp://a.rtmp.youtube.com/live2/KEY

But I keep getting inconsistent results! Sometimes a setting will work, and then I come back to it and it doesn’t. Weird.

Part of the problem is that I have no idea what I’m doing and I didn’t know when i was watching a livestream vs a recorded (on-demand) one! I have since learned to look for the little red Live button. A picture is worth 10^3 words in this case.

[Pic no longer available – try the thousand words instead!]

Observed used bandwidth is about 1450 kbits/sec. But still lots of dropped packets. Here is what ffmpeg reports. I’m not sure yet what most of it means:

[alsa @ 0x1502700] ALSA buffer xrun.
[alsa @ 0x1502700] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)
frame= 5828 fps=5.0 q=-1.0 Lsize=  205496kB time=00:19:26.20 bitrate=1443.5kbits/s dup=0 drop=11138 speed=   1x
video:187265kB audio:17449kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.382063%
[libx264 @ 0x15100e0] frame I:583   Avg QP: 9.41  size: 53819
[libx264 @ 0x15100e0] frame P:5245  Avg QP:13.53  size: 30578
[libx264 @ 0x15100e0] mb I  I16..4: 100.0%  0.0%  0.0%
[libx264 @ 0x15100e0] mb P  I16..4: 38.0%  0.0%  0.0%  P16..4: 60.7%  0.0%  0.0%  0.0%  0.0%    skip: 1.4%
[libx264 @ 0x15100e0] coded y,uvDC,uvAC intra: 93.7% 86.2% 82.4% inter: 77.8% 60.5% 34.1%
[libx264 @ 0x15100e0] i16 v,h,dc,p: 17% 23% 15% 45%
[libx264 @ 0x15100e0] i8c dc,h,v,p: 51% 21% 16% 11%
[libx264 @ 0x15100e0] kb/s:1315.22

The video for that run is here: https://youtu.be/oxJaZv0frGM

Suppressing Audio
This is what worked for me.

ffmpeg \
-f lavfi -i anullsrc=channel_layout=stereo:sample_rate=44100 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 1600k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 2 -qscale 3 \
-b:a 128k \
-r 5 \
-s 640x480 \
-f flv rtmp://a.rtmp.youtube.com/live2/KEY

That is working great – showing the video as before but now with a silent audio track.

Increase Video Quality
Here I’ve increased video quality a tad by requesting more fps (10) and making qscale 0 (which means highest quality).
https://www.youtube.com/watch?v=5Aall8w4Y3E

ffmpeg \
-f alsa -i plughw:1,0 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -b 3000k -g 20 -b:v 1800k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 4 -qscale 0 \
-b:a 128k \
-r 10 \
-s 640x480 \
-f flv rtmp://a.rtmp.youtube.com/live2/KEY

Bitrate was about 1700 kbps. Quality is maybe a little better. Audio still leaves something to be desired.

Still better video quality

ffmpeg \
-f alsa -i plughw:1,0 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -b 3000k -g 60 -b:v 2000k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 4 -qscale 0 \
-b:a 128k \
-r 30 \
-s 640x480 \
-f flv rtmp://a.rtmp.youtube.com/live2/KEY

What is observed to happen is that ffmpeg actually chooses 15 fps rather than 30. I’ve read it decides what it is able to do, so maybe that’s the highest fps it can deliver. Video is pretty smooth (See my Livestream link in references if I happen to have it running. Otherwise I will create a video link.) No drops are recorded, but the sound, though not terrible, has some pops. Bandwidth used is about 1900 kbps. So this is definitely my best effort yet. YouTube complains about the unsupported video size of 640×480, but it permits it and I don’t think it’s a real problem.

Reducing bandwidth
This one is pretty good overall. I have no idea why lowering the audio bandwidth might help. It’s counter intuitive. But video motion is not bad – just a tad blurred. I guess q=23. Audio has good patches and not-as good patches. Not as good spots are staticky, not washboard bad. Total bandwidth used is about 611 kbps. So a great compromise. Why does raising the video bandwidth lower the audio quality? I have no idea… The settings below worked for maybe 20 minutes, then YouTube said this Video is unavailable. I at least found out something about that. That shows a problem with the player, not (for once) your stream. so since I’m only concentrating on the stream, that’s good news. So actually it delivered good sound for three hours straight with a few staticky spots.

ffmpeg \
-thread_queue_size 1024 \
-f alsa -i plughw:1,0 \
-thread_queue_size 256 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 30 -b:v 450k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 4 -q:v 5 \
-q:a 0 \
-b:a 64k \
-r 15 \
-s 480x320 \
-f flv rtmp://a.rtmp.youtube.com/live2/KEY

The audio is creepily sensitive, easily picking up conversations in adjacent rooms.

But then I monkeyed around with the settings, got the washboard sound, came back to this one – a known good – and got washboard audio! What the heck? Why isn’t it consistent?? No idea… Maybe it’s the player that gets messed up?? Now I’m running it again and it’s OK.

Bandwidth talk
It’s important to talk about bandwidth if you haven’t given this any real thought. You have to have a halfway decent broadband connection for this to work, you see? If you have a mid-speed cable modem or DSL, you have much lower upload than download speeds, and you may not be able to pull off a reliable 1.5 mbps upload. For those lucky enough to have Verizon FIOS this is a non-issue. But for instance in the high school where I volunteer they have throttled the guest WiFi network to such an extent that achieving this modest 1.5 mbps is going to present a real challenge. If you rely on a phone’s hotspot you will also probably be unable to get such a speed. So I may look at more ways to reduce the bandwidth required in the future.

Check your bandwidth using speedcheck.org.

And between YouTube and your ISP, it just seems the whole thing about live video broadcasting seems, well, delicate. Stream Health varies between oK, to Excellent to not receiving – all during the same streaming session! It often takes five minutes or so for the stream to appear to be working.

Comparing two webcams
Someone picked up a really cheap DI Chatcam at Microcenter in Paterson. I think that’s Digital Innovations Chatcam. It’s cute. It has a big clip on the end and shines white LEDs when it’s on. I think it was about $12. With the exact same ffmpeg settings (with audio suppressed), the quality was not nearly as good as with the Logitech webcam. Here’s a link to the YouTube video made with the chatcam: https://www.youtube.com/watch?v=OI2IRV1i__k. Note that it has a ministereo plug for audio. I didn;’t even plug it in now that I know how to suppress audio!

The Logitech model is a C525. It was a refurbished model which cost me about $27.The comparable Logitech webcam test is here: https://www.youtube.com/watch?v=L7ZYaRJR7mQ

I need to re-run this test now that I know how to increase the video quality.

A breakthrough: publishing an audio-only stream to YouTube
Besides covering your lens with tape, what’s a software way to blacken the video and concentrate on producing the best audio I wondered?

ffmpeg \
-thread_queue_size 4096 \
-f alsa -i plughw:1,0 \
-thread_queue_size 128 \
-f lavfi -i color=color=darkgray \
-c:v libx264 -pix_fmt yuv420p -b:v 100k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 8 \
-b:a 128k \
-r 30 \
-s 1280x720 \
-f flv rtmp://a.rtmp.youtube.com/live2/KEY

The above gives me good audio, and a sold gray background. I love it – for recording band practice or whatever. The breakthrough is that we can avoid wasting cpu cycles on processing input video but just use a color. Thanks Stackoverflow for the tip. Used bandwidth is about 150 kbs – basically nothing! YouTube Dsahboard complains:

OK Video output low
The stream's current bitrate (138.00 Kbps) is lower than the recommended bitrate. 
We recommend that you use a stream bitrate of 2500 Kbps.

But of course that is bogus because that assumes we are trying to put out a rich 1280×720 video, which we are not.

Then eventually YouTube has this complaint:

Bad Bad video settings
Please use a keyframe frequency of four seconds or less. Currently, keyframes are not being sent often enough, which will cause buffering. 
The current keyframe frequency is 8.5 seconds. Note that ingestion errors can cause incorrect GOP (group of pictures) sizes.

Yet the stream does not seem to suffer in any noticeable way from this problem.

For good measure, we add a few extra arguments allow us to remove the keyframes warning. We need to use the -g parameter (group of pictures) at about twice our frame rate, plus, maybe, a no-scenecut argument. Here’s that version.

ffmpeg \
-thread_queue_size 4096 \
-f alsa -i plughw:1,0 \
-thread_queue_size 128 \
-f lavfi -i color=color=darkgray \
-c:v libx264 -pix_fmt yuv420p -g 60  -x264opts no-scenecut -b:v 150k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 8 \
-b:a 128k \
-r 30 \
-s 1280x720 \
-f flv rtmp://a.rtmp.youtube.com/live2/KEY

Actual fps is 25, quality is 26 and bitrate is 145 kbps. But audio quality is good. I hear white noise in the background, but hey, this isn’t exactly professional equipment we’re working with. But this is a great solution for an audio-only recording that goes straight out to YouTube. stability is also good.

The load average is high – 3.6 (use top to watch it), almost all of it taken by ffmpeg. So it appears ffmpeg is really working it to produce this audio stream. That makes me suspect it just gets overwhelmed when it’s an audio + video stream? Because I never did find setting swhich produced good quality for both…

Switch to Wifi and Yet another problem surfaces
It seems that with this livestreaming project everything that should just work doesn’t! I had been doing all my testing used wired Ethernet connection and WiFi disabled. anticipating a portable solution, I tried it using WiFi and no Ethernet cable. And washboard audio reappeared. quite often ffmpeg hangs as well. I tried a zillion experiments and now my revelation is that essentially, though we tried to minimize and trivialize video, we were probably still overwhelming the CPU. So I reasoned that these actions will make the load easier on the CPU, without compromising the audio quality:

– reduce frame per second dramatically
– reduce key frames
– reduce video size

And…yes, these things in combination really did help and permit me to run over WiFi now. This version, put inside a script I call ffmpegwireless6.sh, looks like this:

Copy Code

#!/bin/sh KEY=691w-uh0z-kx59-c6a7-5xqi # example key ffmpeg \ -thread_queue_size 4096 \ -f alsa -i plughw:1,0 \ -thread_queue_size 64 \ -f lavfi -i color=color=darkgray \ -c:v libx264 -pix_fmt yuv420p -g 18 -x264opts no-scenecut -b:v 50k \ -bufsize 512k \ -acodec libmp3lame -ar 44100 \ -threads 8 \ -b:a 128k \ -r 5 \ -s 480x320 \ -f flv rtmp://a.rtmp.youtube.com/live2/$KEY

It doesn’t start consistently, however, but if you run it enough times it’ll go. So, to provide reliability I also scripted around these deficiencies: I decided to just keep trying to start up ffmpegwireless.sh until I jhave evidence it’s working. I call that script masterwireless.sh:

Copy Code


#!/bin/sh 
# DrJ 5/2019 
LOG="ff.log"`date +%m-%d-%y:%H:%M` 
while /bin/true; do 
nohup ./ffmpegwireless6.sh$LOG 2>&1 & 
sleep 7 
# want s.th like 
#Frame= 84 fps= 11 q=16.0 size= 43kB time=00:00:07.50 bitrate= 47.1kbits/s dup=0 drop=431 speed=0.991x 
#Frame= 84 fps= 11 q=16.0 size= 43kB time=00:00:07.50 bitrate= 47.1kbits/s dup=0 drop=431 speed= 1x 
FFOUT=`tail -1 $LOG` 
echo "last line is $FFOUT" 
KB=`echo $FFOUT|awk '{print $(NF-4)}'` 
echo "orig KB: $KB" 
KB=`echo $FFOUT|awk '{print $(NF-5)" "$(NF-4)}'|sed 's/kbits.*//'|awk '{print $NF}'` 
date 
echo "KB is: $KB" 
if [ $KB -gt 129 2>/dev/null ]; then 
# let our master process exit - we've got a good audio stream 
  echo "Exiting at *** "`date` 
  exit 
else 
# didn't work out: restart and try again 
  echo "*** Restarting ffmpeg at *** "`date` 
  pkill -9 -f 'ffmpeg ' 
fi 
done

And…it works great! Very briefly what it does is t that it calls ffmpegwireless6.sh and backgrounds it, then tests its output. It gives it a few seconds to get going, then kills it unless observed streaming bandwidth is a healthy 135 kbps or so (essentially the video takes almost no bandwidth in ffmpegwireless6.sh.)

Putting it all together – livestreaming audio stream to YouTube automatically upon boot up
So I want to drag this thing to a performance and have a confederate with minimal technical know-how start it up. So basically I want it to start livestreaming when the RasPi is powered up. To do that I made this crontab entry (using crontab -e):

@reboot sleep 20; /home/pi/masterwireless.sh &gt; ff.log 2&gt;&amp;1

It takes a few minutes to get going, but it’s been extremely reliable. It’s started a stream successfully more than 10 times out of 10, at least when I was using my home WiFi connection. When I switched to my phone’s Hotspot, I had one error out of five attempts. The one bad stream just would not start according to Youtube, although per the stats from the log files showed the stream reached the usual good bandwidth. So I don’t know…

And once the stream starts, it is running uninterrupted for hours, anywhere from three to six hours.

Eventually I want to write an API program to automatically check the stream. But before then I may just introduce a refined script which checks the output and restarts ffmpeg when it has ended.

For the record, a typical ff.log file looks like this:

frame=   43 fps= 43 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A dup=0 drop=164 speed=   0x    ed=   0x
orig KB: dup=0
Tue  7 May 12:32:08 BST 2019
KB is: dup=0
*** Restarting ffmpeg at *** Tue 7 May 12:32:08 BST 2019
frame=  213 fps= 35 q=8.0 size=      47kB time=00:01:40.91 bitrate=   3.8kbits/s dup=0 drop=847 speed=16.7x
orig KB: 3.8kbits/s
Tue  7 May 12:38:53 BST 2019
KB is: 3
*** Restarting ffmpeg at *** Tue 7 May 12:38:53 BST 2019
illed=   86 fps= 14 q=8.0 size=     104kB time=00:00:06.21 bitrate= 136.7kbits/s dup=0 drop=336 speed=1.03x
orig KB: 136.7kbits/s
Tue  7 May 12:39:00 BST 2019
KB is: 136
Exiting at *** Tue 7 May 12:39:00 BST 2019

The other file, which has a name like ff.log05-07-19:12:32, looks more like this:

ffmpeg version 3.2.12-1~deb9u1+rpt1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 6.3.0 (Raspbian 6.3.0-18+rpi1+deb9u1) 20170516
  configuration: --prefix=/usr --extra-version='1~deb9u1+rpt1' --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf -
-incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gn
utls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libebur
128 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --ena
ble-libmp3lame --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-
libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --ena
ble-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --enab
le-libzvbi --enable-omx --enable-omx-rpi --enable-mmal --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --e
nable-libiec61883 --arch=armhf --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
Guessed Channel Layout for Input Stream #0.0 : stereo
Input #0, alsa, from 'plughw:1,0':
  Duration: N/A, start: 1557229134.030863, bitrate: 1536 kb/s
    Stream #0:0: Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s
Input #1, lavfi, from 'color=color=darkgray':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
[libx264 @ 0x12db850] VBV maxrate unspecified, assuming CBR
[libx264 @ 0x12db850] using SAR=8/9
[libx264 @ 0x12db850] using cpu capabilities: ARMv6 NEON
[libx264 @ 0x12db850] profile High, level 2.1
[libx264 @ 0x12db850] 264 - core 148 r2748 97eaef2 - H.264/MPEG-4 AVC codec - Copyleft 2003-2016 - http://www.videolan.org/
x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_ran
ge=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=8 lookahead_threads=1 sl
iced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 di
rect=1 weightb=1 open_gop=0 weightp=2 keyint=18 keyint_min=1 scenecut=0 intra_refresh=0 rc_lookahead=40 rc=cbr mbtree=1 bit
rate=50 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 vbv_maxrate=50 vbv_bufsize=512 nal_hrd=none filler=0 ip_ratio=1.40
 aq=1:1.00
Output #0, flv, to 'rtmp://a.rtmp.youtube.com/live2/KEY
  Metadata:
    encoder         : Lavf57.56.101
    Stream #0:0: Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuv420p, 480x320 [SAR 8:9 DAR 4:3], q=-1--1, 50 kb/s, 5 fps
, 1k tbn, 5 tbc
    Metadata:
      encoder         : Lavc57.64.101 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/50000 buffer size: 512000 vbv_delay: -1
    Stream #0:1: Audio: mp3 (libmp3lame) ([2][0][0][0] / 0x0002), 44100 Hz, stereo, s16p, 128 kb/s
    Metadata:
      encoder         : Lavc57.64.101 libmp3lame
Stream mapping:
  Stream #1:0 -&gt; #0:0 (rawvideo (native) -&gt; h264 (libx264))
  Stream #0:0 -&gt; #0:1 (pcm_s16le (native) -&gt; mp3 (libmp3lame))
Press [q] to stop, [?] for help
frame=   69 fps= 27 q=8.0 size=      45kB time=00:00:02.820 bitrate= 138.6kbits/s dup=0 drop=256 speed= 1.1x
frame=   79 fps= 17 q=2.0 size=      79kB time=00:00:04.80 bitrate= 134.6kbits/s dup=0 drop=308 speed=1.04x
frame=   91 fps= 13 q=8.0 size=     112kB time=00:00:06.80 bitrate= 134.8kbits/s dup=0 drop=348 speed=1.04x
frame=  101 fps= 11 q=8.0 size=     153kB time=00:00:09.22 bitrate= 135.0kbits/s dup=0 drop=388 speed=1.03x
frame=  112 fps= 10 q=3.0 size=     186kB time=00:00:11.40 bitrate= 133.8kbits/s dup=0 drop=440 speed=1.02x
av_interleaved_write_frame(): Broken pipe time=05:28:03.40 bitrate= 134.2kbits/s dup=0 drop=393880 speed=   1x
etc.
    Last message repeated 1 times
Error writing trailer of rtmp://a.rtmp.youtube.com/live2/KEY: Broken pipeframe=98474 fps=5.0 q=-1.0 Lsize=  322492kB time=05:28:14.00 bitrate= 134.1kbits/s dup=0 drop=393888 speed=0.998x
video:2213kB audio:306620kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 4.422897%
[libx264 @ 0x125c850] frame I:5471  Avg QP: 0.00  size:    80
[libx264 @ 0x125c850] frame P:27354 Avg QP: 0.00  size:    25
[libx264 @ 0x125c850] frame B:65649 Avg QP: 0.00  size:    17
[libx264 @ 0x125c850] consecutive B-frames: 11.1%  0.0%  0.0% 88.9%
[libx264 @ 0x125c850] mb I  I16..4: 100.0%  0.0%  0.0%
[libx264 @ 0x125c850] mb P  I16..4:  0.0%  0.0%  0.0%  P16..4:  0.0%  0.0%  0.0%  0.0%  0.0%    skip:100.0%
[libx264 @ 0x125c850] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  0.0%  0.0%  0.0%  direct: 0.0%  skip:100.0%
[libx264 @ 0x125c850] 8x8 transform intra:0.0%
[libx264 @ 0x125c850] coded y,uvDC,uvAC intra: 0.0% 0.0% 0.0% inter: 0.0% 0.0% 0.0%
[libx264 @ 0x125c850] i16 v,h,dc,p: 95%  0%  5%  0%
[libx264 @ 0x125c850] i8c dc,h,v,p: 100%  0%  0%  0%
[libx264 @ 0x125c850] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x125c850] kb/s:0.92
Conversion failed!

ffmpeg version 3.2.12-1~deb9u1+rpt1 Copyright (c) 2000-2018 the FFmpeg developers built with gcc 6.3.0 (Raspbian 6.3.0-18+rpi1+deb9u1) 20170516 configuration: --prefix=/usr --extra-version='1~deb9u1+rpt1' --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf - -incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gn utls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libebur 128 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --ena ble-libmp3lame --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable- libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --ena ble-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --enab le-libzvbi --enable-omx --enable-omx-rpi --enable-mmal --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --e nable-libiec61883 --arch=armhf --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared libavutil 55. 34.101 / 55. 34.101 libavcodec 57. 64.101 / 57. 64.101 libavformat 57. 56.101 / 57. 56.101 libavdevice 57. 1.100 / 57. 1.100 libavfilter 6. 65.100 / 6. 65.100 libavresample 3. 1. 0 / 3. 1. 0 libswscale 4. 2.100 / 4. 2.100 libswresample 2. 3.100 / 2. 3.100 libpostproc 54. 1.100 / 54. 1.100 Guessed Channel Layout for Input Stream #0.0 : stereo Input #0, alsa, from 'plughw:1,0': Duration: N/A, start: 1557229134.030863, bitrate: 1536 kb/s Stream #0:0: Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s Input #1, lavfi, from 'color=color=darkgray': Duration: N/A, start: 0.000000, bitrate: N/A Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc [libx264 @ 0x12db850] VBV maxrate unspecified, assuming CBR [libx264 @ 0x12db850] using SAR=8/9 [libx264 @ 0x12db850] using cpu capabilities: ARMv6 NEON [libx264 @ 0x12db850] profile High, level 2.1 [libx264 @ 0x12db850] 264 - core 148 r2748 97eaef2 - H.264/MPEG-4 AVC codec - Copyleft 2003-2016 - http://www.videolan.org/ x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_ran ge=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=8 lookahead_threads=1 sl iced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 di rect=1 weightb=1 open_gop=0 weightp=2 keyint=18 keyint_min=1 scenecut=0 intra_refresh=0 rc_lookahead=40 rc=cbr mbtree=1 bit rate=50 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 vbv_maxrate=50 vbv_bufsize=512 nal_hrd=none filler=0 ip_ratio=1.40 aq=1:1.00 Output #0, flv, to 'rtmp://a.rtmp.youtube.com/live2/KEY Metadata: encoder : Lavf57.56.101 Stream #0:0: Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuv420p, 480x320 [SAR 8:9 DAR 4:3], q=-1--1, 50 kb/s, 5 fps , 1k tbn, 5 tbc Metadata: encoder : Lavc57.64.101 libx264 Side data: cpb: bitrate max/min/avg: 0/0/50000 buffer size: 512000 vbv_delay: -1 Stream #0:1: Audio: mp3 (libmp3lame) ([2][0][0][0] / 0x0002), 44100 Hz, stereo, s16p, 128 kb/s Metadata: encoder : Lavc57.64.101 libmp3lame Stream mapping: Stream #1:0 -> #0:0 (rawvideo (native) -> h264 (libx264)) Stream #0:0 -> #0:1 (pcm_s16le (native) -> mp3 (libmp3lame)) Press [q] to stop, [?] for help frame= 69 fps= 27 q=8.0 size= 45kB time=00:00:02.820 bitrate= 138.6kbits/s dup=0 drop=256 speed= 1.1x frame= 79 fps= 17 q=2.0 size= 79kB time=00:00:04.80 bitrate= 134.6kbits/s dup=0 drop=308 speed=1.04x frame= 91 fps= 13 q=8.0 size= 112kB time=00:00:06.80 bitrate= 134.8kbits/s dup=0 drop=348 speed=1.04x frame= 101 fps= 11 q=8.0 size= 153kB time=00:00:09.22 bitrate= 135.0kbits/s dup=0 drop=388 speed=1.03x frame= 112 fps= 10 q=3.0 size= 186kB time=00:00:11.40 bitrate= 133.8kbits/s dup=0 drop=440 speed=1.02x av_interleaved_write_frame(): Broken pipe time=05:28:03.40 bitrate= 134.2kbits/s dup=0 drop=393880 speed= 1x etc. Last message repeated 1 times Error writing trailer of rtmp://a.rtmp.youtube.com/live2/KEY: Broken pipeframe=98474 fps=5.0 q=-1.0 Lsize= 322492kB time=05:28:14.00 bitrate= 134.1kbits/s dup=0 drop=393888 speed=0.998x video:2213kB audio:306620kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 4.422897% [libx264 @ 0x125c850] frame I:5471 Avg QP: 0.00 size: 80 [libx264 @ 0x125c850] frame P:27354 Avg QP: 0.00 size: 25 [libx264 @ 0x125c850] frame B:65649 Avg QP: 0.00 size: 17 [libx264 @ 0x125c850] consecutive B-frames: 11.1% 0.0% 0.0% 88.9% [libx264 @ 0x125c850] mb I I16..4: 100.0% 0.0% 0.0% [libx264 @ 0x125c850] mb P I16..4: 0.0% 0.0% 0.0% P16..4: 0.0% 0.0% 0.0% 0.0% 0.0% skip:100.0% [libx264 @ 0x125c850] mb B I16..4: 0.0% 0.0% 0.0% B16..8: 0.0% 0.0% 0.0% direct: 0.0% skip:100.0% [libx264 @ 0x125c850] 8x8 transform intra:0.0% [libx264 @ 0x125c850] coded y,uvDC,uvAC intra: 0.0% 0.0% 0.0% inter: 0.0% 0.0% 0.0% [libx264 @ 0x125c850] i16 v,h,dc,p: 95% 0% 5% 0% [libx264 @ 0x125c850] i8c dc,h,v,p: 100% 0% 0% 0% [libx264 @ 0x125c850] Weighted P-Frames: Y:0.0% UV:0.0% [libx264 @ 0x125c850] kb/s:0.92 Conversion failed!

CPU load average is around 1 or so – much less than before. So I think my ideas are on the right track. Why send 30 frames or whatever each and every second to Youtube just to display a gray screen? The CPU has to work to do that. As long as ffmpeg + Youtube has the intelligence to paste together audio snippets 1/5th second in length five times each second the audio should be taken care of, we’re not playing with the sampling rate or anything – is how I reasoned. Key frames are some sort of overhead as well since they’re extra things ffmpeg has to periodically do. Youtube wants one at least every four seconds. We get really close to that limit by multiplying fps * 3.6 s = 5 * 3.6 = 18 for our group-of-pictures (g) parameter. Previously we were sending a key frame more frequently – every two seconds.

Unreliability
Running this command is still hit-or-miss. As often as not it hangs, and then, if it does not hang, as often as not it often outputs washboard audio. You just <Ctrl-C> to get out of it if hangs, or type “q” if it is producing washboard audio.

Note carefully the bandwidth being used, which ffmpeg reports every second. If it is < 128 kbps, you’re hosed and have washboard audio. If it’s about 135 kbps or higher, you’re good. You don’t even need to waste time fiddling with Youtube’s live_dashboard to listen to it. You get this feedback immediately from ffmpeg. And I intend to use these same observed behaviors to script around ffmpeg’s flakiness and keep restarting it automatically until it is producing a good quality audio stream!

Improved startup
This script, which I call continuousaudio.sh, has some debugging at the beginning, then loops to ensure there is always an audio stream being live-streamed as long as the Pi has power. It has been extremely reliable. I settled on this one for my own purposes.

Copy Code


#!/bin/sh
# drJ 5/2019
LOG="ff.log"`date +%m-%d-%y:%H:%M`
# some info for debugging problems
echo "***********"
date; ip add; ping -c2 8.8.8.8; lsusb
nohup ./ffmpegwireless6.sh > $LOG 2>&1 &
while /bin/true; do
 sleep 7
# want s.th like
#Frame=   84 fps= 11 q=16.0 size=      43kB time=00:00:07.50 bitrate=  47.1kbits/s dup=0 drop=431 speed=0.991x
#Frame=   84 fps= 11 q=16.0 size=      43kB time=00:00:07.50 bitrate=  47.1kbits/s dup=0 drop=431 speed= 1x
 FFOUT=`tail -1 $LOG`
# next line only for DrJ debugging - lines are long
#echo "last line is $FFOUT"
# use gawk instead of awk to parse long lines
 KB=`echo $FFOUT|gawk '{print $(NF-5)" "$(NF-4)}'|sed 's/kbits.*//'|gawk '{print $NF}'`
 echo "orig KB: $KB"
 KB=$(echo $KB|sed s/\\..*//)
 date
 echo "KB is: $KB"
 if [ $KB -gt 129 2>/dev/null ]; then
# stream looks good - do nothing
   echo -n ""
 else
# didn't work out: restart and try again
  echo "*** Restarting ffmpeg at *** "`date`
  pkill -9 -f 'ffmpeg '
  nohup ./ffmpegwireless6.sh > $LOG 2>&1 &
 fi
done

gawk

Eventually I found I needed to use gawk instead of awk in continuousaudio.sh because the number of fields exceeded the max of roughly 32700 in the line I was parsing. To install gawk:

$ sudo apt-get install gawk

Note it still calls ffmpegwireless6.sh, which I believe I have provided above.

ff.log now looks like this:

**********
Fri 31 May 01:10:59 BST 2019
1: lo: &lt;loopback,up,lower_up&gt; mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: &lt;no-carrier,broadcast,multicast,up&gt; mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether b8:27:eb:11:fc:06 brd ff:ff:ff:ff:ff:ff
3: wlan0: &lt;broadcast,multicast,up,lower_up&gt; mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:44:a9:53 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.170/24 brd 192.168.1.255 scope global wlan0
       valid_lft forever preferred_lft forever
    inet6 fe80::1119:b46a:cb69:63c9/64 scope link
       valid_lft forever preferred_lft forever
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=14.6 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=17.4 ms
 
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 14.671/16.065/17.460/1.400 ms
Bus 001 Device 004: ID 046d:0825 Logitech, Inc. Webcam C270
Bus 001 Device 005: ID 0424:7800 Standard Microsystems Corp.
Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
last line is frame=   19 fps=0.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A dup=0 drop=69 speed=   0x    ^Mframe=   39 fps= 39 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A dup=0 drop=150 speed=   0x    ^M
orig KB: dup=0
Fri 31 May 01:11:07 BST 2019
KB is: dup=0
*** Restarting ffmpeg at *** Fri 31 May 01:11:07 BST 2019
last line is frame=  193 fps= 35 q=8.0 size=     100kB time=00:00:27.60 bitrate=  29.6kbits/s dup=0 drop=764 speed=4.99x    ^Mframe=  195 fps= 32 q=8.0 size=     108kB time=00:00:28.03 bitrate=  31.4kbits/s dup=0 drop=775 speed=4.65x    ^M
orig KB: 31.4
Fri 31 May 01:11:36 BST 2019
KB is: 31
*** Restarting ffmpeg at *** Fri 31 May 01:11:36 BST 2019
...
&lt;/broadcast,multicast,up,lower_up&gt;&lt;/no-carrier,broadcast,multicast,up&gt;&lt;/loopback,up,lower_up&gt;

********** Fri 31 May 01:10:59 BST 2019 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000 link/ether b8:27:eb:11:fc:06 brd ff:ff:ff:ff:ff:ff 3: wlan0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether b8:27:eb:44:a9:53 brd ff:ff:ff:ff:ff:ff inet 192.168.1.170/24 brd 192.168.1.255 scope global wlan0 valid_lft forever preferred_lft forever inet6 fe80::1119:b46a:cb69:63c9/64 scope link valid_lft forever preferred_lft forever PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=14.6 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=17.4 ms --- 8.8.8.8 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 14.671/16.065/17.460/1.400 ms Bus 001 Device 004: ID 046d:0825 Logitech, Inc. Webcam C270 Bus 001 Device 005: ID 0424:7800 Standard Microsystems Corp. Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub last line is frame= 19 fps=0.0 q=0.0 size= 0kB time=00:00:00.00 bitrate=N/A dup=0 drop=69 speed= 0x ^Mframe= 39 fps= 39 q=0.0 size= 0kB time=00:00:00.00 bitrate=N/A dup=0 drop=150 speed= 0x ^M orig KB: dup=0 Fri 31 May 01:11:07 BST 2019 KB is: dup=0 *** Restarting ffmpeg at *** Fri 31 May 01:11:07 BST 2019 last line is frame= 193 fps= 35 q=8.0 size= 100kB time=00:00:27.60 bitrate= 29.6kbits/s dup=0 drop=764 speed=4.99x ^Mframe= 195 fps= 32 q=8.0 size= 108kB time=00:00:28.03 bitrate= 31.4kbits/s dup=0 drop=775 speed=4.65x ^M orig KB: 31.4 Fri 31 May 01:11:36 BST 2019 KB is: 31 *** Restarting ffmpeg at *** Fri 31 May 01:11:36 BST 2019 ... </broadcast,multicast,up,lower_up></no-carrier,broadcast,multicast,up></loopback,up,lower_up>

My crontab now looks like this:

Copy Code


@reboot sleep 20;/home/pi/continuousaudio.sh > ff.log 2>&1

Portability
I wanted to record a practice session in my house where no Ethernet port is available (hence I had to get WiFi working, which I believe I have). And I wanted convenience – to not worry about being tethered to the wall by an adapter. So I decided to look for an economical power solution for Raspberry Pi. And I found the ones purpose-built are just too expensive to justify. Pijuice, I’m talking about you. So, really, I realized any old portable USB power stick would work. But I wanted something which could last hours. This Omars 10000 mAh portable USB charger seemed like it would do the trick. $16. And it did. It works great! Two hours later, the LEDs show three bars instead of four, so I think this will supply power for about 8 – 9 hours if I pushed it. And it has the form factor of a smartphone. Ideally I’d want a little on/off switch to avoid plugging/unplugging the power cable, but I didn’t find that as of yet. Maybe there’s a cheap USB cable with that…?

So now I’m not tethered by Ethernet cables nor by a power plug. See where this is progressing? If I use my smartphone’s hotspot I should be able to livestream anywhere I can get a signal, so, for instance, at band performances. I haven’t tried that yet, but I’m hopeful…

YouTube quirks
As previously mentioned (I think)( you need to be enabled for livestreaming. It takes about 24 hours for the approval. I suppose they check to make sure you aren’t a perceived threat.

Recording NPR will give you a copyright violation flag! This has happened to me more than once. I think because they play snippets of new music which are flagged.

Lag. I’ve seen lag time as short as four seconds and maybe as long as 20 seconds or so. It is never instantaneous.

My longest video was 20 hours but the processing took days. In fact I’m not sure it ever completed. So I guess the service falls apart after video lengths of I don’t know, maybe 12 hours or so. So if the desire is to have a continuous security webcam I guess you’ll have to break it into chunks. That’s what I’m thinking about next.

A livestream gets converted to a video by YouTube. That takes awhile – maybe as long as the video length itself is? It slaps a date and time onto the video which you see in your video manager. Unfortunately, using this ffmpeg streaming method it chooses the Pacific standard time timezone. I actually don’t see a simple way to change that either. It may require use of the API, which is beyond what I’m willing to tackle right now. So for me, being in the Eastern time zone all the timestamps are off by three hours, which is kind of annoying.

I wondered, does my livestream ID remain constant, or will it change from broadcast to broadcast? This is important for future use of the API. Well, it changes each time I start a new livestream, even though I use a single (my own) account. Each livestream gets a unique ID which then becomes the ID for the DVR of the video which you can view on-demand. And this ID is the part that changes in the URL of an “unpublished” Youtube video. Say your unpublished livestream is
https://www.youtube.com/watch?v=r1wtZwQ-Tk8.
The part of the URL following the v=, namely, in this example, r1wtZwQ-Tk8, is the ID of that video. I would say YouTube tries to be somewhat robust and will not declare your stream has ended until maybe 30 seconds after you have stopped your program. Or maybe it’s a minute or two, I’m not really sure. But I’ve seen that if you restart the streaming quickly enough you’ll be put onto that same livestream. If on the other hand you wait long enough until you see in live_dashboard that stream ended message then It will assign yuo a new video ID if you start your stream again – and don’t forget to reload the live_dashboard page so it can pick up the new ID.

Can you pause a livestream, and later resume, keeping the same URL? In a word, No. Unfortunately. Youtube livestreaming is pretty limited in this way. And how useful would that be? I would use my smartphone to control ffmpeg on my Raspberry Pi to pause our band practice during our lengthy chat breaks, keeping the stream focussed on the music. But no… Not possible.

Logitech webcam quirks
When you pull both video and audio from your Logitech webcam the usage LED illuminates as you’d expect. However, when you’re pulling just the audio, as I show above, that LED does not illuminate, yet it is being used to record all the sounds in its vicinity. I guess I have accidentally and unintentionally stumbled upon a stealth mode, which is a little disconcerting.

Yeti USB microphone quirks
A Yeti mic is extremely sensitive and seems more suited for conversation than music recording in my opinion. Even with the gain all the way down (a must) a loud sound is often distorted. I felt the omni recording mode was the worst in this regard. Stereo recording tolerated sounds better. But, if you want to pikc up every little sound, Yeti is great. More importantly to me, it just worked with the USB settings I used for Logitech. I didn’t have to change a single thing in the way I used ffmpeg.

Testing if the livestream is still running
My idea to do this is to use the YouTube API and periodically test if the livestream is still working. I have read that it can go down for various reason, and there is no goo way from within ffmpeg itself to tell that your stream is no longer live! It will make for a good project to test the livestream using the Google Developer’s API. that will be a separate post if I ever get it working. If it’s found to be down, the Pi could restart ffmpeg, in my thinking.

To do list
I never really perfected the video. Audio I got pretty well.
I will borrow my friend’s Yeti USB mic to see how my audio stream works with a high quality microphone. DONE.
I would like to have a simple external control to turn stream off/ on, whether it is physical or virtual. DONE – see references.
Scripting to monitor stream and restart it once it fails – to have a recording 24×7 like an audio-only security camera. DONE – continuousaudio.sh as documented above.
Pause feature. PARTIALLY DONE.

Conclusion
A Raspberry Pi 3 running Raspbian Stretch Lite is used, along with a Logitech USB webcam, to livestream to YouTube. I showed how to stream video-only with a silent audio track. Then I turned it around and spent most of my time putting a virtual piece of tape over the lens and doing an audio-only livestream. This, after a crap-load of testing and tweaking, eventually began to work in a reliable fashion. Then I showed how to launch the audio-only livestream upon power-up of the Ras Pi.

Since it is a Raspberry Pi, this whole thing lends itself to portability and interesting use cases. With a $17 portable USB battery source and your own Hotspot, you can stream (audio at least) from anywhere you have 4G cell signal – good for recording a banquet, your band performance, or any other long, live event.

I spoke about some of the many quirks of YouTube which are relevant to this project.

References and related
Where I debug YouTube’s messages: https://www.youtube.com/live_dashboard

Fishcam implemented with Raspberry Pi + webcam + help of my AWS server.

One of my test videos: https://youtu.be/oxJaZv0frGM

Check your upload bandwith: speedtest.net

YouTube’s links have me confused. If you’re trying to produce a Live Stream you’ll want the live dashboard page to watch it and check its quality as Youtube judges it. Here’s that link: https://www.youtube.com/live_dashboard

Use ffmpeg to deal with live input audio and make a matrix LED dance in real time: Raspberry Pi + LED Matrix Display project

Microcenter in Paterson, NJ – best to visit in person, or so I have been told.

My livestream is https://www.youtube.com/watch?v=r1wtZwQ-Tk8

Put virtual tape over your lens by using this tip discussed in Stackoverflow!

Portable, proven (by me) economical USB power supply for your Raspberry Pi – $16.

Economical on/off switch for your Raspberry Pi. This is a great way to stop having to pull out/push in power connectors from your micro USB power source. $10 gets you a four-pack! https://smile.amazon.com/iUniker-Raspberry-Switch-Supply-MicroUSB/dp/B07CTHKXDW/ref=sr_1_2_sspa?keywords=raspberry+pi+on+off+switch&qid=1559477662&s=gateway&sr=8-2-spons&psc=1

Tags ffmpeg, Logitech webcam, micro USB on/off switch, Microcenter, Speedcheck, Yeti, Youtube

Admin Linux Raspberry Pi

Fishcam using Raspberry Pi and some network tricks

Post author By john
Post date February 3, 2019
No Comments on Fishcam using Raspberry Pi and some network tricks

Intro
There are more articles about running a webcam using Raspberry Pi than Carter has pills. Why bother to create another? This one is unique insofar as I created a fishcam at a school with a restricted network. None of the reference articles I found discussed a way to get your stream onto the Internet except the simplistic approach only available to homeowners of setting up a rule on a home router. Pimylifeup’s article is typical of that genre.

Cooperating third party
To push this webcam out to the Internet when I had no way to allow inbound traffic to the Pi, I realized that I needed a cooperating third party. I looked briefly for a commercial service specializing in this. I did not find one. I suppose there is, but I don’t know. It was actually quicker to stop the search and use my own AWS server as the cooperating third party.

With a cooperating third party what you can do is set up a forwarder from the Pi to cooperating server on the Internet. So that’s what I did. More on that below.

Network restrictions
The Pi was given WiFi access to a school’s bring your own device (BYOD) WiFi. By trial and error (I did not initiate extensive port scans, etc so as to avoid acting like a hacker). I’m familiar with running a almost completely open Guest wireless. This BYOD was not that for some reason unknown to me. One of the first things I tried, to ssh to my server, was not going through. So I knew there were restrictions. Also PING 8.8.8.8 did not work. So ICMP was blocked as well. But web browsing worked, and so did DNS queries. So TCP ports 80 and 443 were allowed, as well as UDP port 53 and possibly TCP port 53. I also observed there was no proxy server involved in the communication. So I simply tested a few other ports that I know are used from time-to-time: 2443 and 8443. If you a hit a server that is not protected by a firewall and not listening on a port that you are testing you will get a Connection reset if your packets are not blocked by a local firewall. I tested with the nc utility. nc -v <my_server> <port> I found a couple open ports this way. Next question: does the network care what protocol is running on that port? They might be looking for https and I was planning to run ssh. For a simple port blocker it might not distinguish what’s going on. That was indeed the case as I was able to run ssh on this non-standard port.

The single most comlicated thing was formulating the appropriate ssh command. I created a dedicated account on my server for this purpose. I embedded the password into the startup script I created using a utility called sshpass. This is not super secure but I wanted something running quickly.

Here’s that complicated command

sshpass -p <PASSWORD> ssh ‐f ‐N ‐R 8443:localhost:8081 ‐p 2443 <USERNAME>@<SERVER_IP>

That’s a mouthful! Let’s break it down. sshpass just permits you to run the command and not get a login prompt. It needs to be installed with a sudo apt-get sshpass.

The ssh command sets up a reverse tunnel. I have discussed it in my Access your Raspberry Pi from anywhere blog post, however, some things are different and more complicated here. Here we are setting up port 8443 on my server as the tunnel port which will be accessible to the Internet. It is terminated on the Raspberry Pi’s local port 8081 (the port that the motion package uses for the webcam). We had to use ssh to connect to port 2443 on my server because the school network blocked the standard port 22. Then <PASSWORD>, <USERNAME> and <SERVER_IP> are to be replaced with values specific for my server. I don’t want to publish them.

How I got my server to run ssh on port 2443 as well as port 22
This turned out to be one of the easiest things. It’s good to run your own server… In the file /ets/ssh/sshd_config the listening port was commented out, letting the defaul 22 be in effect. So I uncommented that and added port 2443 like this:

...
# Listen on multiple ports - DrJ 2/1/19
Port 22
Port 2443
...

Then a sudo service sshd restart and the server listens on both ports for ssh connections!

About the webcam itself
I just followed the Pimylife article as I mentioned. It talks about using the motion package which I’ve never used before. Now in my other posts you’ll see I do stuff with video on Raspberry Pi. In those we had to fight to get the lag time down and keep bandwidth low. I have to say by comparison this motion package is awful. Lag is a couple seconds. There is no sense whatsoever of true video. Just image, wait, next image, wait. No matter the fps setting. I did not have time to switch to a video package that works better. Anyway motion may provide some other advantages we could eventually use. So I just set it to 2 fps (frames per second) since it doesn’t really matter.

The fishcam is at fishcam. It’s not working right now – just showing black. I’m not sure why.

Auto starting
I’ve documented elsewhere the poor man’s way to start something upon initially booting the Pi: stuff the appropriate command into the crontab file.

So you edit your crontab file with a crontab -e. Then you enter

@reboot sleep 20; sshpass -p <PASSWORD> ssh ‐f ‐N ‐R 8443:localhost:8081 ‐p 2443 <USERNAME>@<SERVER_IP>

That just sleeps for 20 seconds as your Pi boots up, then establishes the reverse tunnel with that complicated command I explained earlier.

Equipment
Usually thes tutorials start with an equipment list. For me that is the least interesting part. I used a Raspberry Pi 3 running Raspbian. For a camera I used one of my spare USB ELP cameras from my extensive work with USB cameras. While developing the solution I needed a keyboard, mouse and HDMI monitor. Once running, the only thing connected to the Pi is the USB camera and the micro USB power supply.

To be continued…

References and related
A very good guide for your typcial webcam-using-a-Raspberry-Pi situation, i.e., not what I am addressing in my article.

Access your Raspberry Pi from anywhere blog post

Run multiple USB cameras on your Raspberry Pi while keeping lag minimal.

For supplies we love visits to The Micro Center in Paterson, NJ. This past weekend we got Raspberry Pi 3’s for only $29. And the sales tax is only 3% and change.

Tags fishcam

Linux Raspberry Pi

Evaluation of WPI’s multiple camera coprocessor using Raspberry Pi

Post author By john
Post date January 12, 2019
No Comments on Evaluation of WPI’s multiple camera coprocessor using Raspberry Pi

Intro
There’s some good and some not-so-good about the new WPI-provided way to handle multiple video streams using a Raspebrry Pi.

ELP Cameras problems
I have bought many of these ELP cameras last year and this. I may be a slow learner, but eventually it dawned on me that the problems I noticed seem to occur because of this model of USB camera. Finally this year we got a chance to explore this further. I got my hands on a Logitech webcam, the kind you use perched on top of a display monitor. We had this set up as a second camera while an ELP camera was the first. Then we rebooted the Pi a whole bunch of times to gather some stats. About 25% of the time there were problems with the ELP over about 10 tries. There were no problems with the Logitech. Here are various problems we’e seen:
– horizontal lines superimposed over image, and image dull
– ghosting, a corner of the field of view is shown in the center of the image
– sometimes the stream never starts and we’re not yet sure if that’s a camera problem or a software problem though I begin to suspect it’s an ELP problem
– one of my pinhole ELP cameras died

So: Logitech webcam is decidedly better.

Power problem
We pay extra attention to the power draw of the Pi. With two cameras attached and a 2 amp or 1.8 amp power supply the red LED power flashes. That is not good. It’s a sign of undervoltage. The command

vcgencmd get_throttled

on your Pi will tell you if there was an undervoltage condition. I see

throttled=0x50005

when using a 2 amp power supply. Note that as far as we can see the camera display itself works just as well. We also have a 3 amp power supply. That produces a solid red led light and vcgencmd get_throttled produces a response of

throttled=0x0,

which probably indicates there were no undervoltage conditions.

The problem we need to avoid for the Pi to attempt to draw more than 2 amps from the regulator. Doing so may shut it down. So we will try to use the Pi along with a powered USB hub.

Bandwidth constraints
We want to be well below 3 mbps for two cameras. How to get there while still providing a useful service. Initially we felt we could run the cameras at 320×240 resolution, 10 fps. But after much playing we found conditions under which we exceed 3 mbps though normally we were below that. I believe that the compressibility of the image is what matters. So a “rich” visudal field with lots of contrasting objects is the least compressible. That vaguely fits our findings as well. So we felt it important to prepare for the worst case. So we actually looked at supported resolutions and settled on 176×144 pixels! It sure isn’t much, agreed, but it’s still helpful. We blow up the images during the display. We use YUYV mode. MJPEG uses considerably more bandwidth.

Refresh trick
With this WPI software, the video streams never display the first time. You have to refresh the page for some reason. We wished to have a one click operation for viewing, however, to minimize the risk of operator error. So we used some old-fashioned META HTML tags to force a page refresh.

Our initial approach was to simply have the web page refresh itself every five seconds. This worked, but caused instability in the video stream itself and given a few minutes, would always crash the video stream. So we came up with an alternative that does a single page refresh. Unfortunately we’re not that conversant in Javascript (I’m sure there’s a way to do this with Javascript) so instead we wrote two HTML pages: a source page with the refresh, and a target page that does not refresh.

Initial page HTML source

<html><head>
<meta http-equiv="Refresh" content="1;url=file:///C:/users/aperture/Desktop/2019-no-refresh.htm">
<title>stream</title></head>
<body>
<img src="http://10.31.42.18:1182/stream.mjpg" width=560 height=400>
<img src="http://10.31.42.18:1181/stream.mjpg" width=560 height=400>
</body>
</html>

And we size the browser window to just fit the two video streams side-by-side.

Target HTML source for 2019-no-refresh.htm

<html><head>
<title>stream</title></head>
<body>
<img src="http://10.31.42.18:1182/stream.mjpg" width=560 height=400>
<img src="http://10.31.42.18:1181/stream.mjpg" width=560 height=400>
</body>
</html>

Timing and sequence of events
After the Ras Pi is powered up, we launch the initial page from the task bar where it was pinned, 20 seconds later.

It takes a bit of time, then it displays the side-by-side video streams as broken images.

The red LEDs on the Logitech webcams begin to glow.

(We know when we see both red LEDs glowing we are good to go by the way).

the refresh occurs automatically to the 2019-no-refresh.htm web page.

Two side-by-side video streams are displayed, each with 560×400 display dimensions.

References and related
My 2018 version of using the Raspebrry Pi to handle two USB cameras: USB webcam on Raspberry Pi

Field Management System spec for 2019

https://s3.amazonaws.com/screensteps_live/exported/Wpilib/2078/103766/Using_a_Raspberry_PI_as_a_video_coprocessor.pdf?1546622998
WPI PDF manual, Using a Ras Pi as Video coprocessor

Download compressed image from Github: https://github.com/wpilibsuite/FRCVision-pi-gen/releases/. Scroll down to Assets and look for FRCVision_image_2019xxxx.zip. (2019.3.1 is the latest at time of this writing.

Logitech webcam: https://smile.amazon.com/gp/product/B01IC2UDMC/ref=ppx_yo_dt_b_asin_title_o00__o00_s00?ie=UTF8&psc=1

FIRST FRC Networking Basics

Linux Raspberry Pi

Solution to this week’s NPR puzzle using simple Linux commands, again

Post author By john
Post date September 23, 2018
No Comments on Solution to this week’s NPR puzzle using simple Linux commands, again

Intro
As I understood it, this week’s NPR puzzle is as follows. Think of a figure from the Bible with five letters. Move each letter three back, e.g., an “e” becomes a “b.” Find the Biblical figure which becomes an ailment after doing this transformation.

Initial thoughts
I figured this would be eminently amenable to some simple linux commands like I’ve done with previous puzzles (most are not, by the way). I was having a hard time doing these transformations in my head while I was driving, and the first names I tried came up empty, such as Jesus or Moses.

So I figured I could write a program to do the character transformations on each and every word and I could probably find a downloadable text version of the Bible. I didn’t find a pure text version, but I did download an HTML version, which is close enough for our purposes.

Then I was going to just keep the five-letter words and do this transformation on all of them and match against dictionary words. Then I would have taken just those matches and scanned by hand to look for words that are ailments, hoping there wouldn’t be too many matched words to contend with.

Finally settled on a different approach
That looked like a bit of work so I thought about it and decided there had to be a resource for just the figures in the Bible, and voila, there is, in Wikipedia, see the references.

rot13
Rot13 is a famous cipher (encryption is too strong a word to describe this simple approach), where A becomes N, B becomes O, etc. I had a feeling the tr command in linux might be able to do this but didn’t know how. So I searched for linux, tr and rot13 and found an example online. It was easy to adapt.

We need what you could call a rot -3. Here is the command.

$ tr 'A‐Za‐z' 'X‐ZA‐Wx‐za‐w'

So I put the text of the Wikipedia page of Biblical figures into a text file on my linux server, into a file called list-of-biblical-figures. It looks like this:

Adam to David according to the Bible
Creation to Flood
 
    Adam Seth Enos Kenan Mahalalel Jared Enoch Methuselah Lamech Noah Shem
 
Cain line
 
    Adam Cain Enoch Irad Mehujael Methusael Lamech Tubal-cain
 
Patriarchs after Flood
 
    Arpachshad Cainan Shelah Eber Peleg Reu Serug Nahor Terah Abraham Isaac Jacob
 
Tribe of Judah to Kingdom
 
    Judah Perez Hezron Ram Amminadab Nahshon Salmon Boaz Obed Jesse David
...

I was going to tackle just pulling the figures with five-character names, but the whole list isn’t that long so I skipped even that step and just put the list through as is:

$ cat list-of-biblical-figures|tr 'A‐Za‐z' 'X‐ZA‐Wx‐za‐w'

comes back as

Xaxj ql Axsfa xzzloafkd ql qeb Yfyib
Zobxqflk ql Cilla
 
    Xaxj Pbqe Bklp Hbkxk Jxexixibi Gxoba Bklze Jbqerpbixe Ixjbze Klxe Pebj
 
Zxfk ifkb
 
    Xaxj Zxfk Bklze Foxa Jbergxbi Jbqerpxbi Ixjbze Qryxi-zxfk
 
Mxqofxozep xcqbo Cilla
 
    Xomxzepexa Zxfkxk Pebixe Bybo Mbibd Obr Pbord Kxelo Qboxe Xyoxexj Fpxxz Gxzly
 
Qofyb lc Graxe ql Hfkdalj
 
    Graxe Mbobw Ebwolk Oxj Xjjfkxaxy Kxepelk Pxijlk Ylxw Lyba Gbppb Axsfa
...
    Ebola
...

So it’s all gibberish as you might hope. Then towards the end you come across this one thing and it just pops out at you. As is my custom I won’t give it away before the deadline. [update] OK. Submission deadline has passed. Ebola just really popped out. Going back to the original text, you see it lines up with Herod. So there you have it.

I double-checked and confirmed this also works on a Raspberry Pi. I’ve come to realize that most people don’t have their own server, but hundreds of thousands or perhaps millions have a Raspberry Pi, which is a linux server, which makes techiques like this accessible. And fun.

Conclusion
I show a technique for using a linux server such as a Raspberry Pi to solve this week’s NPR puzzle. A very simple approach worked. In fact I was able to solve the puzzle and write this post in about an hour!

References and related
HTML version of Bible: https://ebible.org/Scriptures/eng-web_html.zip
Biblical figures: https://en.wikipedia.org/wiki/List_of_major_biblical_figures
An earlier NPR puzzle solved with linux command line techniques

Tags NPR, rot13, tr

Tip: Undervoltage thunderbolt suppression

Appendix 1: rclone configuration

400 Bad Request

client. badssl.com

2021 pre-cap

gawk

client.
badssl.com