Network Technologies Web Site Technologies

The IT Detective Agency: the case of Failed to convert character

A user of a web form noticed any password that includes an accented character is rejected. He came to use as the operator of the web application firewall for a fix.

More details
The web server was behind an F5 device running ASM – application security manager. The reported error that we saw was Failed to convert character. What does it all mean?

One suggestion is that the policy may have the wrong language, but the application language of this policy is unicode (utf-8), just like all our others we set up. And they don’t have any issues. I see where I can remove the block on this particular input violation, but that seems kind of an extreme measure, like throwing out the baby with the bathwater.

I wondered about a more granular way to deal with this?

Check characters on this parameter value is already disabled I notice, so we can’t further loosen there.

Ask the expert
So I ask someone who speaks a foreign language and has to deal with this stuff a lot more than I do. He responds:

Looking at the website I think that form just defaults to ISO-8859-1 instead of UTF-8 and that causes your problem.
Umlauts or accented letters are double byte encoded in UTF-8 and single byte in ISO-8859-1

To confirm the problem with the form, he enters an “ä” as the username, which the event log shows encoded to %E4 which is not a valid UTF-8 sequence.

Our takeaway
To repeat a key learning from this little problem:
Umlauts or accented letters are double byte encoded in UTF-8 and single byte in ISO-8859-1

So the web form itself was the problem in this case; and I went back to the user/developer with this informatoin.

So he fixed it?
Well, turns out his submission form was a private page he quickly threw together to test another problem, the real problem, when he noticed this particular issue.

So, yes, his form needed to mention utf-8 if he were going to properly encode accented characters, but that did not resolve the real issue, which remains unresolved.

It happens that way sometimes.

But, yes, the problem reported to us was resolved by the developer based on our feedback, so at least we have that success.

If like me, your eyes glaze over when someone mentions ISO-8859-1 versus UTF-8, the differences are pretty stark, easy-to-understand, and, just sometimes, really, important! I think ISO-8859-1 will represent some of the popular accented characters in positions 128 – 255, but not utf-8. utf-8 will use additional bytes to represent characters outside of the Latin alphabet plus the usual special characters.

We’ll call this one Case Closed!

References and related
I like to do a man ascii on any linux system to see the representation of the various Latin characters. I had to install the man-pages package on my RHEL system before that man page was available on my system.

Admin IT Operational Excellence Network Technologies

No Internet, secure WiFi status message in Windows 10

Finding out how Windows decides if there is an Internet connection or not can be a challenge often posed by trying to do an Internet search comprised or words that are common and therefore used in many other contexts. I have to give credit to someone else who found most of these pertinent links that help explain how Windows decides whether or not your PC has an Internet connection.

What they don’t tell you
I think there are a lot more tests microsoft does than what they’ve documented. In my opinion, based on observation, in addition to the sites they recommend to whitelist, also whitelist

Some PCs get stuck in a loop requesting indefinitely, which isn’t good for anyone.

Here’s one they don’t mention, of the same ilk:

I’m thinking to just leave that one alone, unless you really are fully running on ipv6.

Now if you have a PAC file, what you’re going to see are accesses for

I don’t think that one’s documented either. I’m not yet sure how best to have the PAC file web server respond, where best means the reply which would make the PC most likely to decide Yes I really do have an Internet connection.

References and related
This Pulse Secure article is pretty good. You start with an Internet connection, then launch Pulse Secure vpn, then find you are told there is no longer an Internet connection. This explains why it might be, but in my opinion it is incomplete as it does not even consider the case where an authenticating proxy is the sole gateway to the Internet:

These are two more articles about VPN tunneling

network Location Awareness (NLA) and Network Connection Status Indicator (NCSI) are explained in these articles

Linux Network Technologies Raspberry Pi

Raspberry Pi photo frame using your pictures on your Google Drive

All my spouse’s digital photo frames are either broken or nearly broken – probably she got them from garage sales. Regardless, they spend 99% of the the time black. Now, since I had bought that Raspberry Pi PiDisplay awhile back, and it is underutilized, and I know a thing or two about linux, I felt I could create a custom photo frame with things I already have lying around – a Raspberry Pi 3, a PiDisplay, and my personal Google Drive. We make a point to copy all our cameras’ pictures onto the Google Drive, which we do the old-fashioned, by-hand way. After 17 years of digital photos we have about 40,000 of them, over 200 GB.

So I also felt obliged to create features you will never have in a commercial product, to make the effort worthwhile. I thought, what about randomly picking a few for display from amongst all the pictures, displaying that subset for a few days, and then moving on to a new randomly selected sample of images, etc? That should produce a nice review of all of them over time, eventually. You need an approach like that because you will never get to the end if you just try to display 40000 images in order!

The scripts
Here is the master file which I call

# DrJ 8/2019
# call this from cron once a day to refesh random slideshow once a day
STARTFOLDER="MaryDocs/Pictures and videos"
echo "Starting master process at "`date`
#listing of all Google drive files starting from the picture root
if [ $DEBUG -eq 1 ]; then echo Listing all files from Google drive; fi
rclone ls remote:"$STARTFOLDER" > files
# filter down to only jpegs, lose the docs folders
if [ $DEBUG -eq 1 ]; then echo Picking out the JPEGs; fi
egrep '\.[jJ][pP][eE]?[gG]$' files |awk '{$1=""; print substr($0,2)}'|grep -i -v /docs/ > jpegs.list
# throw NUMFOLDERS or so random numbers for picture selection, select triplets of photos by putting
# names into a file
if [ $DEBUG -eq 1 ]; then echo Generate random filename triplets; fi
./ -f $NUMFOLDERS -j jpegs.list -r $RANFILE
# copy over these 60 jpegs
if [ $DEBUG -eq 1 ]; then echo Copy over these random files; fi
cat $RANFILE|while read line; do
  rclone copy remote:"${STARTFOLDER}/$line" $DISPLAYFOLDERTMP
# kill any qiv slideshow
if [ $DEBUG -eq 1 ]; then echo Killing old qiv slideshow; fi
pkill -9 -f qiv
# remove old pics
if [ $DEBUG -eq 1 ]; then echo Removing old pictures; fi
#run looping qiv slideshow on these pictures
if [ $DEBUG -eq 1 ]; then echo Start qiv slideshow in background; fi
cd $DISPLAYFOLDER ; nohup ~/ &
if [ $DEBUG -eq 1 ]; then echo "And now it is "`date`; fi

Needless to say, but I’d better say it, the STARTFOLDER in this script is particular to my own Google drive. Customize it as appropriate for your situation.

Then qiv (quick image viewer) is called with a bunch of arguments and some trickery to ensure proper display of files with spaces in the filenames (an anathema for Linux but my spouse doesn’t know that so I gotta deal with it). I call this script

# -f : full-screen; -R : disable deletion; -s : slideshow; -d : delay <secs>; -i : status-bar;
# -m : zoom; [-r : ranomdize]
# this doesn't handle filenames with spaces:
##cd /media; qiv -f -R -s -d 5 -i -m `find /media -regex ".+\.jpe?g$"`
# this one does:
export DISPLAY=:0
if [ "$1" = "l" ]; then
# print out proposed filenames
  find . -regex ".+\.[jJ][pP][eE]?[gG]$"
# args: f fullscreen d delay s slideshow l autorotate R readonly I statusbar
# i nostatusbar m maxspect
  find . -regex ".+\.[jJ][pP][eE]?[gG]$" -print0|xargs -0 qiv -fRsmil -d 5

Here is the perl script which generates the random numbers and associates them to the file listing we’ve just made with rclone,

use Getopt::Std;
my %opt=();
$nofolders = $opt{f} ? $opt{f} : 20;
$DEBUG = $opt{d} ? 1 : 0;
$jpegs = $opt{j} ? $opt{j} : "jpegs.list";
$ranpicfile = $opt{r} ? $opt{r} : "jpegs-random.list";
print "d,f,j,r: $opt{d}, $opt{f}, $opt{j}, $opt{r}\n" if $DEBUG;
open(JPEGS,$jpegs) || die "Cannot open jpegs listing file $jpegs!!\n";
@jpegs = <JPEGS>;
# remove newline character
$nopics = chomp @jpegs;
open(RAN,"> $ranpicfile") || die "Cannot open random picture file $ranpicfile!!\n";
for($i=0;$i<$nofolders;$i++) {
  $t = int(rand($nopics-2));
  print "random number is: $t\n" if $DEBUG;
  ($dateTime) = $jpegs[$t] =~ /(\d{8}_\d{6})/;
  if ($dateTime) {
    print "dateTime\n" if $DEBUG;
  $priorPic = $jpegs[$t-2];
  $Pic = $jpegs[$t];
  $postPic = $jpegs[$t+2];
  print RAN qq($priorPic

Note that to display 60 pictures only 20 random numbers are used, and then the picture 2 prior and the picture two after the one selected by the random number are also displayed. This helps to provide, hopefully, some context to what is being shown without showing all those duplicate pictures that everyone takes nowadays.

There is an attempt to favor recently uploaded pictures but I really haven’t perfected that part of, it’s more of a thought at this point.

My crontab entries take care of starting a slideshow upon first boot as well as a daily pick of 60 new random pictures!

@reboot sleep 40; cd ~/Pictures; ~/ >> ~/qiv.log 2>&1
12 10 * * * ~/ >> ~/master.log 2>&1

Use crontab -e to edit your crontab file.

qiv – an easy install
To install qiv

$ sudo apt-get install qiv

Rclone shown in some detail
The real magic is tapping into the Google Drive, which is done with rclone. There are older packages but they are awful by comparison so don’t waste your time on any other package.

$ sudo apt-get install rclone
$ rclone config

2019/08/05 20:22:42 NOTICE: Config file "/home/pi/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Choose a number from below, or type in your own value
 1 / Amazon Drive
   \ "amazon cloud drive"
 2 / Amazon S3 (also Dreamhost, Ceph, Minio)
   \ "s3"
 3 / Backblaze B2
   \ "b2"
 4 / Dropbox
   \ "dropbox"
 5 / Encrypt/Decrypt a remote
   \ "crypt"
 6 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
 7 / Google Drive
   \ "drive"
 8 / Hubic
   \ "hubic"
 9 / Local Disk
   \ "local"
10 / Microsoft OneDrive
   \ "onedrive"
11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
12 / Yandex Disk
   \ "yandex" 
Google Application Client Id
Leave blank normally.
Enter a string value. Press Enter for the default ("").
Google Application Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine or Y didn't work
y) Yes
n) No
y/n> N
If your browser doesn't open automatically go to the following link:
Log in and authorize rclone for access

You sign in to your Google account with a regular browser.

After sign-in you see:

rclone wants to access your Google Account
This will allow rclone

See, edit, create, and delete all of your Google Drive files

Make sure you trust rclone

After clicking Allow you get:

Please copy this code, switch to your application and paste it there:
Enter verification code>4/nQEXJZOTdP_asMs6UQZ5ucs6ecvoiLPelQbhI76rnuj4sFjptxbjm7w
client_id =
client_secret =
token = {"access_token":"ya29.Il-KB3eniEpkdUGhwdi8XyZyfBFIF2ahRVQtrr7kR-E2lIExSh3C1j-PAB-JZucL1j9D801Wbh2_OEDHthV2jk_MsrKCMiLSibX7oa_YtFxts-V9CxRRUirF1_kPHi5u_Q","token_type":"Bearer","refresh_token":"1/MQP8jevISJL1iEXH9gaNc7LIsABC-92TpmqwtRJ3zV8","expiry":"2019-09-21T08:34:19.251821011-04:00"}
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name                 Type
====                 ====
remote               drive
e) Edit existing remote
n) New remote
d) Delete remote
s) Set configuration password
q) Quit config

Note you can very well keep the root folder id blank. In my case we store all our pictures in one top-level folder and the nested folders get pretty deep, plus there’s a busload of other things on the drive, so I wanted to give rclone the best possible shot at running well. Still, listing our 40,000+ pictures takes 90 seconds or so.

Goofed up your config of rclone? No worries. Remove .config/rclone and start over.

Don’t forget to make all these scripts executable (chmod +x <script_name&gt:)or you will end up seeing messages like this:

-bash: ./ Permission denied

Some noteworthy rclone commands
rclone ls remote: – lists all files, going recursively, no problem with MORE
rclone lsd remote: lists directories in top level of drive
rclone copy remote:”MaryDocs/Pictures and videos/Shutterfly books collection of photos/JJH birth photos/img2165.jpg” . : copies picture to current directory (does not create directory hierarchy)

Do a complete directory listing, capture the results in a file and see how long it took:
$ time rclone ls remote: > lsf-complete

real    1m12.201s
user    0m15.270s
sys     0m1.816s

My initial thought was to do a remote mount of the Google Drive onto a Raspberry Pi mount point, but it’s just so slow that it really provides no advantage to do it that way.

Some encountered issues
Well, I blew up on crontab, which in all my years working with linux/unix I’ve never done before. But I managed to fix it.

Prior to discovering rclone I made the mistake of using gdrivefs to create a mounted Google Drive – sounds great in principle, right? What a disaster. The files’ binary data were not correctly preserved when accessed through the mount though the size was! I have also never encountered a mounting software that corrupted files, but this piece of garbage does. One way to detect corruption in a binary file is to do a cksum (or md5sum, just be consistent and use one or the other) of source file and destination version of same file. The result should be the same number.

Imagined but avoided issue: JPEG orientation

I had prepared a whole python program to orient my pictures correctly, but lo and behold I “discovered” that the -l switch in qiv does that for you! So I actually ripped that whole unnecessary step out.

Re-purposing equipment I had lying around: Raspberry Pi 3, Pi Display, and 40,000 JPEG images on Google Drive, I put together a novel photoframe slideshow which randomly displays a different set of 60 pictures each day. It’s a nice way for us to be exposed to our collection of 17+ years of digital photos.

The qiv really is a quick image viewer, i.e., the slideshow runs clean, like a real one.

Long Todo list

  • Improve selection of recent pictures if we’ve just uploaded a bunch of pictures from our smartphones.
  • Hey, how about also showing some of those short videos we also shot with our camera phones and uploaded to Google Drive? And while we’re at it, re-purposing those cheap USB speakers I bought for RetroPi gaming to get the sound, or play a soundtrack!?
  • I realize that although the selection of the 20 anchor pictures is initially random, when they plus the 40 additional photos are presented for display additional order is imposed by the shell’s expansion of the regex and this has a tendency to make the pictures more chronologically organized than they would be by chance.
  • References and related

    RetroPi, the gaming emulation project for which I bought economical USB speakers.

    The rclone home page.

    A detailed write-up on using pipresents program where we had a Raspberry Pi drive a mixed media display 9pictures and videos) for a kiosk.

    IT Operational Excellence Network Technologies Web Site Technologies

    F5 Big-IP: When your virtual server does not present your chain certificate

    While I was on vacation someone replaced a certificate which had expired on the F5 Big-IP load balancer. Maybe they were not quite as careful as I would like to hope I would have been. In any case, shortly afterwards our SiteScope monitoring reported there was an untrusted server certificate chain. It took me quite some digging to get to the bottom of it.

    The details
    Well, the web site came up just fine in my browser. I checked it with SSLlabs and its grade was capped at B because of problems with the server certificate chain. I also independently confirmed usnig openssl that no intermediate certificate was being presented by this virtual server. To see what that looks like with an exampkle of this problem knidly privided by, do:

    $ openssl s_client ‐showcerts ‐connect

    depth=0 /C=US/ST=California/L=San Francisco/O=BadSSL Fallback. Unknown subdomain or no SNI./CN=badssl-fallback-unknown-subdomain-or-no-sni
    verify error:num=20:unable to get local issuer certificate
    verify return:1
    depth=0 /C=US/ST=California/L=San Francisco/O=BadSSL Fallback. Unknown subdomain or no SNI./CN=badssl-fallback-unknown-subdomain-or-no-sni
    verify error:num=27:certificate not trusted
    verify return:1
    depth=0 /C=US/ST=California/L=San Francisco/O=BadSSL Fallback. Unknown subdomain or no SNI./CN=badssl-fallback-unknown-subdomain-or-no-sni
    verify error:num=21:unable to verify the first certificate
    verify return:1
    Certificate chain
     0 s:/C=US/ST=California/L=San Francisco/O=BadSSL Fallback. Unknown subdomain or no SNI./CN=badssl-fallback-unknown-subdomain-or-no-sni
       i:/C=US/ST=California/L=San Francisco/O=BadSSL/CN=BadSSL Intermediate Certificate Authority
    -----END CERTIFICATE-----
    Server certificate
    subject=/C=US/ST=California/L=San Francisco/O=BadSSL Fallback. Unknown subdomain or no SNI./CN=badssl-fallback-unknown-subdomain-or-no-sni
    issuer=/C=US/ST=California/L=San Francisco/O=BadSSL/CN=BadSSL Intermediate Certificate Authority
        Verify return code: 21 (unable to verify the first certificate)

    So you get that message about benig unable to verify the first certificate.

    Here’s the weird thing, the certificate in question was issued by Globalsign, and we have used them for years so we had the intermediate certificate configured already in the SSL client profile. The so-called chain certificate was GlobalsignIntermediate. But it wasn’t being presented. What the heck? Then I checked someone else’s Globalsign certificate and found the same issue.

    Then I began to get suspicious about the certificate. I checked the issuer more carefully and found that it wasn’t from the intermediate we had been using all these past years. Globalsign changed their intermediate certificate! The new one dates frmo November 2018 and expires in 2028.

    And, to compound matters, F5 “helpfully” does not complain and simply does not send the wrong intermediate certificate we had specified in the SSL client profile. It just sends no intermediate certificate at all to accompany the server certificate.

    The case of the missing intermediate certificate was resolved. It is not the end of the world to miss an intermediate certificate, but on the other hand it is not professional either. Sooner or later it will get you into trouble.

    References and related is a great resource.
    My favorite openssl commands can be very helpful.

    Admin Linux Network Technologies Raspberry Pi Security Web Site Technologies

    How to test if a web site requires a client certificate

    I can not find a link on the Internet for this, yet I think some admins would appreciate a relatively simple test to know is this a web site which requires a client certificate to work? The errors generated in a browser may be very generic in these situations. I see many ways to offer help, from a recipe to a tool to some pointers. I’m not yet sure how I want to proceed!

    why would a site require a client CERT? Most likely as a form of client authentication.

    Pointers for the DIY crowd plus access to a linux command line – such as using a Raspberry Pi I so often write about – will do it for you guys.

    The Client Certificate section of has most of what you need. The page is getting big, look for this:

    So as a big timesaver has created a client certificate for you which you can use to test with. Download it as follows.

    Go to your linux prompt and do something like this:
    $ wget‐client.pem has a web page you can test with which only shows success if you access it using a client certificate,

    to see how this works, try to access it the usual way, without supplying a client CERT:

    $ curl ‐i ‐k

    HTTP/1.1 400 Bad Request
    Server: nginx/1.10.3 (Ubuntu)
    Date: Thu, 20 Jun 2019 17:53:38 GMT
    Content-Type: text/html
    Content-Length: 262
    Connection: close
    <head><title>400 No required SSL certificate was sent</title></head>
    <body bgcolor="white">
    <center><h1>400 Bad Request</h1></center>
    <center>No required SSL certificate was sent</center>
    <hr><center>nginx/1.10.3 (Ubuntu)</center>

    Now try the same thing, this time using the client CERT you just downloaded:

    $ curl ‐v ‐i ‐k ‐E ./‐

    * About to connect() to port 443 (#0)
    *   Trying connected
    * Connected to ( port 443 (#0)
    * Initializing NSS with certpath: sql:/etc/pki/nssdb
    * warning: ignoring value of ssl.verifyhost
    * skipping SSL peer certificate verification
    * NSS: client certificate from file
    *       subject: CN=BadSSL Client Certificate,O=BadSSL,L=San Francisco,ST=California,C=US
    *       start date: Nov 16 05:36:33 2017 GMT
    *       expire date: Nov 16 05:36:33 2019 GMT
    *       common name: BadSSL Client Certificate
    *       issuer: CN=BadSSL Client Root Certificate Authority,O=BadSSL,L=San Francisco,ST=California,C=US
    * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    * Server certificate:
    *       subject: CN=*,O=Lucas Garron,L=Walnut Creek,ST=California,C=US
    *       start date: Mar 18 00:00:00 2017 GMT
    *       expire date: Mar 25 12:00:00 2020 GMT
    *       common name: *
    *       issuer: CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US
    > GET / HTTP/1.1
    > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
    > Host:
    > Accept: */*
    < HTTP/1.1 200 OK
    HTTP/1.1 200 OK
    < Server: nginx/1.10.3 (Ubuntu)
    Server: nginx/1.10.3 (Ubuntu)
    < Date: Thu, 20 Jun 2019 17:59:08 GMT
    Date: Thu, 20 Jun 2019 17:59:08 GMT
    < Content-Type: text/html
    Content-Type: text/html
    < Content-Length: 662
    Content-Length: 662
    < Last-Modified: Wed, 12 Jun 2019 15:43:39 GMT
    Last-Modified: Wed, 12 Jun 2019 15:43:39 GMT
    < Connection: keep-alive
    Connection: keep-alive
    < ETag: "5d011dab-296"
    ETag: "5d011dab-296"
    < Cache-Control: no-store
    Cache-Control: no-store
    < Accept-Ranges: bytes
    Accept-Ranges: bytes
    <!DOCTYPE html>
      <meta name="viewport" content="width=device-width, initial-scale=1">
      <link rel="shortcut icon" href="/icons/favicon-green.ico"/>
      <link rel="apple-touch-icon" href="/icons/icon-green.png"/>
      <link rel="stylesheet" href="/style.css">
      <style>body { background: green; }</style>
    <div id="content">
      <h1 style="font-size: 12vw;">
    <div id="footer">
      This site requires a <a href="">client-authenticated</a> TLS handshake.
    * Connection #0 to host left intact
    * Closing connection #0

    No more 400 error status – that looks like success to me. Note that we had to provide the password for our client CERT, which they kindly provided as

    Here’s an example of a real site which requires client CERTs:

    $ curl ‐v ‐i ‐k ‐E ./‐

    * About to connect() to port 443 (#0)
    *   Trying connected
    * Connected to ( port 443 (#0)
    * Initializing NSS with certpath: sql:/etc/pki/nssdb
    * warning: ignoring value of ssl.verifyhost
    * skipping SSL peer certificate verification
    * NSS: client certificate from file
    *       subject: CN=BadSSL Client Certificate,O=BadSSL,L=San Francisco,ST=California,C=US
    *       start date: Nov 16 05:36:33 2017 GMT
    *       expire date: Nov 16 05:36:33 2019 GMT
    *       common name: BadSSL Client Certificate
    *       issuer: CN=BadSSL Client Root Certificate Authority,O=BadSSL,L=San Francisco,ST=California,C=US
    * NSS error -12227
    * Closing connection #0
    * SSL connect error
    curl: (35) SSL connect error

    OK, so you get an error, but that’s to be expected because our certificate is not one it will accept.

    The point is that if you don’t send it a certificate at all, you get a different error:

    $ curl ‐v ‐i ‐k

    * About to connect() to port 443 (#0)
    *   Trying connected
    * Connected to ( port 443 (#0)
    * Initializing NSS with certpath: sql:/etc/pki/nssdb
    * warning: ignoring value of ssl.verifyhost
    * skipping SSL peer certificate verification
    * NSS: client certificate not found (nickname not specified)
    * NSS error -12227
    * Closing connection #0
    curl: (35) NSS: client certificate not found (nickname not specified)

    See that client certificate not found? That is the error we eliminated by supplying a client certificate, albeit one which it will not accept.

    what if we have a client certificate but we use the wrong password? Here’s an example of that:

    $ curl ‐v ‐i ‐k ‐E ./‐client.pem:badpassword

    * About to connect() to port 443 (#0)
    *   Trying connected
    * Connected to ( port 443 (#0)
    * Initializing NSS with certpath: sql:/etc/pki/nssdb
    * warning: ignoring value of ssl.verifyhost
    * Unable to load client key -8025.
    * NSS error -8025
    * Closing connection #0
    curl: (58) Unable to load client key -8025.

    Chrome gives a fairly intelligible error

    Possibly to be continued…

    We have given a recipe for testing form a linux command line if a web site requires a client certificate or not. thus it could be turned into a program

    References and related
    My article about ciphers has been popular.

    I’ve also used for other related tests.

    Can you use openssl directly? You’d hope so, but I haven’t had time to explore it… Here are my all-time favorite openssl commands. – lots of cool tests here. The creators have been really thorough.

    Admin Network Technologies

    Postfix Operational tips

    I’m trying out the system-supplied postfix on a SLES system. i had been using sendmail but there doesn’t seem to be any development on that software.

    Some commands I needed right away
    Well, right away I had thousands of queued messages so I needed a way to make sense of what was happening.

    For these commands to make sense you need to know that I am running a second postfix configuraiton out of /etc/postfixEXT.

    Display the queue

    postqueue -c /etc/postfixEXT -p

    Force delivery from the queue

    postqueue -c /etc/postfixEXT -f

    List one email in detail

    postcat -vq -c /etc/postfixEXT QUEUEID

    Delete one email

    postsuper -c /etc/postfixEXT -d QUEUEID

    Put mail on hold

    postsuper -c /etc/postfixEXT -h ALL|QUEUEID

    Release mail form hold

    postsuper -c /etc/postfixEXT -H ALL|QUEUEID

    How to force delivery of a single message
    This command is not documented anywhere – because it doesn’t exist so you have to get creative. If you have the luxury of halting all email for a few seconds simply do this:

    Display the queue to find the queue ID of the email you want to force delivery of

    postqueue -c /etc/postfixEXT -p

    Put all mail on hold

    postsuper -c /etc/postfixEXT -h ALL

    Now release the hold on that one email

    postsuper -c /etc/postfixEXT -H QUEUEID

    QUEUEID is, of course, the queue id .e.g., F2A1A27891E, of the email in question.

    Look for what happened
    Check your mail log’s last lines in /var/log/mail

    Revert back to normal running

    postsuper -c /etc/postfixEXT -H ALL

    Since mail is store-and-forward and not real time, you can do these steps, quickly, even on a production system and no one will be the wiser if you are pretty quick. Probably takes two minutes even for a slow typer.

    How to run multiple listeners
    I didn’t want to disturb the system-installed postfix too much. I would let it “have” the loopback address,, leaving me the other IPs for my relay config to listen on. I added these lines to /etc/postfix/

    multi_instance_enable = yes
    multi_instance_directories = /etc/postfixEXT

    service postfix start starts up the local postfix plus my relay. Grep the process table for either master or postfix to see. However, to be honest, service postfix stop does not kill all processes. So I always end up killing one of the master processes by hand. Update: postmulti -p stop does the trick to kill all. There is also a status or start option instead of stop.

    Sendmail to Postfix migration tips
    This could be a separate post but I am too lazy to do that.

    What happens to the access file? I kept the name of the file access but just list all the IPs, one per line, without any further arguments, to permit just those IPs relay access. In my I have a line like this to tie it together:

    mynetworks = /etc/postfixEXT/access

    Note that there is no hashed or .db version of this file any longer, unlike in the sendmail case.

    References and related
    Since I mentioned sendmail I have to give a shout out to one of my old sendmail posts.

    More info on postfix multiple instances. A pretty complete giude.

    Linux Network Technologies Raspberry Pi

    Live stream to YouTube from a Raspberry Pi + webcam or USB microphone

    I’ve been looking at this off and on for awhile now. I finally made a breakthrough this week and started to generate some decent live streams on my Youtube channel, after a lot of misfires.

    Note this is applicable for Raspbian Stretch Lite on a Raspberry Pi 3. However, I firmly believe it will work just the same for regular Raspbian Stretch.

    There’s a lot of wrong, misleading or outdated information out there on the Internet. Hopefully this will help others to avoid wasting as much time as I had to do.

    This project was prompted by my desire to make a more generalized fishcam! Described in this post, my original fishcam implementation – and I realized this form the get-go – has very limited applicability because very few people are in a position to have their own AWS server. And if you don’t know what you’re doing, please don’t run your own server – the security exposure is too great.

    So I eventually realized that maybe I could generalize what I had done – essentially remove the dependency on the AWS server – by utilizing Youtube Live Streaming. And, I believe I was right. It’s still a work in progress however.

    The command – ffmpeg
    I was playing with ffmpeg. The version I am playing with now comes with Raspbian – no need to compile like in the bad old days. ffmpeg -version shows the version to be 3.2.12. I get the impression that its capabilities are version-dependent, so that’s why this information is particularly relevant in this case.

    The details
    In some of my early attempts I was getting a lot of this (looking at YouTube Live Dashboard)

    Dashboard When stream is not quite right

    Another attempt
    Video works, audio like driving in a car with the windows down. For the record, the command was this:

    ffmpeg \
    -f alsa -i plughw:CARD=U0x46d0x825,DEV=0 \
    -f v4l2 -i /dev/video0 \
    -c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 2500k \
    -bufsize 512k \
    -acodec libmp3lame -ar 44100 \
    -threads 2 -qscale 3 \
    -b:a 96K \
    -r 10 \
    -s 1280x720 \
    -f flv rtmp://
    Video OK, audio choppy message

    For the record, the bandwidth required was about 2100 kbps.

    List the formats your video device supports

    ffmpeg -f video4linux2 -list_formats all -i /dev/video0

    Results using my Logitech Webcam

    [video4linux2,v4l2 @ 0xcc45c0] Raw       :     yuyv422 :           YUYV 4:2:2 : 640x480 160x120 176x144 320x176 320x240 352x288 432x240 544x288 640x360 752x416 800x448 800x600 864x480 960x544 960x720 1024x576 1184x656 1280x720 1280x960
    [video4linux2,v4l2 @ 0xcc45c0] Compressed:       mjpeg :          Motion-JPEG : 640x480 160x120 176x144 320x176 320x240 352x288 432x240 544x288 640x360 752x416 800x448 800x600 864x480 960x544 960x720 1024x576 1184x656 1280x720 1280x960
    ffmpeg \
    -f alsa -i plughw:CARD=U0x46d0x825,DEV=0 \
    -f v4l2 -i /dev/video0 \
    -c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 1200 \
    -bufsize 512k \
    -acodec libmp3lame -ar 44100 \
    -threads 2 -qscale 3 \
    -b:a 128K \
    -r 5 \
    -s 640x480 \
    -f flv rtmp://

    Audio good, video not working

    video terrible, but audio good!

    It is not so pretty to use that hardware address for the Logitech webcam device. Where do you see that hardware address? Either a lsusb or a ls /dev/snd/by-id shows addresses of sound devices. I found a simpler substitute:

    ffmpeg \
    -f alsa -i plughw:1,0 \
    -f v4l2 -i /dev/video0 \
    -c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 1200k \
    -bufsize 512k \
    -acodec libmp3lame -ar 44100 \
    -threads 2 -qscale 3 \
    -b:a 128k \
    -r 5 \
    -s 640x480 \
    -f flv rtmp://
    With this audio's, not too bad, video's a bit choppy. Google reports the stream quality as OK, check resolution.
    So I fix the bandwidth (which was a typo in the above, but one with an interesting result). I set video bandwidth to -b:v 1200k. Now the video is OK once again, but the audio is choppy again! Weird. bandwidth is about 1100 kbps.
    This version had OK video and OK audio
    <pre lang="text'>
    ffmpeg \
    -f alsa -i plughw:CARD=U0x46d0x825,DEV=0 \
    -f v4l2 -i /dev/video0 \
    -c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 1600k \
    -bufsize 512k \
    -acodec libmp3lame -ar 44100 \
    -threads 2 -qscale 3 \
    -b:a 128k \
    -r 5 \
    -s 640x480 \
    -f flv rtmp://

    But I keep getting inconsistent results! Sometimes a setting will work, and then I come back to it and it doesn’t. Weird.

    Part of the problem is that I have no idea what I’m doing and I didn’t know when i was watching a livestream vs a recorded (on-demand0 one! I have since learned to look for the little red Live button. A picture is worth 10^3 words in this case.

    Observed used bandwidth is about 1450 kbits/sec. But still lots of dropped packets. Here is what ffmpeg reports. I’m not sure yet what most of it means:

    [alsa @ 0x1502700] ALSA buffer xrun.
    [alsa @ 0x1502700] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)
    frame= 5828 fps=5.0 q=-1.0 Lsize=  205496kB time=00:19:26.20 bitrate=1443.5kbits/s dup=0 drop=11138 speed=   1x
    video:187265kB audio:17449kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.382063%
    [libx264 @ 0x15100e0] frame I:583   Avg QP: 9.41  size: 53819
    [libx264 @ 0x15100e0] frame P:5245  Avg QP:13.53  size: 30578
    [libx264 @ 0x15100e0] mb I  I16..4: 100.0%  0.0%  0.0%
    [libx264 @ 0x15100e0] mb P  I16..4: 38.0%  0.0%  0.0%  P16..4: 60.7%  0.0%  0.0%  0.0%  0.0%    skip: 1.4%
    [libx264 @ 0x15100e0] coded y,uvDC,uvAC intra: 93.7% 86.2% 82.4% inter: 77.8% 60.5% 34.1%
    [libx264 @ 0x15100e0] i16 v,h,dc,p: 17% 23% 15% 45%
    [libx264 @ 0x15100e0] i8c dc,h,v,p: 51% 21% 16% 11%
    [libx264 @ 0x15100e0] kb/s:1315.22

    The video for that run is here:

    Suppressing Audio
    This is what worked for me.

    ffmpeg \
    -f lavfi -i anullsrc=channel_layout=stereo:sample_rate=44100 \
    -f v4l2 -i /dev/video0 \
    -c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 1600k \
    -bufsize 512k \
    -acodec libmp3lame -ar 44100 \
    -threads 2 -qscale 3 \
    -b:a 128k \
    -r 5 \
    -s 640x480 \
    -f flv rtmp://

    That is working great – showing the video as before but now with a silent audio track.

    Increase Video Quality
    Here I’ve increased video quality a tad by requesting more fps (10) and making qscale 0 (which means highest quality).

    ffmpeg \
    -f alsa -i plughw:1,0 \
    -f v4l2 -i /dev/video0 \
    -c:v libx264 -pix_fmt yuv420p -preset ultrafast -b 3000k -g 20 -b:v 1800k \
    -bufsize 512k \
    -acodec libmp3lame -ar 44100 \
    -threads 4 -qscale 0 \
    -b:a 128k \
    -r 10 \
    -s 640x480 \
    -f flv rtmp://

    Bitrate was about 1700 kbps. Quality is maybe a little better. Audio still leaves something to be desired.

    Still better video quality

    ffmpeg \
    -f alsa -i plughw:1,0 \
    -f v4l2 -i /dev/video0 \
    -c:v libx264 -pix_fmt yuv420p -preset ultrafast -b 3000k -g 60 -b:v 2000k \
    -bufsize 512k \
    -acodec libmp3lame -ar 44100 \
    -threads 4 -qscale 0 \
    -b:a 128k \
    -r 30 \
    -s 640x480 \
    -f flv rtmp://

    What is observed to happen is that ffmpeg actually chooses 15 fps rather than 30. I’ve read it decides what it is able to do, so maybe that’s the highest fps it can deliver. Video is pretty smooth (See my Livestream link in references if I happen to have it running. Otherwise I will create a video link.) No drops are recorded, but the sound, though not terrible, has some pops. Bandwidth used is about 1900 kbps. So this is definitely my best effort yet. YouTube complains about the unsupported video size of 640×480, but it permits it and I don’t think it’s a real problem.

    Reducing bandwidth
    This one is pretty good overall. I have no idea why lowering the audio bandwidth might help. It’s counter intuitive. But video motion is not bad – just a tad blurred. I guess q=23. Audio has good patches and not-as good patches. Not as good spots are staticky, not washboard bad. Total bandwidth used is about 611 kbps. So a great compromise. Why does raising the video bandwidth lower the audio quality? I have no idea… The settings below worked for maybe 20 minutes, then YouTube said this Video is unavailable. I at least found out something about that. That shows a problem with the player, not (for once) your stream. so since I’m only concentrating on the stream, that’s good news. So actually it delivered good sound for three hours straight with a few staticky spots.

    ffmpeg \
    -thread_queue_size 1024 \
    -f alsa -i plughw:1,0 \
    -thread_queue_size 256 \
    -f v4l2 -i /dev/video0 \
    -c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 30 -b:v 450k \
    -bufsize 512k \
    -acodec libmp3lame -ar 44100 \
    -threads 4 -q:v 5 \
    -q:a 0 \
    -b:a 64k \
    -r 15 \
    -s 480x320 \
    -f flv rtmp://

    The audio is creepily sensitive, easily picking up conversations in adjacent rooms.

    But then I monkeyed around with the settings, got the washboard sound, came back to this one – a known good – and got washboard audio! What the heck? Why isn’t it consistent?? No idea… Maybe it’s the player that gets messed up?? Now I’m running it again and it’s OK.

    Bandwidth talk
    It’s important to talk about bandwidth if you haven’t given this any real thought. You have to have a halfway decent broadband connection for this to work, you see? If you have a mid-speed cable modem or DSL, you have much lower upload than download speeds, and you may not be able to pull off a reliable 1.5 mbps upload. For those lucky enough to have Verizon FIOS this is a non-issue. But for instance in the high school where I volunteer they have throttled the guest WiFi network to such an extent that achieving this modest 1.5 mbps is going to present a real challenge. If you rely on a phone’s hotspot you will also probably be unable to get such a speed. So I may look at more ways to reduce the bandwidth required in the future.

    Check your bandwidth using

    And between YouTube and your ISP, it just seems the whole thing about live video broadcasting seems, well, delicate. Stream Health varies between oK, to Excellent to not receiving – all during the same streaming session! It often takes five minutes or so for the stream to appear to be working.

    Comparing two webcams
    Someone picked up a really cheap DI Chatcam at Microcenter in Paterson. I think that’s Digital Innovations Chatcam. It’s cute. It has a big clip on the end and shines white LEDs when it’s on. I think it was about $12. With the exact same ffmpeg settings (with audio suppressed), the quality was not nearly as good as with the Logitech webcam. Here’s a link to the YouTube video made with the chatcam: Note that it has a ministereo plug for audio. I didn;’t even plug it in now that I know how to suppress audio!

    The Logitech model is a C525. It was a refurbished model which cost me about $27.The comparable Logitech webcam test is here:

    I need to re-run this test now that I know how to increase the video quality.

    A breakthrough: publishing an audio-only stream to YouTube
    Besides covering your lens with tape, what’s a software way to blacken the video and concentrate on producing the best audio I wondered?

    ffmpeg \
    -thread_queue_size 4096 \
    -f alsa -i plughw:1,0 \
    -thread_queue_size 128 \
    -f lavfi -i color=color=darkgray \
    -c:v libx264 -pix_fmt yuv420p -b:v 100k \
    -bufsize 512k \
    -acodec libmp3lame -ar 44100 \
    -threads 8 \
    -b:a 128k \
    -r 30 \
    -s 1280x720 \
    -f flv rtmp://

    The above gives me good audio, and a sold gray background. I love it – for recording band practice or whatever. The breakthrough is that we can avoid wasting cpu cycles on processing input video but just use a color. Thanks Stackoverflow for the tip. Used bandwidth is about 150 kbs – basically nothing! YouTube Dsahboard complains:

    OK Video output low
    The stream's current bitrate (138.00 Kbps) is lower than the recommended bitrate. 
    We recommend that you use a stream bitrate of 2500 Kbps.

    But of course that is bogus because that assumes we are trying to put out a rich 1280×720 video, which we are not.

    Then eventually YouTube has this complaint:

    Bad Bad video settings
    Please use a keyframe frequency of four seconds or less. Currently, keyframes are not being sent often enough, which will cause buffering. 
    The current keyframe frequency is 8.5 seconds. Note that ingestion errors can cause incorrect GOP (group of pictures) sizes.

    Yet the stream does not seem to suffer in any noticeable way from this problem.

    For good measure, we add a few extra arguments allow us to remove the keyframes warning. We need to use the -g parameter (group of pictures) at about twice our frame rate, plus, maybe, a no-scenecut argument. Here’s that version.

    ffmpeg \
    -thread_queue_size 4096 \
    -f alsa -i plughw:1,0 \
    -thread_queue_size 128 \
    -f lavfi -i color=color=darkgray \
    -c:v libx264 -pix_fmt yuv420p -g 60  -x264opts no-scenecut -b:v 150k \
    -bufsize 512k \
    -acodec libmp3lame -ar 44100 \
    -threads 8 \
    -b:a 128k \
    -r 30 \
    -s 1280x720 \
    -f flv rtmp://

    Actual fps is 25, quality is 26 and bitrate is 145 kbps. But audio quality is good. I hear white noise in the background, but hey, this isn’t exactly professional equipment we’re working with. But this is a great solution for an audio-only recording that goes straight out to YouTube. stability is also good.

    The load average is high – 3.6 (use top to watch it), almost all of it taken by ffmpeg. So it appears ffmpeg is really working it to produce this audio stream. That makes me suspect it just gets overwhelmed when it’s an audio + video stream? Because I never did find setting swhich produced good quality for both…

    Switch to Wifi and Yet another problem surfaces
    It seems that with this livestreaming project everything that should just work doesn’t! I had been doing all my testing used wired Ethernet connection and WiFi disabled. anticipating a portable solution, I tried it using WiFi and no Ethernet cable. And washboard audio reappeared. quite often ffmpeg hangs as well. I tried a zillion experiments and now my revelation is that essentially, though we tried to minimize and trivialize video, we were probably still overwhelming the CPU. So I reasoned that these actions will make the load easier on the CPU, without compromising the audio quality:

    – reduce frame per second dramatically
    – reduce key frames
    – reduce video size

    And…yes, these things in combination really did help and permit me to run over WiFi now. This version, put inside a script I call, looks like this:

    ffmpeg \
    -thread_queue_size 4096 \
    -f alsa -i plughw:1,0 \
    -thread_queue_size 64 \
    -f lavfi -i color=color=darkgray \
    -c:v libx264 -pix_fmt yuv420p -g 18  -x264opts no-scenecut -b:v 50k \
    -bufsize 512k \
    -acodec libmp3lame -ar 44100 \
    -threads 8 \
    -b:a 128k \
    -r 5 \
    -s 480x320 \
    -f flv rtmp://

    It doesn’t start consistently, however, but if you run it enough times it’ll go. So, to provide reliability I also scripted around these deficiencies: I decided to just keep trying to start up until I jhave evidence it’s working. I call that script

    # DrJ 5/2019
    LOG="ff.log"`date +%m-%d-%y:%H:%M`
    while /bin/true; do
     nohup ./</dev/null>$LOG 2>&1 &
     sleep 7
    # want like
    #Frame=   84 fps= 11 q=16.0 size=      43kB time=00:00:07.50 bitrate=  47.1kbits/s dup=0 drop=431 speed=0.991x
    #Frame=   84 fps= 11 q=16.0 size=      43kB time=00:00:07.50 bitrate=  47.1kbits/s dup=0 drop=431 speed= 1x
     FFOUT=`tail -1 $LOG`
     echo "last line is $FFOUT"
     KB=`echo $FFOUT|awk '{print $(NF-4)}'`
     echo "orig KB: $KB"
     KB=`echo $FFOUT|awk '{print $(NF-5)" "$(NF-4)}'|sed 's/kbits.*//'|awk '{print $NF}'`
     echo "KB is: $KB"
     if [ $KB -gt 129 2>/dev/null ]; then
    # let our master process exit - we've got a good audio stream
       echo "Exiting at *** "`date`
    # didn't work out: restart and try again
      echo "*** Restarting ffmpeg at *** "`date`
      pkill -9 -f 'ffmpeg '

    And…it works great! Very briefly what it does is t that it calls and backgrounds it, then tests its output. It gives it a few seconds to get going, then kills it unless observed streaming bandwidth is a healthy 135 kbps or so (essentially the video takes almost no bandwidth in

    Putting it all together – livestreaming audio stream to YouTube automatically upon boot up
    So I want to drag this thing to a performance and have a confederate with minimal technical know-how start it up. So basically I want it to start livestreaming when the RasPi is powered up. To do that I made this crontab entry (using crontab -e):

    @reboot sleep 20; /home/pi/ > ff.log 2>&1

    It takes a few minutes to get going, but it’s been extremely reliable. It’s started a stream successfully more than 10 times out of 10, at least when I was using my home WiFi connection. When I switched to my phone’s Hotspot, I had one error out of five attempts. The one bad stream just would not start according to Youtube, although per the stats from the log files showed the stream reached the usual good bandwidth. So I don’t know…

    And once the stream starts, it is running uninterrupted for hours, anywhere from three to six hours.

    Eventually I want to write an API program to automatically check the stream. But before then I may just introduce a refined script which checks the output and restarts ffmpeg when it has ended.

    For the record, a typical ff.log file looks like this:

    frame=   43 fps= 43 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A dup=0 drop=164 speed=   0x    ed=   0x
    orig KB: dup=0
    Tue  7 May 12:32:08 BST 2019
    KB is: dup=0
    *** Restarting ffmpeg at *** Tue 7 May 12:32:08 BST 2019
    frame=  213 fps= 35 q=8.0 size=      47kB time=00:01:40.91 bitrate=   3.8kbits/s dup=0 drop=847 speed=16.7x
    orig KB: 3.8kbits/s
    Tue  7 May 12:38:53 BST 2019
    KB is: 3
    *** Restarting ffmpeg at *** Tue 7 May 12:38:53 BST 2019
    illed=   86 fps= 14 q=8.0 size=     104kB time=00:00:06.21 bitrate= 136.7kbits/s dup=0 drop=336 speed=1.03x
    orig KB: 136.7kbits/s
    Tue  7 May 12:39:00 BST 2019
    KB is: 136
    Exiting at *** Tue 7 May 12:39:00 BST 2019

    The other file, which has a name like ff.log05-07-19:12:32, looks more like this:

    ffmpeg version 3.2.12-1~deb9u1+rpt1 Copyright (c) 2000-2018 the FFmpeg developers
      built with gcc 6.3.0 (Raspbian 6.3.0-18+rpi1+deb9u1) 20170516
      configuration: --prefix=/usr --extra-version='1~deb9u1+rpt1' --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf -
    -incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gn
    utls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libebur
    128 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --ena
    ble-libmp3lame --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-
    libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --ena
    ble-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --enab
    le-libzvbi --enable-omx --enable-omx-rpi --enable-mmal --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --e
    nable-libiec61883 --arch=armhf --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
      libavutil      55. 34.101 / 55. 34.101
      libavcodec     57. 64.101 / 57. 64.101
      libavformat    57. 56.101 / 57. 56.101
      libavdevice    57.  1.100 / 57.  1.100
      libavfilter     6. 65.100 /  6. 65.100
      libavresample   3.  1.  0 /  3.  1.  0
      libswscale      4.  2.100 /  4.  2.100
      libswresample   2.  3.100 /  2.  3.100
      libpostproc    54.  1.100 / 54.  1.100
    Guessed Channel Layout for Input Stream #0.0 : stereo
    Input #0, alsa, from 'plughw:1,0':
      Duration: N/A, start: 1557229134.030863, bitrate: 1536 kb/s
        Stream #0:0: Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s
    Input #1, lavfi, from 'color=color=darkgray':
      Duration: N/A, start: 0.000000, bitrate: N/A
        Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
    [libx264 @ 0x12db850] VBV maxrate unspecified, assuming CBR
    [libx264 @ 0x12db850] using SAR=8/9
    [libx264 @ 0x12db850] using cpu capabilities: ARMv6 NEON
    [libx264 @ 0x12db850] profile High, level 2.1
    [libx264 @ 0x12db850] 264 - core 148 r2748 97eaef2 - H.264/MPEG-4 AVC codec - Copyleft 2003-2016 -
    x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_ran
    ge=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=8 lookahead_threads=1 sl
    iced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 di
    rect=1 weightb=1 open_gop=0 weightp=2 keyint=18 keyint_min=1 scenecut=0 intra_refresh=0 rc_lookahead=40 rc=cbr mbtree=1 bit
    rate=50 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 vbv_maxrate=50 vbv_bufsize=512 nal_hrd=none filler=0 ip_ratio=1.40
    Output #0, flv, to 'rtmp://
        encoder         : Lavf57.56.101
        Stream #0:0: Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuv420p, 480x320 [SAR 8:9 DAR 4:3], q=-1--1, 50 kb/s, 5 fps
    , 1k tbn, 5 tbc
          encoder         : Lavc57.64.101 libx264
        Side data:
          cpb: bitrate max/min/avg: 0/0/50000 buffer size: 512000 vbv_delay: -1
        Stream #0:1: Audio: mp3 (libmp3lame) ([2][0][0][0] / 0x0002), 44100 Hz, stereo, s16p, 128 kb/s
          encoder         : Lavc57.64.101 libmp3lame
    Stream mapping:
      Stream #1:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
      Stream #0:0 -> #0:1 (pcm_s16le (native) -> mp3 (libmp3lame))
    Press [q] to stop, [?] for help
    frame=   69 fps= 27 q=8.0 size=      45kB time=00:00:02.820 bitrate= 138.6kbits/s dup=0 drop=256 speed= 1.1x
    frame=   79 fps= 17 q=2.0 size=      79kB time=00:00:04.80 bitrate= 134.6kbits/s dup=0 drop=308 speed=1.04x
    frame=   91 fps= 13 q=8.0 size=     112kB time=00:00:06.80 bitrate= 134.8kbits/s dup=0 drop=348 speed=1.04x
    frame=  101 fps= 11 q=8.0 size=     153kB time=00:00:09.22 bitrate= 135.0kbits/s dup=0 drop=388 speed=1.03x
    frame=  112 fps= 10 q=3.0 size=     186kB time=00:00:11.40 bitrate= 133.8kbits/s dup=0 drop=440 speed=1.02x
    av_interleaved_write_frame(): Broken pipe time=05:28:03.40 bitrate= 134.2kbits/s dup=0 drop=393880 speed=   1x
        Last message repeated 1 times
    Error writing trailer of rtmp:// Broken pipeframe=98474 fps=5.0 q=-1.0 Lsize=  322492kB time=05:28:14.00 bitrate= 134.1kbits/s dup=0 drop=393888 speed=0.998x
    video:2213kB audio:306620kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 4.422897%
    [libx264 @ 0x125c850] frame I:5471  Avg QP: 0.00  size:    80
    [libx264 @ 0x125c850] frame P:27354 Avg QP: 0.00  size:    25
    [libx264 @ 0x125c850] frame B:65649 Avg QP: 0.00  size:    17
    [libx264 @ 0x125c850] consecutive B-frames: 11.1%  0.0%  0.0% 88.9%
    [libx264 @ 0x125c850] mb I  I16..4: 100.0%  0.0%  0.0%
    [libx264 @ 0x125c850] mb P  I16..4:  0.0%  0.0%  0.0%  P16..4:  0.0%  0.0%  0.0%  0.0%  0.0%    skip:100.0%
    [libx264 @ 0x125c850] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  0.0%  0.0%  0.0%  direct: 0.0%  skip:100.0%
    [libx264 @ 0x125c850] 8x8 transform intra:0.0%
    [libx264 @ 0x125c850] coded y,uvDC,uvAC intra: 0.0% 0.0% 0.0% inter: 0.0% 0.0% 0.0%
    [libx264 @ 0x125c850] i16 v,h,dc,p: 95%  0%  5%  0%
    [libx264 @ 0x125c850] i8c dc,h,v,p: 100%  0%  0%  0%
    [libx264 @ 0x125c850] Weighted P-Frames: Y:0.0% UV:0.0%
    [libx264 @ 0x125c850] kb/s:0.92
    Conversion failed!

    CPU load average is around 1 or so – much less than before. So I think my ideas are on the right track. Why send 30 frames or whatever each and every second to Youtube just to display a gray screen? The CPU has to work to do that. As long as ffmpeg + Youtube has the intelligence to paste together audio snippets 1/5th second in length five times each second the audio should be taken care of, we’re not playing with the sampling rate or anything – is how I reasoned. Key frames are some sort of overhead as well since they’re extra things ffmpeg has to periodically do. Youtube wants one at least every four seconds. We get really close to that limit by multiplying fps * 3.6 s = 5 * 3.6 = 18 for our group-of-pictures (g) parameter. Previously we were sending a key frame more frequently – every two seconds.

    Running this command is still hit-or-miss. As often as not it hangs, and then, if it does not hang, as often as not it often outputs washboard audio. You just <Ctrl-C> to get out of it if hangs, or type “q” if it is producing washboard audio.

    Note carefully the bandwidth being used, which ffmpeg reports every second. If it is < 128 kbps, you’re hosed and have washboard audio. If it’s about 135 kbps or higher, you’re good. You don’t even need to waste time fiddling with Youtube’s live_dashboard to listen to it. You get this feedback immediately from ffmpeg. And I intend to use these same observed behaviors to script around ffmpeg’s flakiness and keep restarting it automatically until it is producing a good quality audio stream!

    Improved startup
    This script, which I call, has some debugging at the beginning, then loops to ensure there is always an audio stream being live-streamed as long as the Pi has power. It has been extremely reliable. I settled on this one for my own purposes.

    # drJ 5/2019
    sleep 20
    LOG="ff.log"`date +%m-%d-%y:%H:%M`
    # some info for debugging problems
    echo "***********"
    date; ip add; ping -c2; lsusb
    nohup ./</dev/null>$LOG 2>&1 &
    while /bin/true; do
     sleep 7
    # want like
    #Frame=   84 fps= 11 q=16.0 size=      43kB time=00:00:07.50 bitrate=  47.1kbits/s dup=0 drop=431 speed=0.991x
    #Frame=   84 fps= 11 q=16.0 size=      43kB time=00:00:07.50 bitrate=  47.1kbits/s dup=0 drop=431 speed= 1x
     FFOUT=`tail -1 $LOG`
     echo "last line is $FFOUT"
     KB=`echo $FFOUT|awk '{print $(NF-5)" "$(NF-4)}'|sed 's/kbits.*//'|awk '{print $NF}'`
     echo "orig KB: $KB"
     KB=$(echo $KB|sed s/\\..*//)
     echo "KB is: $KB"
     if [ $KB -gt 129 2>/dev/null ]; then
    # stream looks good - do nothing
       echo -n ""
    # didn't work out: restart and try again
      echo "*** Restarting ffmpeg at *** "`date`
      pkill -9 -f 'ffmpeg '
      nohup ./</dev/null>$LOG 2>&1 &

    Note it still calls, which I believe I have provided above.

    ff.log now looks like this:

    Fri 31 May 01:10:59 BST 2019
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
        link/ether b8:27:eb:11:fc:06 brd ff:ff:ff:ff:ff:ff
    3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        link/ether b8:27:eb:44:a9:53 brd ff:ff:ff:ff:ff:ff
        inet brd scope global wlan0
           valid_lft forever preferred_lft forever
        inet6 fe80::1119:b46a:cb69:63c9/64 scope link
           valid_lft forever preferred_lft forever
    PING ( 56(84) bytes of data.
    64 bytes from icmp_seq=1 ttl=56 time=14.6 ms
    64 bytes from icmp_seq=2 ttl=56 time=17.4 ms
    --- ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 1001ms
    rtt min/avg/max/mdev = 14.671/16.065/17.460/1.400 ms
    Bus 001 Device 004: ID 046d:0825 Logitech, Inc. Webcam C270
    Bus 001 Device 005: ID 0424:7800 Standard Microsystems Corp.
    Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
    Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
    Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    last line is frame=   19 fps=0.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A dup=0 drop=69 speed=   0x    ^Mframe=   39 fps= 39 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A dup=0 drop=150 speed=   0x    ^M
    orig KB: dup=0
    Fri 31 May 01:11:07 BST 2019
    KB is: dup=0
    *** Restarting ffmpeg at *** Fri 31 May 01:11:07 BST 2019
    last line is frame=  193 fps= 35 q=8.0 size=     100kB time=00:00:27.60 bitrate=  29.6kbits/s dup=0 drop=764 speed=4.99x    ^Mframe=  195 fps= 32 q=8.0 size=     108kB time=00:00:28.03 bitrate=  31.4kbits/s dup=0 drop=775 speed=4.65x    ^M
    orig KB: 31.4
    Fri 31 May 01:11:36 BST 2019
    KB is: 31
    *** Restarting ffmpeg at *** Fri 31 May 01:11:36 BST 2019

    My crontab now looks like this:

    @reboot /home/pi/ > ff.log 2>&1

    I wanted to record a practice session in my house where no Ethernet port is available (hence I had to get WiFi working, which I believe I have). And I wanted convenience – to not worry about being tethered to the wall by an adapter. So I decided to look for an economical power solution for Raspberry Pi. And I found the ones purpose-built are just too expensive to justify. Pijuice, I’m talking about you. So, really, I realized any old portable USB power stick would work. But I wanted something which could last hours. This Omars 10000 mAh portable USB charger seemed like it would do the trick. $16. And it did. It works great! Two hours later, the LEDs show three bars instead of four, so I think this will supply power for about 8 – 9 hours if I pushed it. And it has the form factor of a smartphone. Ideally I’d want a little on/off switch to avoid plugging/unplugging the power cable, but I didn’t find that as of yet. Maybe there’s a cheap USB cable with that…?

    So now I’m not tethered by Ethernet cables nor by a power plug. See where this is progressing? If I use my smartphone’s hotspot I should be able to livestream anywhere I can get a signal, so, for instance, at band performances. I haven’t tried that yet, but I’m hopeful…

    YouTube quirks
    As previously mentioned (I think)( you need to be enabled for livestreaming. It takes about 24 hours for the approval. I suppose they check to make sure you aren’t a perceived threat.

    Recording NPR will give you a copyright violation flag! This has happened to me more than once. I think because they play snippets of new music which are flagged.

    Lag. I’ve seen lag time as short as four seconds and maybe as long as 20 seconds or so. It is never instantaneous.

    My longest video was 20 hours but the processing took days. In fact I’m not sure it ever completed. So I guess the service falls apart after video lengths of I don’t know, maybe 12 hours or so. So if the desire is to have a continuous security webcam I guess you’ll have to break it into chunks. That’s what I’m thinking about next.

    A livestream gets converted to a video by YouTube. That takes awhile – maybe as long as the video length itself is? It slaps a date and time onto the video which you see in your video manager. Unfortunately, using this ffmpeg streaming method it chooses the Pacific standard time timezone. I actually don’t see a simple way to change that either. It may require use of the API, which is beyond what I’m willing to tackle right now. So for me, being in the Eastern time zone all the timestamps are off by three hours, which is kind of annoying.

    I wondered, does my livestream ID remain constant, or will it change from broadcast to broadcast? This is important for future use of the API. Well, it changes each time I start a new livestream, even though I use a single (my own) account. Each livestream gets a unique ID which then becomes the ID for the DVR of the video which you can view on-demand. And this ID is the part that changes in the URL of an “unpublished” Youtube video. Say your unpublished livestream is
    The part of the URL following the v=, namely, in this example, r1wtZwQ-Tk8, is the ID of that video. I would say YouTube tries to be somewhat robust and will not declare your stream has ended until maybe 30 seconds after you have stopped your program. Or maybe it’s a minute or two, I’m not really sure. But I’ve seen that if you restart the streaming quickly enough you’ll be put onto that same livestream. If on the other hand you wait long enough until you see in live_dashboard that stream ended message then It will assign yuo a new video ID if you start your stream again – and don’t forget to reload the live_dashboard page so it can pick up the new ID.

    Can you pause a livestream, and later resume, keeping the same URL? In a word, No. Unfortunately. Youtube livestreaming is pretty limited in this way. And how useful would that be? I would use my smartphone to control ffmpeg on my Raspberry Pi to pause our band practice during our lengthy chat breaks, keeping the stream focussed on the music. But no… Not possible.

    Logitech webcam quirks
    When you pull both video and audio from your Logitech webcam the usage LED illuminates as you’d expect. However, when you’re pulling just the audio, as I show above, that LED does not illuminate, yet it is being used to record all the sounds in its vicinity. I guess I have accidentally and unintentionally stumbled upon a stealth mode, which is a little disconcerting.

    Yeti USB microphone quirks
    A Yeti mic is extremely sensitive and seems more suited for conversation than music recording in my opinion. Even with the gain all the way down (a must) a loud sound is often distorted. I felt the omni recording mode was the worst in this regard. Stereo recording tolerated sounds better. But, if you want to pikc up every little sound, Yeti is great. More importantly to me, it just worked with the USB settings I used for Logitech. I didn’t have to change a single thing in the way I used ffmpeg.

    Testing if the livestream is still running
    My idea to do this is to use the YouTube API and periodically test if the livestream is still working. I have read that it can go down for various reason, and there is no goo way from within ffmpeg itself to tell that your stream is no longer live! It will make for a good project to test the livestream using the Google Developer’s API. that will be a separate post if I ever get it working. If it’s found to be down, the Pi could restart ffmpeg, in my thinking.

    To do list
    I never really perfected the video. Audio I got pretty well.
    I will borrow my friend’s Yeti USB mic to see how my audio stream works with a high quality microphone. DONE.
    I would like to have a simple external control to turn stream off/ on, whether it is physical or virtual. DONE – see references.
    Scripting to monitor stream and restart it once it fails – to have a recording 24×7 like an audio-only security camera. DONE – as documented above.
    Pause feature. PARTIALLY DONE.

    A Raspberry Pi 3 running Raspbian Stretch Lite is used, along with a Logitech USB webcam, to livestream to YouTube. I showed how to stream video-only with a silent audio track. Then I turned it around and spent most of my time putting a virtual piece of tape over the lens and doing an audio-only livestream. This, after a crap-load of testing and tweaking, eventually began to work in a reliable fashion. Then I showed how to launch the audio-only livestream upon power-up of the Ras Pi.

    Since it is a Raspberry Pi, this whole thing lends itself to portability and interesting use cases. With a $17 portable USB battery source and your own Hotspot, you can stream (audio at least) from anywhere you have 4G cell signal – good for recording a banquet, your band performance, or any other long, live event.

    I spoke about some of the many quirks of YouTube which are relevant to this project.

    References and related
    Where I debug YouTube’s messages:

    Fishcam implemented with Raspberry Pi + webcam + help of my AWS server.

    One of my test videos:

    Check your upload bandwith:

    YouTube’s links have me confused. If you’re trying to produce a Live Stream you’ll want the live dashboard page to watch it and check its quality as Youtube judges it. Here’s that link:

    Microcenter in Paterson, NJ – best to visit in person, or so I have been told.

    My livestream is

    Put virtual tape over your lens by using this tip discussed in Stackoverflow!

    Portable, proven (by me) economical USB power supply for your Raspberry Pi – $16.

    Economical on/off switch for your Raspberry Pi. This is a great way to stop having to pull out/push in power connectors from your micro USB power source. $10 gets you a four-pack!

    Admin DNS Network Technologies Security

    The IT Detective agency: Live hack caught, partially stopped

    In my years at cybersecurity I’ve been sufficiently removed from the action that I’ve rarely been involved in an actual case. Until last night. A friend, whom I’ll call Jute, got a formal complaint about one of his hosted Windows servers.

    We have detected multiple hacking attempts from your ip address (Hilfer Online) to access our systems.
    > Log of attempts:
    > – Hack attempt failed at 2019-01-17T14:22:41.6539784Z. Attempted user name: Not specified (typical for port scanners or denial of service attacks), system accessed: RDP, ip address accessed:
    > – Hack attempt failed at 2019-01-17T14:22:26.2213808Z. Attempted user name: Not specified (typical for port scanners or denial of service attacks), system accessed: RDP, ip address accessed:
    > – Hack attempt failed at 2019-01-17T14:22:10.6304194Z. Attempted user name: Not specified (typical for port scanners or denial of service attacks), system accessed: RDP, ip address accessed:
    > Please investigate this problem.
    > Sent using IP Ban Pro

    Hack, cont.
    I’ve changed the IPs to protect the guilty! But I’ve conveyed the specificity of the error reporting. Nice and detailed.

    Jute has a Windows Server 2012 at that IP. He is not running a web server, so that conveniently and dramatically narrows the hackable footprint of his server. I ran a port scan and found ports 135, 139 and 3389 open. His provider (which is not AWS) offers a simple firewall which I suggested we use to block ports 135 and 139 which are for Microsoft stuff. He was running it as a local sever so I don’t tihnk he needed it.

    Bright idea: use good ole netstat
    The breakthrough came when I showed him the poor man’s packet trace:

    netstat -an

    from a CMD prompt. He ran that and I saw not one but two RDP connections. One we easily identified as his, but the other? It was coming form another IP belonging to the same provider! RDP is easily identified by just looking for the port 3389 connections. Clearly we had caught first-hand an unauthorized user.

    I suggested a firewall rule to allow only his Verizon range to connect to server on port 3389. But, I am an enterprise guy, used to stateful firewalls. When we set it up we cut off his RDP session to his own server! Why? I quickly concluded this was amateur hour and a primitive, ip-chains-like stateless firewall. So we have to think about rules for each packet, not for each tcp connection.

    Once we put in a rule to block access to ports 135 and 139, we also blocked Jute’s own RP session. So the instructions said once you use the firewall, an implicit DENY ALL is added to the bottom of the rules.

    So we needed to add a rule like:


    But his server needs to access web sites. That’s a touch difficult with a stateless firewall. You have to enter the “backwards” rule (outbound traffic is not restricted by firewall):


    But he also needs to send smtp email, and look up DNS! This is getting messy, but we can do it:


    We looked up the users and saw Administrator and another user Update. We did not recognize Update so he deleted it! And changed the password to Administrator.

    Finally we decided we had to bump this hacker.

    So we made two rules to allow him but deny the zombie computer:


    Pyrrhic victory
    Success. We bumped that user right out while permitting Jute’s access to continue. The bad news? A new interloper replaced it!

    OK. So with another rule we can bump that one too.

    Yup. Another success. another interloper jumps on in its place. So we bump that one. But I begin to suspect we are bailing the Titanic with a thimble. It’s amazing. Within seconds a blocked IP is being replaced with a new one.

    We need a more sweeping restriction. So we reasoned that Jute will RP from his provider where his IP does not really change.

    So we replace




    and we also delete the specific reject rules.

    But now at this point for some reason the implicit DENY ALL rule stops working. From my server I could do an nc -v 3389 and see that that port was open, though it should ont have been. So we have to add a cleanup rule at the bottom:


    That did the trick. Port no longer opened.

    There still appears in netstat -an listing the last interloper, but I think it just hasn’t been timed out yet. netstat -an also clearly shows (to me anyway) what they were doing: scanning large swaths of the Internet for other vulnerable servers! The tables were filled with SYN-SENT to port 3389 of consecutive IPs! Amazing.

    So I think Jute’s server was turned into a zombie which was tasked with recruiting new zombies.

    We had finally frozen them out.

    Later that night
    Late that same night he calls me in a panic. He uses a bunch of downstream servers and that wasn’t working! The downstream servers run on a range of ports 14800 – 15200.

    Now bear in mind the provider only permits us 10 firewall rules, so it’s getting kind of tight. But we manage to squeeze in another rule:

    SRC: ANY DST: SRC_PORT: 14800-15200 DST_PORT: ANY ACTION: allow

    He breathes a sigh of relief because this works! But I want him we are opening a slight hole now. Short term there’s nothing we can do. It’s a small exposure: 400 open ports out of 65000 possible. It should hold him for awile with any luck.

    He also tried to apply all updates at my suggestion. I’m still not sure what vulnerability was (is) exploited.

    Case: tentaively closed

    Our first attempt to use the Windows firewall itself was not initially successful. We may return to it.

    We catch a zombie computer totally exploiting RDP on a Windows 2012 server. We knocked it off and it was immediately repaced with another zombie doing the same thing. Their task was to find more zombies to join to the botnet. Inbound firewall rules defined on a stateless firewall were identified which stopped this exploitation while permitting desired traffic. Not so easy when you are limited to 10 firewall rules!

    This is a case where IPBAN did us a favor. The system worked as it was supposed to. We got the alert, and acted on it immediately.

    I’m not 100% sure which RP vulnerability was exploited. It may have been an RCE – remote code execution not even requiring a valid logon.

    References and related
    The rest of the security world finally caught up with this, with Microsoft releasing a critical patch in May. I believe I was one of the first to publicly document this exploit.

    Network Technologies

    Voice and data vlans on one switch port, no vlan tagging: how does that work?

    We had a Cisco video conference unit pick up an IP from a data vlan whereas we expected it to pick it up from a voice vlan, where we had assigned it a static IP. What happened?

    The details
    I have to admit I never paid attention to the switch ports in the offices. All these years and I didn’t really appreciate the fact that you can plug in either a PC or a Cisco phone to the same switch port, yet the PC “knows” to go onto a data vlan while the phone “knows” to put itself onto a voice vlan. How cuold that be?

    Naively, just talking it out, I had this jumble of “facts” in my mind:

    – sharing vlans on one switch port is done through vlan tagging
    – the equipment plugged in must know the switch port is using vlan tagging or else disastrous results occur (see this post for some examples)
    – if in addition you’re a PC using DHCP, how would you know which valn to go onto? How would you learn the connection is tagged?
    – well, there can be a native vlan in addition to tagged vlans. Maybe they used that?

    Fortunately I have some friends with access to the switch config. Here it is for one specific typical port:

    interface FastEthernet0/2
    description Data & Voice vlanC
    switchport access vlan 103
    switchport mode access
    switchport voice vlan 703

    I puzzled over that for awhile because, well, what does it mean?? In my world of servers you have two port types: access ports and truink ports. Trunk ports are the ones that have tagged vlans. Access ports provide a single unttagged vlan’s traffic to the port.

    It’s pretty clearly declaring this switch port to be an access port, not a trunk port. And yet two vlans are referred to. There’s this command I’ve never seen or used before swithcport voice. How does this fit with the jumble of facts above? The jumble of facts need to be amended…

    I asked another expert and he said he heard that the Cisco phones use something called LLDP – link layer discovery porotocol. From researching the predecessor protocol was CDP – Cisco Discovery protcol.

    Switchport voice vlan 703 is something like introducing tagging for vlan703, if I read the Cisco documentation correctly.

    The magic happens
    This is often described as magic or voodoo so we will treat it like that too! A Cisco phone uses LLDP to learn from the switch that the voice vlan is 703. Then somehow it tags(?) its traffic to use only that vlan, even for its DHCP discover. A PC or any other normal host by contrast does not use LLDP and is only exposed to the data vlan 103 (the “native” vlan) so it gets an IP from doing DHCP discover on that vlan.

    Do I believe my own explanation? Not really. It’s the best I got. I really should do a packet trace to confirm but who has the time?

    That video conference unit? They say when they boot it a second time it jumps onto the correct vlan and picks up the desired static IP. Again, no one’s really sure why.

    Strange DHCP behavious on the part of a Cisco video conference unit forces us to think through how data + voice on one switch port might actually be working on a typical Cisco-powered office environment. We probably – definitely – didn’t nail it, but we must be close to the essentially correct answer.

    References and related
    As always Wikipedia has an article somewhat explaining LLDP

    Admin Linux Network Technologies SLES

    Linux tip: how to enable remote syslog on SLES

    I write this knowing I still don’t know anything to speak of about syslog, but, sometimes you gotta act without knowing. I needed to send syslog to somewhere in a big hurry so I figured out the absolute minimum I needed to do to get it running on one of my other systems.

    The details
    This all started because of a deficiency in the F5 ASM. At best it’s do slow when looking through the error log. But in particular there was one error that always timed out when I tried to bring up the details, a severity 5 error, so it looked pretty important. Worse, local logging, even though it is selected, also does not work – the /var/log/asm file exists but contains basically nothing of interest. I suppose there is some super-fancy and complicated MySQL command you could run to view the logs, but that would take a long time to figure out.

    So for me the simplest route was to enable remote syslog on a Linux server and send the ASM logging to it. This seems to be working, by the way.

    The minimal steps
    Again, this was for Suse Enterprise Linux running syslog-ng.

    1. modify /etc/sysconfig/syslog as per the next step
    2. SYSLOGD_PARAMS=”-r”
    3. modify /etc/syslog-ng/syslog-ng.conf as per the next step
    4. uncomment this line: udp(ip(“”) port(514));
    5. launch yast (I use curses-based yast [no X-Windows] which is really cantankerous)
    6. go to Security and Users -> Firewall -> Allowed services -> Internal Zone -> Advanced
    7. add udp port 514 as additional allowed Ports in internal zone and save it
    8. service syslog stop
    9. service syslog start
    10. You should start seeing entries in /var/log/localmessages as in this suitably anonymized example (I added a couple line breaks for clarity:
    Jul 27 14:42:22 f5-drj-mgmt ASM:"7653503868885627313","","/Common/drjohnstechtalk.com_profile","blocked","/drjcrm/bi/tjhmore345","0","Illegal URL,Attack signature detected","200021075","Automated client access ""curl""","US","<?xml version='1.0' encoding='UTF-8'?><BAD_MSG><violation_masks><block>44e7f1ffebff2dfb-8000000000000000</block><alarm>44f7f1ffebff2dfb-8000000000000000</alarm><learn>44e7f1ffe3ff2dfb-8000000000000000</learn><staging>0000000000000000-0000000000000000</staging></violation_masks><request-violations><violation><viol_index>42</viol_index><viol_name>VIOL_ATTACK_SIGNATURE</viol_name><context>request</context><sig_data><sig_id>200021075</sig_id>
    <viol_name>VIOL_URL</viol_name></violation></request-violations></BAD_MSG>","GET /drjcrm/bi/tjhmore345 HTTP/1.1\r\nUser-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2\r\nHost:\r\nAccept: */*\r\n\r\n"

    Interestingly, there is no syslogd on this particular system, and yet the “-r” flag is designed for syslogd – it’s what turns it into a remote syslogging daemon. And yet it works.

    It’s easy enough to log these messages to their own file, I just don’t know how to do it yet because I don’t need to. I learn as I need to. just as I learned enough to publish this tip.

    We have demonstrated activating the simplest possible remote syslogger on Suse Linux Enterprise Server.