Postfix Operational tips

I’m trying out the system-supplied postfix on a SLES system. i had been using sendmail but there doesn’t seem to be any development on that software.

Some commands I needed right away
Well, right away I had thousands of queued messages so I needed a way to make sense of what was happening.

For these commands to make sense you need to know that I am running a second postfix configuraiton out of /etc/postfixEXT.

Display the queue

postqueue -c /etc/postfixEXT -p

Force delivery from the queue

postqueue -c /etc/postfixEXT -f

List one email in detail

postcat -vq -c /etc/postfixEXT QUEUEID

Delete one email

postsuper -c /etc/postfixEXT -d QUEUEID

Put mail on hold

postsuper -c /etc/postfixEXT -h ALL|QUEUEID

Release mail form hold

postsuper -c /etc/postfixEXT -H ALL|QUEUEID

How to force delivery of a single message
This command is not documented anywhere – because it doesn’t exist so you have to get creative. If you have the luxury of halting all email for a few seconds simply do this:

Display the queue to find the queue ID of the email you want to force delivery of

postqueue -c /etc/postfixEXT -p

Put all mail on hold

postsuper -c /etc/postfixEXT -h ALL

Now release the hold on that one email

postsuper -c /etc/postfixEXT -H QUEUEID

QUEUEID is, of course, the queue id .e.g., F2A1A27891E, of the email in question.

Look for what happened
Check your mail log’s last lines in /var/log/mail

Revert back to normal running

postsuper -c /etc/postfixEXT -H ALL

Since mail is store-and-forward and not real time, you can do these steps, quickly, even on a production system and no one will be the wiser if you are pretty quick. Probably takes two minutes even for a slow typer.

How to run multiple listeners
I didn’t want to disturb the system-installed postfix too much. I would let it “have” the loopback address,, leaving me the other IPs for my relay config to listen on. I added these lines to /etc/postfix/

multi_instance_enable = yes
multi_instance_directories = /etc/postfixEXT

service postfix start starts up the local postfix plus my relay. Grep the process table for either master or postfix to see. However, to be honest, service postfix stop does not kill all processes. So I always end up killing one of the master processes by hand. Update: postmulti -p stop does the trick to kill all. There is also a status or start option instead of stop.

References and related
Since I mentioned sendmail I have to give a shout out to one of my old sendmail posts.

More info on postfix multiple instances. A pretty complete giude.

Posted in Admin, Network Technologies | Tagged , | Leave a comment

Live stream to YouTube from a Raspberry Pi + webcam

I’ve been looking at this off and on for awhile now. I finally made a breakthrough this week and started to generate some decent live streams on my Youtube channel, after a lot of misfires.

Note this is applicable for Raspbian stretch Lite on a Raspberry Pi 3. However, I firmly believe it will work just the same for regular Raspbian Stretch.

There’s a lot of wrong, misleading or outdated information out there on the Internet. Hopefully this will help others to avoid wasting as much time as I had to do.

This project was prompted by my desire to make a more generalized fishcam! Described in this post, my original fishcam implementation – and I realized this form the get-go – has very limited applicability because very few people are in a position to have their own AWS server. And if you don’t know what you’re doing, please don’t run your own server – the security exposure is too great.

So I eventually realized that maybe I could generalize what I had done – essentially rmeove the dependency on the AWS server – by utilizing Youtube Live Streaming. And, I believe I was right. It’s still a work in progress however.

The command – ffmpeg
I was playing with ffmpeg. The version I am playing with now comes with Raspbian – no need to compile like in the bad old days. ffmpeg -version shows the version to be 3.2.12. I get the impression that its capabilities are version-dependent, so that’s why this information is particularly relevant in this case.

The details
In some of my early attempts I was getting a lot of this (looking at Youtube Live Dashboard)

Dashboard When stream is not quite right

Another attempt
Video works, audio like driving in a car with the windows down. For the record, the command was this:

ffmpeg \
-f alsa -i plughw:CARD=U0x46d0x825,DEV=0 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 2500k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 2 -qscale 3 \
-b:a 96K \
-r 10 \
-s 1280x720 \
-f flv rtmp://

Video OK, audio choppy message

For the record, the bandwidth required was about 2100 kbps.

List the formats your video device supports

ffmpeg -f video4linux2 -list_formats all -i /dev/video0

Results using my Logitech Webcam

[video4linux2,v4l2 @ 0xcc45c0] Raw       :     yuyv422 :           YUYV 4:2:2 : 640x480 160x120 176x144 320x176 320x240 352x288 432x240 544x288 640x360 752x416 800x448 800x600 864x480 960x544 960x720 1024x576 1184x656 1280x720 1280x960
[video4linux2,v4l2 @ 0xcc45c0] Compressed:       mjpeg :          Motion-JPEG : 640x480 160x120 176x144 320x176 320x240 352x288 432x240 544x288 640x360 752x416 800x448 800x600 864x480 960x544 960x720 1024x576 1184x656 1280x720 1280x960
ffmpeg \
-f alsa -i plughw:CARD=U0x46d0x825,DEV=0 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 1200 \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 2 -qscale 3 \
-b:a 128K \
-r 5 \
-s 640x480 \
-f flv rtmp://

Audio good, video not working

video terrible, but audio good!

It is not so pretty to use that hardware address for the Logitech webcam device. Where do you see that hardware address? Either a lsusb or a ls /dev/snd/by-id shows addresses of sound devices. I found a simpler substitute:

ffmpeg \
-f alsa -i plughw:1,0 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 1200k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 2 -qscale 3 \
-b:a 128k \
-r 5 \
-s 640x480 \
-f flv rtmp://
With this audio's, not too bad, video's a bit choppy. Google reports the stream quality as OK, check resolution.
So I fix the bandwidth (which was a typo in the above, but one with an interesting result). I set video bandwidth to -b:v 1200k. Now the video is OK once again, but the audio is choppy again! Weird. bandwidth is about 1100 kbps.
This version had OK video and OK audio
<pre lang="text'>
ffmpeg \
-f alsa -i plughw:CARD=U0x46d0x825,DEV=0 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 1600k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 2 -qscale 3 \
-b:a 128k \
-r 5 \
-s 640x480 \
-f flv rtmp://

But I keep getting inconsistent results! Sometimes a setting will work, and then I come back to it and it doesn’t. Weird.

Part of the problem is that I have no idea what I’m doing and I didn’t know when i was watching a livestream vs a recorded (on-demand0 one! I have since learned to look for the little red Live button. A picture is worth 10^3 words in this case.

Observed used bandwidth is about 1450 kbits/sec. But still lots of dropped packets. Here is what ffmpeg reports. I’m not sure yet what most of it means:

[alsa @ 0x1502700] ALSA buffer xrun.
[alsa @ 0x1502700] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)
frame= 5828 fps=5.0 q=-1.0 Lsize=  205496kB time=00:19:26.20 bitrate=1443.5kbits/s dup=0 drop=11138 speed=   1x
video:187265kB audio:17449kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.382063%
[libx264 @ 0x15100e0] frame I:583   Avg QP: 9.41  size: 53819
[libx264 @ 0x15100e0] frame P:5245  Avg QP:13.53  size: 30578
[libx264 @ 0x15100e0] mb I  I16..4: 100.0%  0.0%  0.0%
[libx264 @ 0x15100e0] mb P  I16..4: 38.0%  0.0%  0.0%  P16..4: 60.7%  0.0%  0.0%  0.0%  0.0%    skip: 1.4%
[libx264 @ 0x15100e0] coded y,uvDC,uvAC intra: 93.7% 86.2% 82.4% inter: 77.8% 60.5% 34.1%
[libx264 @ 0x15100e0] i16 v,h,dc,p: 17% 23% 15% 45%
[libx264 @ 0x15100e0] i8c dc,h,v,p: 51% 21% 16% 11%
[libx264 @ 0x15100e0] kb/s:1315.22

The video for that run is here:

Suppressing Audio
This is what worked for me.

ffmpeg \
-f lavfi -i anullsrc=channel_layout=stereo:sample_rate=44100 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 10 -b:v 1600k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 2 -qscale 3 \
-b:a 128k \
-r 5 \
-s 640x480 \
-f flv rtmp://

That is working great – showing the video as before but now with a silent audio track.

Increase Video Quality
Here I’ve increased video quality a tad by requesting more fps (10) and making qscale 0 (which means highest quality).

ffmpeg \
-f alsa -i plughw:1,0 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -b 3000k -g 20 -b:v 1800k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 4 -qscale 0 \
-b:a 128k \
-r 10 \
-s 640x480 \
-f flv rtmp://

Bitrate was about 1700 kbps. Quality is maybe a little better. Audio still leaves something to be desired.

Still better video quality

ffmpeg \
-f alsa -i plughw:1,0 \
-f v4l2 -i /dev/video0 \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -b 3000k -g 60 -b:v 2000k \
-bufsize 512k \
-acodec libmp3lame -ar 44100 \
-threads 4 -qscale 0 \
-b:a 128k \
-r 30 \
-s 640x480 \
-f flv rtmp://

What is observed to happen is that ffmpeg actually chooses 15 fps rather than 30. I’ve read it decides what it is able to do, so maybe that’s the highest fps it can deliver. Video is pretty smooth (See my Livestream link in references if I happen to have it running. Otherwise I will create a video link.) No drops are recorded, but the sound, though not terrible, has some pops. Bandwidth used is about 1900 kbps. So this is definitely my best effort yet. Youtube complains about the unsupported video size of 640×480, but it permits it and I don’t think it’s a real problem.

Bandwidth talk
It’s importasnt to talk about bandwidth if you haven’t given this any real thought. You have to have a halfway decent broadband connection for this to work, you see? If yuo have a mid-speed cable modem or DSL, yuo have much lower upload than download speeds, and yuo may not be able to pull off a reliable 1.5 mbps upload. For those lucky enough to have Verizon FIOS this is a non-issue. But for instance in the highschool where I volunteer they have throttled the guest wifi network to such an extent that achieving this modest 1.5 mbps is going to present a real challenge. If yuo rely on a phone’s hotspot you will also probably be unable to get such a speed. So I may look at more ways to reduce the bandwidth required in the future.

Check your bandwidth using

And between YouTube and your ISP, it just seems the whole thing about live video broadcasting seems, well, delicate. Stream Health varies between oK, to Excellent to not receiving – all during the same streaming session! It often takes five minutes or so for the stream to appear to be working.

Comparing two webcams
Someone picked up a really cheap DI Chatcam at Microcenter in Paterson. I think that’s Digital Innovations Chatcam. It’s cute. It has a big clip on the end and shines white LEDs when it’s on. I think it was about $12. With the exact same ffmpeg settings (with audio suppressed), the quality was not nearly as good as with the Logitech webcam. Here’s a link to the YouTube video made with the chatcam: Note that it has a ministereo plug for audio. I didn;’t even plug it in now that I know how to suppress audio!

The Logitech model is a C525. It was a refurbished model which cost me about $27.The comparable Logitech webcam test is here:

I need to re-run this test now that I know how to increase the video quality.

Testing if the livestream is still running
My idea to do this is to use the Youtube API and periodically test if the livestream is still working. I have read that it can go down for various reason, and there is no goo way from within ffmpeg itself to tell that your stream is no longer live! It will make for a good project to test the livestream using the Google Developer’s API. that will be a separate post if I ever get it working. If it’s found to be down, the Pi could restart ffmpeg, in my thinking.

References and related
Where I debug YouTube’s messages:

Fishcam implemented with Raspberry Pi + webcam + help of my AWS server.

One of my test videos:

Check your upload bandwith:

YouTube’s links have me confused. If you’re trying to produce a Live Stream you’ll want the live dashboard page to watch it and check its quality as Youtube judges it. Here’s that link:

Microcenter in Paterson, NJ – best to visit in person, or so I have been told.

My livestream is

Posted in Linux, Network Technologies, Raspberry Pi | Tagged , , , | Leave a comment

How to add private root CAs in SLES or Redhat

From time-to-time I run my own PKI infrastructure, namely issuing my own certificates form my private root CA. I wanted this root CA to be recognized by Linux utilities running on Suse Linux (SLES), in particular, lftp, which I was trying to use to access an ftps site, which itself is a post for another day.

The details
Let’s say you have your root certificate in the standard form like this example


Then you can put the certificate inline and within one script install it so that it permanently joins the other root CAs in /etc/ssl/certs with a script like this example:

3\n-----END CERTIFICATE-----\n"
cd /etc/pki/trust/anchors/
echo -e -n $DrJ_Root_CA > DrJ_Root_CA.pem

So the key commands are c_rehash and update-ca-certificates.

Usually SLES is similar to Redhat. But it seems to be different in this case.

This was tested on a SLES 12 SP3 system.

It copies the certificate to /etc/pki/trust/anchors, which by itself is insufficient. Then it creates some kind of hash symlink to the CA file and makes sure that this new certificate doesn’t get wiped out by subsequent system patching. That’s the purpose of the c_rehash and update-ca-certificates commands.

You may also see these hashes and certificates in /etc/ssl/certs. I’m not sure because that’s where I started with all this. But merely dropping the private root CA into /etc/ssl/certs is insufficient, I can say from experience!

Redhat is better documented, but for completeness I include it here. You have your inline certificate as in the SLES script, then following that:

cd /etc/pki/ca-trust/source/anchors/
echo -e -n $DrJ_Root_CA > DrJ_Root_CA.pem

So update-ca-trust is the key command for Redhat Linux.

We show how to add your own root CA to a SLES 12 system. I did not find a good reference for this informaiton anywhere on the Internet.

References and related
My favorite openssl commands.

The basics of working with cipher settings

For Reedhat/CentOS I am evaluating this blog post on the proper way to add your own private CA:

For the Redhat approach I used this blog post:

Posted in Admin, Linux, SLES | Tagged , , | Leave a comment

Fishcam using Raspberry Pi and some network tricks

There are more articles about running a webcam using Raspberry Pi than Carter has pills. Why bother to create another? This one is unique insofar as I created a fishcam at a school with a restricted network. None of the reference articles I found discussed a way to get your stream onto the Internet except the simplistic approach only available to homeowners of setting up a rule on a home router. Pimylifeup’s article is typical of that genre.

Cooperating third party
To push this webcam out to the Internet when I had no way to allow inbound traffic to the Pi, I realized that I needed a cooperating third party. I looked briefly for a commercial service specializing in this. I did not find one. I suppose there is, but I don’t know. It was actually quicker to stop the search and use my own AWS server as the cooperating third party.

With a cooperating third party what you can do is set up a forwarder from the Pi to cooperating server on the Internet. So that’s what I did. More on that below.

Network restrictions
The Pi was given WiFi access to a school’s bring your own device (BYOD) WiFi. By trial and error (I did not initiate extensive port scans, etc so as to avoid acting like a hacker). I’m familiar with running a almost completely open Guest wireless. This BYOD was not that for some reason unknown to me. One of the first things I tried, to ssh to my server, was not going through. So I knew there were restrictions. Also PING did not work. So ICMP was blocked as well. But web browsing worked, and so did DNS queries. So TCP ports 80 and 443 were allowed, as well as UDP port 53 and possibly TCP port 53. I also observed there was no proxy server involved in the communication. So I simply tested a few other ports that I know are used from time-to-time: 2443 and 8443. If you a hit a server that is not protected by a firewall and not listening on a port that you are testing you will get a Connection reset if your packets are not blocked by a local firewall. I tested with the nc utility. nc -v <my_server> <port> I found a couple open ports this way. Next question: does the network care what protocol is running on that port? They might be looking for https and I was planning to run ssh. For a simple port blocker it might not distinguish what’s going on. That was indeed the case as I was able to run ssh on this non-standard port.

The single most comlicated thing was formulating the appropriate ssh command. I created a dedicated account on my server for this purpose. I embedded the password into the startup script I created using a utility called sshpass. This is not super secure but I wanted something running quickly.

Here’s that complicated command

sshpass -p <PASSWORD> ssh ‐f ‐N ‐R 8443:localhost:8081 ‐p 2443 <USERNAME>@<SERVER_IP>

That’s a mouthful! Let’s break it down. sshpass just permits you to run the command and not get a login prompt. It needs to be installed with a sudo apt-get sshpass.

The ssh command sets up a reverse tunnel. I have discussed it in my Access your Raspberry Pi from anywhere blog post, however, some things are different and more complicated here. Here we are setting up port 8443 on my server as the tunnel port which will be accessible to the Internet. It is terminated on the Raspberry Pi’s local port 8081 (the port that the motion package uses for the webcam). We had to use ssh to connect to port 2443 on my server because the school network blocked the standard port 22. Then <PASSWORD>, <USERNAME> and <SERVER_IP> are to be replaced with values specific for my server. I don’t want to publish them.

How I got my server to run ssh on port 2443 as well as port 22
This turned out to be one of the easiest things. It’s good to run your own server… In the file /ets/ssh/sshd_config the listening port was commented out, letting the defaul 22 be in effect. So I uncommented that and added port 2443 like this:

# Listen on multiple ports - DrJ 2/1/19
Port 22
Port 2443

Then a sudo service sshd restart and the server listens on both ports for ssh connections!

About the webcam itself
I just followed the Pimylife article as I mentioned. It talks about using the motion package which I’ve never used before. Now in my other posts you’ll see I do stuff with video on Raspberry Pi. In those we had to fight to get the lag time down and keep bandwidth low. I have to say by comparison this motion package is awful. Lag is a couple seconds. There is no sense whatsoever of true video. Just image, wait, next image, wait. No matter the fps setting. I did not have time to switch to a video package that works better. Anyway motion may provide some other advantages we could eventually use. So I just set it to 2 fps (frames per second) since it doesn’t really matter.

The fishcam is at fishcam. It’s not working right now – just showing black. I’m not sure why.

Auto starting
I’ve documented elsewhere the poor man’s way to start something upon initially booting the Pi: stuff the appropriate command into the crontab file.

So you edit your crontab file with a crontab -e. Then you enter

@reboot sleep 20; sshpass -p <PASSWORD> ssh ‐f ‐N ‐R 8443:localhost:8081 ‐p 2443 <USERNAME>@<SERVER_IP>

That just sleeps for 20 seconds as your Pi boots up, then establishes the reverse tunnel with that complicated command I explained earlier.

Usually thes tutorials start with an equipment list. For me that is the least interesting part. I used a Raspberry Pi 3 running Raspbian. For a camera I used one of my spare USB ELP cameras from my extensive work with USB cameras. While developing the solution I needed a keyboard, mouse and HDMI monitor. Once running, the only thing connected to the Pi is the USB camera and the micro USB power supply.

To be continued…

References and related
A very good guide for your typcial webcam-using-a-Raspberry-Pi situation, i.e., not what I am addressing in my article.

Access your Raspberry Pi from anywhere blog post

Run multiple USB cameras on your Raspberry Pi while keeping lag minimal.

For supplies we love visits to The Micro Center in Paterson, NJ. This past weekend we got Raspberry Pi 3’s for only $29. And the sales tax is only 3% and change.

Posted in Admin, Linux, Raspberry Pi | Tagged | Leave a comment

The IT Detective agency: Live hack caught, partially stopped

In my years at cybersecurity i’ve been sufficiently removed from the action that I’ve rarely been involved in an actual case. Until last night. A friend, whom I’ll call Jute, got a formal complaint about one of his hosted Windows servers.

We have detected multiple hacking attempts from your ip address (Hilfer Online) to access our systems.
> Log of attempts:
> – Hack attempt failed at 2019-01-17T14:22:41.6539784Z. Attempted user name: Not specified (typical for port scanners or denial of service attacks), system accessed: RDP, ip address accessed:
> – Hack attempt failed at 2019-01-17T14:22:26.2213808Z. Attempted user name: Not specified (typical for port scanners or denial of service attacks), system accessed: RDP, ip address accessed:
> – Hack attempt failed at 2019-01-17T14:22:10.6304194Z. Attempted user name: Not specified (typical for port scanners or denial of service attacks), system accessed: RDP, ip address accessed:
> Please investigate this problem.
> Sent using IP Ban Pro

Hack, cont.
I’ve changed the IPs to protect the guilty! But I’ve conveyed the specificity of the error reporting. Nice and detailed.

Jute has a Windows Server 2012 at that IP. He is not running a web server, so that conveniently and dramatically narrows the hackable footprint of his server. I ran a port scan and found ports 135, 139 and 3389 open. His provider (which is not AWS) offers a simple firewall which I suggested we use to block ports 135 and 139 which are for Microsoft stuff. He was running it as a local sever so I don’t tihnk he needed it.

Bright idea: use good ole netstat
The breakthrough came when I showed him the poor man’s packet trace:

netstat -an

from a CMD prompt. He ran that and I saw not one but two RDP connections. One we easily identified as his, but the other? It was coming form another IP belonging to the same provider! RDP is easily identified by just looking for the port 3389 connections. Clearly we had caught first-hand an unauthorized user.

I suggested a firewall rule to allow only his Verizon range to connect to server on port 3389. But, I am an enterprise guy, used to stateful firewalls. When we set it up we cut off his RDP session to his own server! Why? I quickly concluded this was amateur hour and a primitive, ip-chains-like stateless firewall. So we have to think about rules for each packet, not for each tcp connection.

Once we put in a rule to block access to ports 135 and 139, we also blocked Jute’s own RP session. So the instructions said once you use the firewall, an implicit DENY ALL is added to the bottom of the rules.

So we needed to add a rule like:


But his server needs to access web sites. That’s a touch difficult with a stateless firewall. You have to enter the “backwards” rule (outbound traffic is not restricted by firewall):


But he also needs to send smtp email, and look up DNS! This is getting messy, but we can do it:


We looked up the users and saw Administrator and another user Update. We did not recognize Update so he deleted it! And changed the password to Administrator.

Finally we decided we had to bump this hacker.

So we made two rules to allow him but deny the zombie computer:


Pyrrhic victory
Success. We bumped that user right out while permitting Jute’s access to continue. The bad news? A new interloper replaced it!

OK. So with another rule we can bump that one too.

Yup. Another success. another interloper jumps on in its place. So we bump that one. But I begin to suspect we are bailing the Titanic with a thimble. It’s amazing. Within seconds a blocked IP is being replaced with a new one.

We need a more sweeping restriction. So we reasoned that Jute will RP from his provider where his IP does not really change.

So we replace




and we also delete the specific reject rules.

But now at this point for some reason the implicit DENY ALL rule stops working. From my server I could do an nc -v 3389 and see that that port was open, though it should ont have been. So we have to add a cleanup rule at the bottom:


That did the trick. Port no longer opened.

There still appears in netstat -an listing the last interloper, but I think it just hasn’t been timed out yet. netstat -an also clearly shows (to me anyway) what they were doing: scanning large swaths of the Internet for other vulnerable servers! The tables were filled with SYN-SENT to port 3389 of consecutive IPs! Amazing.

So I think Jute’s server was turned into a zombie which was tasked with recruiting new zombies.

We had finally frozen them out.

Later that night
Late that same night he calls me in a panic. He uses a bunch of downstream servers and that wasn’t working! The downstream servers run on a range of ports 14800 – 15200.

Now bear in mind the provider only permits us 10 firewall rules, so it’s getting kind of tight. But we manage to squeeze in another rule:


He breathes a sigh of relief because this works! But I want him we are opening a slight hole now. Short term there’s nothing we can do. It’s a small exposure: 400 open ports out of 65000 possible. It should hold him for awile with any luck.

He also tried to apply all updates at my suggestion. I’m still not sure what vulnerability was (is) exploited.

Case: tentaively closed

Our first attempt to use the Windows firewall itself was not initially successful. We may return to it.

We catch a zombie computer totally exploiting RDP on a Windows 2012 server. We knocked it off and it was immediately repaced with another zombie doing the same thing. Their task was to find more zombies to join to the botnet. Inbound firewall rules defined on a stateless firewall were identified which stopped this exploitation while permitting desired traffic. Not so easy when you are limited to 10 firewall rules!

This is a case where IPBAN did us a favor. The system worked as it was supposed to.We got the alert, and acted on it immediately.

I’m not 100% sure which RP vulnerability was epxloited. It may have been an RCE – remote code execution not even requiring a valid logon.

Posted in Admin, DNS, Network Technologies, Security | Tagged | Leave a comment

Evaluation of WPI’s multiple camera coprocessor using Raspberry Pi

There’s some good and some not-so-good about the new WPI-provided way to handle multiple video streams using a Raspebrry Pi.

ELP Cameras problems
I have bought many of these ELP cameras last year and this. I may be a slow learner, but eventually it dawned on me that the problems I noticed seem to occur because of this model of USB camera. Finally this year we got a chance to explore this further. I got my hands on a Logitech webcam, the kind you use perched on top of a display monitor. We had this set up as a second camera while an ELP camera was the first. Then we rebooted the Pi a whole bunch of times to gather some stats. About 25% of the time there were problems with the ELP over about 10 tries. There were no problems with the Logitech. Here are various problems we’e seen:
– horizontal lines superimposed over image, and image dull
– ghosting, a corner of the field of view is shown in the center of the image
– sometimes the stream never starts and we’re not yet sure if that’s a camera problem or a software problem though I begin to suspect it’s an ELP problem
– one of my pinhole ELP cameras died

So: Logitech webcam is decidedly better.

Power problem
We pay extra attention to the power draw of the Pi. With two cameras attached and a 2 amp or 1.8 amp power supply the red LED power flashes. That is not good. It’s a sign of undervoltage. The command

vcgencmd get_throttled

on your Pi will tell you if there was an undervoltage condition. I see


when using a 2 amp power supply. Note that as far as we can see the camera display itself works just as well. We also have a 3 amp power supply. That produces a solid red led light and vcgencmd get_throttled produces a response of


which probably indicates there were no undervoltage conditions.

The problem we need to avoid for the Pi to attempt to draw more than 2 amps from the regulator. Doing so may shut it down. So we will try to use the Pi along with a powered USB hub.

Bandwidth constraints
We want to be well below 3 mbps for two cameras. How to get there while still providing a useful service. Initially we felt we could run the cameras at 320×240 resolution, 10 fps. But after much playing we found conditions under which we exceed 3 mbps though normally we were below that. I believe that the compressibility of the image is what matters. So a “rich” visudal field with lots of contrasting objects is the least compressible. That vaguely fits our findings as well. So we felt it important to prepare for the worst case. So we actually looked at supported resolutions and settled on 176×144 pixels! It sure isn’t much, agreed, but it’s still helpful. We blow up the images during the display. We use YUYV mode. MJPEG uses considerably more bandwidth.

Refresh trick
With this WPI software, the video streams never display the first time. You have to refresh the page for some reason. We wished to have a one click operation for viewing, however, to minimize the risk of operator error. So we used some old-fashioned META HTML tags to force a page refresh.

Our initial approach was to simply have the web page refresh itself every five seconds. This worked, but caused instability in the video stream itself and given a few minutes, would always crash the video stream. So we came up with an alternative that does a single page refresh. Unfortunately we’re not that conversant in Javascript (I’m sure there’s a way to do this with Javascript) so instead we wrote two HTML pages: a source page with the refresh, and a target page that does not refresh.

Initial page HTML source

<meta http-equiv="Refresh" content="1;url=file:///C:/users/aperture/Desktop/2019-no-refresh.htm">
<img src="" width=560 height=400>
<img src="" width=560 height=400>

And we size the browser window to just fit the two video streams side-by-side.

Target HTML source for 2019-no-refresh.htm

<img src="" width=560 height=400>
<img src="" width=560 height=400>

Timing and sequence of events
After the Ras Pi is powered up, we launch the initial page from the task bar where it was pinned, 20 seconds later.

It takes a bit of time, then it displays the side-by-side video streams as broken images.

The red LEDs on the Logitech webcams begin to glow.

(We know when we see both red LEDs glowing we are good to go by the way).

the refresh occurs automatically to the 2019-no-refresh.htm web page.

Two side-by-side video streams are displayed, each with 560×400 display dimensions.

References and related
My 2018 version of using the Raspebrry Pi to handle two USB cameras: USB webcam on Raspberry Pi

Field Management System spec for 2019
WPI PDF manual, Using a Ras Pi as Video coprocessor

Download compressed image from Github: Scroll down to Assets and look for (2019.3.1 is the latest at time of this writing.

Logitech webcam:

FIRST FRC Networking Basics

Posted in Linux, Raspberry Pi | Leave a comment

Google Hangouts Meet – what do these IPs all have in common?

They all have been used by Google’s Hangouts Meet based on my observation.
If you have an environment which uses proxy authentication, the above IPs do not play well with that. So you’ll need to disable proxy authentication for them for Hangouts Meet to work.

Otherwise you can do the initial connect but wioll be dropped after about 45 seconds.

Finally, although you can look up each individually and learn of its association to Google, Google’s own documentation is devoid of any references to them. That is unfortunate.

So the IPs in actual use is probably much larger, but these are what I’ve observed over the course of a few days of testing.

What I’ve done
Google is very hard to reach. They only provide indirect means for regular users. So not knowing any insiders I submitted feedback at which is what they suggest. I provided a detailed description. I doubt they will do anything with my feedback. We’ll see.

Google has an undocumented dependency on a whole set of IPs for hangout Meets to actually work through a proxy which requires authentication. Contacting Google for more information is probably impossible, but I will try.

Posted in Admin, Proxy | Tagged | Leave a comment

Consumer Tech: amazing little Bluetooth adapter for your car

My old car I decided finally needed a touch of Bluetooth connectivity. But how to do it without spending a fortune?

The details
I found this amazingly inexpensive gadget on Amazon that describes itself as follows: Handsfree Call Car Charger,Wireless Bluetooth FM Transmitter Radio Receiver&Mp3 Music Stereo Adapter,Dual USB Port Charger Compatible for All Smartphones,Samsung Galaxy,LG,HTC,etc.

I guess it’s normally $16 but I bought on Cyber Monday so it was about $12. 12 dollars! I think if I had gone to the dealer for an after-market solution it would have been $500.

The bad first
Let’s get the bad out of the way, bearing in mind my expectations were rather low so I may be leaving out some obvious “of course it doesn’t have that…” type of stuff.

This device does an FM broadcast and you pick an unoccupied FM frequency to listen to it. On the other side it connects to a Bluetooth device such as a phone. In IT terms I’d call it a gateway since it converts one protocl to another (Bluetooth to FM). Having read the reviews, but not finding anything addressing my interests, my idea was to continue to use my FM stations, and put the station used by this device on a preset so when a call comes in to my phone I hit the station preset and accept the call on the device.
Well, you basically can’t listen to other FM stations as long as this thing is powered up. So even tuned to 88.7 at the far end of the dial, it interferes with stations up and down the dial for some reason. It’ not impossible but you’d have to have a high tolerance for static to use it that way.

Also, I’ve read that these little things break on some people after a few months. But I would no feel cheated, remember, low expectations?

The good
– microphone is good
– compatible with Samsung
– capable of audio program controls
– boots up quickly, maybe 10 seconds

My solution
I only really listen to NPR. I tried the WNYC app in the Playstore. I used to think it was buggy – stopping at random times. But I’ve had good luck my first few trips. Plays great through this thing! In many ways it’s better than FM because I don’t have to change NPR stations as I drive to work, etc, and its signal is often better. There is some static background however. Now my phone GPS talks to me through the speakers (it used to be silent in the car).

The unknown
Not sure about outbound calls to much. If you hit the button twice it seems to call the last dialed number. There are these settings U01, U02, U30. maybe those are for speed dialing?
Precisely why it has to so broadly interfere with all FM stations is a mystery.

Being a thoughtful person, I also wonder what it does to the FM reception of cars around me, ha ha? Could they pick up my call? Now that I think of it, I do remember hearing someone’s phone call on my FM radio once, many years ago. Use of an aux cord would be a way around that, but this little device does not support that.

The amazing
How cuold they pack all this tech, make something that basically works as it’s supposed to, and sell it for $12? I’m in awe.

References and related

Posted in Consumer Tech | Tagged | Leave a comment

Great serial port concentrator: Raritan Dominion

Every now and then you find a product that is a leap ahead of where you were. Such is the case for us with regards to our product of choice for serial consoles.

The old
For Bluecoat (Symantec) proxy and AV systems, we had been using an ancient Avocent CPS device. It permitted ssh connection. It was slow and the menu very limited. But it did permit us to connect multiple serial consoles to one concentrator device at least.
For low-end firewalls we had been using DigiConnects, one per firewall. They are small, which may be thir one advantage. They are tricky to initially set up. Then they are slow to use.

In with the new
We heard about the Raritan Dominion line of products, stranegly enough, from some IT guys in Europe. It’s strange because they are right here in New Jersey – the company name probably comes form the Raritan river. But our usual reseller never heard of them. The specific device is a Dominion SXII.

It’s so much better than those older products. You can use their GUI to connect. This is a vendor who got their act together and eliminated Java. So many other security vendors have yet to do that, incredible as it is to say that.
It tries to autosense the wiring of the serial connector. That doesn’t always work, but it’s very easy for you to hardwire a port as DCE, or if that dosn’t work, try DTE. I use one type for my Symantec devices, another for firewalls.

Labelling the port with meaningful names is a snap, of course.

The Digis would interfere with the reboot process of the firewall so we’d have to detach them if we were going to rbeoot the firewall. These do not. So much better…

You can combine them with power control but we aren’t going to do that.

Don’t want to use the GUI? No problem, console access through ssh is also possible. Of configure dedicated ports that you ssh to for individual consoles.

Sending signals and cleanly disconnecting is easy with their menuss. Connecting to multiple consoles is alsono problem.

They have something called in-the-rack access. I know this will be useful but I haven’t figured out how to use it yet. But if it is what it sounds like it is, it will allow me to be in the server room and access any console by using a direct connection of some sort to the Dominion SXII.

And they’re just plain faster. A lot faster.

And, considering, they’re not so expensive.

They worked so much better than expected that we pretty much immediately filled up the ports with firewalls and other stuff.

A leap forward in productivity was realized by utilizing Raritn’s Dominion SXII serial port concentrator. Commissioning new security gear has never been esaier…

References and related
Raritan’s web site:

Posted in Admin, Security | Tagged , , , | Leave a comment

Counting active leases on an old ISC DHCP server

Checkpoint Gaia offers a DHCP service, but it ias based on a crude and old dhcp daemon implementation frmo ISC. Doesn’t give you much. Mostly just the file /var/lib/dhcpd/dhcpd.leases, which it constantly updates. A typical dhcp client entry looks like this:

lease {
  starts 5 2018/11/16 22:32:59;
  ends 6 2018/11/17 06:32:59;
  binding state active;
  next binding state free;
  hardware ethernet 30:d9:d9:20:ca:4f;
  uid "\0010\331\331 \312O";
  client-hostname "KeNoiPhone";

The details

So I modified a perl script to take all those lines and make sense of them.
I called it
Here it is

# from - DrJ 11/15/18
my $VERSION=0.03;
##my $leases_file = "/var/lib/dhcpd/dhcpd.leases";
my $leases_file = "/tmp/dhcpd.leases";
##use strict;
use Date::Parse;
my $now = time;
##print $now;
# 12:22 PM 11/15/18 EST
#my $now = "1542302555";
my %seen;       # leases file has dupes (because logging failover stuff?). This hash will get rid of them.
open(L, $leases_file) or die "Cant open $leases_file : $!\n";
undef $/;
my @records = split /^lease\s+([\d\.]+)\s*\{/m, <L>;
shift @records; # remove stuff before first "lease" block
## process 2 array elements at a time: ip and data
foreach my $i (0 .. $#records) {
    next if $i % 2;
    ($ip, $_) = @records[$i, $i+1];
    ($ip, $_) = @records[$i, $i+1];
    s/^\n+//;     # && warn "leading spaces removed\n";
    s/[\s\}]+$//; # && warn "trailing junk removed\n";
    my ($s) = /^\s* starts \s+ \d+ \s+ (.*?);/xm;
    my ($e) = /^\s* ends   \s+ \d+ \s+ (.*?);/xm;
    ##my $start = str2time($s);
    ##my $end   = str2time($e);
    my $start = str2time($s,UTC);
    my $end   = str2time($e,UTC);
    my %h; # to hold values we want
    foreach my $rx ('binding', 'hardware', 'client-hostname') {
        my ($val) = /^\s*$rx.*?(\S+);/sm;
        $h{$rx} = $val;
    my $formatted_output;
    if ($end && $end < $now) {
        $formatted_output =
            sprintf "%-15s : %-26s "              . "%19s "         . "%9s "     . "%24s    "              . "%24s\n",
                    $ip,     $h{'client-hostname'}, ""              , $h{binding}, "expired"               , scalar(localti
me $end);
    else {
        $formatted_output =
            sprintf "%-15s : %-26s "              . "%19s "         . "%9s "     . "%24s -- "              . "%24s\n",
                    $ip,     $h{'client-hostname'}, "($h{hardware})", $h{binding}, scalar(localtime $start), scalar(localti
me $end);
    next if $seen{$formatted_output};
    print $formatted_output;

Even that script produces a thicket of confusing information. So then I further process it. I call this script

# DrJ 11/15/18
# bring over current dhcp lease file from firewall FW-1
echo fetching lease file dhcpd.leases
scp admin@FW-1:/var/lib/dhcpd/dhcpd.leases /tmp
# analyze it. this should show us active leases
echo analyze dhcpd.leases
DIR=`dirname $0`
$DIR/|grep active|grep -v expired > /tmp/intermed-results
# intermed-results looks like:
#   : "android-7fe22a415ce21c55" (50:92:b9:b8:92:a0)    active Thu Nov 15 11:32:13 2018 -- Thu Nov 15 15:32:13 2018
#   : "android-283a4cb47edf3b8c" (98:39:8e:a6:4f:15)    active Thu Nov 15 11:37:23 2018 -- Thu Nov 15 15:32:14 2018
#   : "other-Phone"            (38:25:6b:79:31:60)    active Thu Nov 15 11:32:24 2018 -- Thu Nov 15 15:32:24 2018
#   : "iPhone-de-Lucia"          (34:08:bc:51:0b:ae)    active Thu Nov 15 07:32:26 2018 -- Thu Nov 15 15:32:26 2018
# further processing. remove the many duplicate lines
echo count active leases
awk '{print $1}' /tmp/intermed-results|sort -u|wc -l > /tmp/dhcp-active-count
echo count is `cat /tmp/dhcp-active-count`

And that script gives my what I believe is an accurate count of the active leases. I run it every 10 minutes from SiteScope and voila, we have a way to make sure we’re coming close to running out of IP addresses.

Posted in Admin, Perl | Tagged | Leave a comment