Categories
Admin Linux Network Technologies

Querying AD via LDAP – reference documentation

Intro
Suppose you managed to stuff the user’s ID into the description field of every computer object. Then independently the name of the computer object appears in a log such as a web server log and you want to know the user associated with that.

These examples show how to get at that description field from the computer object name.

The details

ldapsearch is a useful tool. I have two versions of it installed on Windows 7 and different Unix/Linux versions. The syntax is slightly different in all cases. Let’s assume the AD domain DRJOHNSAD is mapped to DNS domain drjohnsad.drjohns.net, and the user is drj. Then we have:

Linux
> ldapsearch -h drjohnsad.drjohns.net -b dc=drjohnsad,dc=drjohns,dc=net -D ‘drjohnsad\drj’ -W cn=computerName description

The -W switch prompts for the password. That is a nice switch, and not available in all versions of ldapsearch. If not, use -w password instead. drjohnsad\drj needs the single quote to prevent the “\” character from being treated as a special character by the shell. Windows doesn’t need that.

Windows 7 CMD Window

Oracle-provided ldapsearch

> ldapsearch -h drjohnsad.drjohns.net -b dc=drjohnsad,dc=drjohns,dc=net -D drjohnsad\drj -q cn=computerName description

So -q is used to prompt for a password instead of Linux’s -W.

Lotus Notes ldapsearch

> ldapsearch -h drjohnsad.drjohns.net -b dc=drjohnsad,dc=drjohns,dc=net -D drjohnsad\drj -w password cn=computerName description

You gotta put in the password on the command line.

Of course Windows also has applications which can be used for ldap queries in a GUI, but I don’t use them.

Conclusion
The syntax for a simple ldap query against an AD domain controller is shown.

Categories
Admin Internet Mail Linux

Sendmail: getting mailertable, smarttable and virtusertable to play nice together

Intro
Remember when I posted about some obscure but cool sendmail features? Well in that posting I mentioned having trouble with a catch-all mailertable entry co-existing with a smarttable. I recently got an incentive to work through that and I am sharing my results here. Plus use of virtusertable thrown in for good measure.

The setup
I have a Secure Mail Gateway (SMGW). Some of my users use it, most do not. So for the few who do I wish to divert my outgoing mail, if it sent by one of them, to the SMGW1. Everything else that’s outbound gets forwarded to an Internet-facing relay2. And for inbound mail I also want to divert their email over to it so it can do S/MIME or PGP decryption3. All other inbound emails should go to my native email system4.

So consider the outbound stream, requirement 1. Sender-based routing. That’s the normal thing in sendmail. That’s why you use smarttable, which Andrzej Filip has developed. See my references for more details on that. But my mailertable has a catch-all entry like:

.              relay:emgw.drj.com

I found if I got rid of this entry my smarttable entries began to work, but they stopped working if I put it back. I rolled up my sleeves and tried to understand ruleset 0. I became pretty convinced that mailertable runs before smarttable, and that if mailertable made a successful lookup of the recipie4nt domain that’s it. You’re done. Then I read a very brief comment in Andrzej’s writeup of smarttable. He bascially said mailertable runs first, and I guess form that brief comment you’re supposed to know that this setup I am trying will never work. And yet I have a colleague with a similar setup who says it does work, which got my competitive juices flowing – if the other guy can do it, then I know it can be done and I can do it as well! I knew I had to eliminate the catch-all entry, but still needed a catch-all feature. What to do? Fudge DNS to the point where all TLDs have a fake MX entry pointing at my external mail server?? Sounds too kludgy.

I was reading here and there in the Sendmail book and the cf/README file when SMARTHOST caught my eye. A smarthost can be defined to deliver all email to a specified relay. I always viewed it as an alternative to the mailertable, where you can specify much more specific delivery rules. But maybe they could co-exist? Mailertable for domain-specific delivery instructions, and smarthost for everything else? Yes, you can indeed do that! And with the catchall entry gone from mailertable does smarttable begin to work? Yes! It does! So our outbound stream is in good shape and all requirements are met. Each user of the SMGW is entered in the smarttable:

[email protected]    relay:smgw.drj.com
[email protected]    relay:smgw.drj.com
etc.

The mc configuration for smarthost looks like this:

dnl smarthost: to take care of the everything-else delivery case - DrJ 2/28/14
define(`SMART_HOST',`internet-facing-relay.drj.com')

Inbound
For inbound we can use virtusertable to rewrite the user domain to a fictional domain, then a mailertable entry which describes that this fictional domain should be routed over to the SMGW! Like this.

Virtusertable

and

Mailertable

drj.com   relay:native-mail-system-gw.drj.com
.smgw     relay:smgw.drj.com

Although this sounded good on paper, I found it was not enough by itself. In addition I needed to throw in a virtuser domain file which included the domain drj.com.

The mc configuration for virtusertable and virtuserdomain look like this.

dnl
dnl add virtusertable to do recipient re-writing to accomodate SMGW routing -DrJ 3/3/14
FEATURE(`virtusertable',`hash -o /etc/mail/virtusertable')dnl
dnl virtuser domain file location - DrJ 2/27/14
VIRTUSER_DOMAIN_FILE(`/etc/mail/virtuserdomains')dnl

virtuserdomains is a standard text file which contains drj.com.

Testing
It is helpful to know how to debug this stuff. Smarttable is probably the hardest. Here’s how to see the before and after effect that smarttable has:

$ sendmail -Csendmail-test.cf -bt<<END
> 3,0 [email protected]
> [email protected]
> 3,0 [email protected]
> END

The 3,0 lines show the result of running rulesets 3 and 0, I guess. The .Df…line defines the sender, so the last 3,0 line shows how delivery to the sender is now altered if the defined sender exists in the smarttable table.

Regular addresses can be tested with the same -bt switch if you want to see the gory details, or simply -bv:

$ sendmail -Csendmail-test.cf -bt< 3,0 [email protected]
> END

$ sendmail -Csendmail-test.cf -bv [email protected]

It’s assumed you put your sendmail config in sendmail-test.cf to not interfere with production.

Even if all the tests succeed, what I found is that smarthost did not take effect dynamically. I needed to re-start sendmail.

Conclusion
By digging into the innards of sendmail we learned enough to see how things should work together and found that it is indeed possible for smarttable, virtusertable and mailertable to peacefully co-exist, but only with a helping of smarthost and virtuserdomains!

References
I describe smarttable here.
Andrzej’s smarttable page is here.

Categories
Admin Linux Network Technologies Raspberry Pi

Using your Raspberry Pi as a router

Intro
Most Raspberry Pi router HowTos describe how to make the Pi act like an Access Point. That can be very useful, and kind of tricky. Here I’ve turned wireless and wired network roles around and show how to make it route traffic from a wired LAN over its WiFi connection.

I bought the EDIMAX nano USB adapter to play around with wireless on my Pi. It wasn’t long before the network lover in me realized, Hey I got an ethernet port on one end, a wireless on the other – that sounds like a potential router – let’s have some fun! So I hooked up my cantankerous Sony Blueray player to it. The thing has always been touchy about using wireless but it also has a wired LAN connection option – in a room where I don’t have a wired ethernet network available. Enter the routing Pi…

WiFi on the Pi
There are many ways to get the thing going. The important thing to note is that this EDIMAX is tested and is known to be compatible with the Raspberry Pi. I guess they even included the appropriate driver for it, because there is no need at any time to read the mini-CD that the EDIMAX comes with and try to pull of, or worse, compile, a Linux driver like I initially feared. I think you merely need to do one of these numbers:

$ sudo apt-get update

to ensure you have the latest of everything on your Pi and that includes the driver for this WiFi adapter.

Also note that the Pi, being a tiny computer, is perfectly complemented by this nano adapter. The thing is so tiny you barely have enough surface area to pull it out of the USB slot.

As I said there are probably many ways to get your WiFi going. Being a command line lover I present that way, and even then there are alternative setups to choose from, of which I present only one here.

If you do one of these numbers:

$ cd /etc/network; sudo nano interfaces

I would recommend to add a line towards the top:

auto wlan0

and at the bottom:

# following http://antael.blogspot.com/2013/01/wifi-is-live-and-kicking.html
# to see Wifi signal strength and available signals run iwlist wlan0 scan
#allow-hotplug wlan0
iface wlan0 inet static
wpa-ssid 
wpa-psk 
wpa-key_mgmt WPA-PSK
address 192.168.0.90
gateway 192.168.0.254
netmask 255.255.255.0
broadcast 192.168.0.255

substitute the name of your WiFi SSID and the password (WPA Pre-Shared Key) in place of <MYSSID> and <MYSSIDPASSWORD>.

I prefer static IPs to dynamic ones. That way I ssh to the IP and it is the same every time.

As I am setting up a router this is not my complete interfaces file, just the part I want to emphasize for now. The complete file is listed below.

A word on what we are setting out to do – our network architecture

OK. So I am going to put the Pi next to the Blueray Player. The Pi will connect to my TP-LINK WiFi network. The Pi’s ethernet interface will connect to the LAN port on the Blueray player using a standard ethernet patch cable (these days it is not necessary to use a crossover cable as there’s always a device that auto-senses how it is wired). I will create a new network for this interface on the Pi, and make sure I’ve assigned a valid IP on this network to the Blueray player. The network I choose is 10.31.42.0/24 – something completely different from anything I may already have in use at home.

The Blueray player can also be assigned a static IP address. I give it 10.31.42.13 and the Pi I assign 10.31.42.11. The gateway of the Blueray player is our routing Pi, so that makes the gateway IP 10.31.42.11.

But I have another requirement: I want to conveniently put the Pi back on my wired home network in case something goes wrong with wireless. so I want to keep its valid static IP (192.168.2.100) which it used to have when it was wired to my switch. The way to accomplish all this is to use virtual IPs on the eth0 interface. See the full /etc/network/interfaces file below which shows eht0 still assigned to IP 192.168.2.200, but as well a virtual interface eth0:0 assigned to 10.31.42.11.

Convenient way to test things
Initially I was focused on the architecture I’ve outline above and getting all my interfaces up and running. And I could run, for instance,

$ sudo service networking restart

to make my changes to the interfaces file be dynamically enabled, and I can do a

$ ifconfig -a

to show all my interfaces, their IPs and other information. And I especially like

$ iwlist wlan0 scan

to show the available WiFi networks and their signal strength. But how do I know that routing is working?? Here’s how. You know the ping command, right? Normally you do

$ ping 8.8.8.8

to show you can reach the Internet. Why? because that’s a valid IP address on the Internet that responds to PING – thank you Google – and it’s easy to remember!

But since our default route is out of our WiFi-connected interface, the IP it picks for the source of that PING is the IP assigned to wlan0, namely 192.168.0.90. There’s no Pi-routing involved in that so far. But now, you can choose a different source IP for your pings. So we pick our new virtual IP, 10.31.42.11 like this:

$ ping -I 10.31.42.11 8.8.8.8

but instead of a nice result like

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_req=1 ttl=44 time=172 ms
64 bytes from 8.8.8.8: icmp_req=2 ttl=44 time=92.7 ms
64 bytes from 8.8.8.8: icmp_req=3 ttl=44 time=69.6 ms
64 bytes from 8.8.8.8: icmp_req=4 ttl=44 time=70.6 ms
64 bytes from 8.8.8.8: icmp_req=5 ttl=44 time=70.3 ms
64 bytes from 8.8.8.8: icmp_req=6 ttl=44 time=70.5 ms
^C
--- 8.8.8.8 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5006ms
rtt min/avg/max/mdev = 69.676/91.121/172.740/37.409 ms

we don’t get much encouragement:

PING 8.8.8.8 (8.8.8.8) from 10.31.42.11 : 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms

The ^C above means <CTRL-C> was typed in at the keyboard.

So we’re seeing 100% packet loss and we can safely conclude no routing is occurring. By the way, as a reality and typo check on this approach, try to specify as source your wlan0 IP, as in

$ ping -I 192.168.0.90 8.8.8.8

This should work.

Turn on routing
I feared it would be mess to get routing turned on, but it turns out to be pretty tidy, considering. Probably your Pi is already enabled for routing. Mine was. Just

$ cat /proc/sys/net/ipv4/ip_forward

and make sure the value is 1. That means it is set up to do routing. If you’re not so lucky and you have 0, enable routing:

$ sudo sysctl -w net.ipv4.ip_forward=1

But of course there’s more we have to do or else our source ping would have worked!

We need to turn on network address translation. This is the part I was worried about, never having done it before on Linux, but only with expensive products like commercial firewalls. But it’s really not bad at all.

I created a file iptables-NAT in the home directory of the pi user with these contents:

#!/bin/sh
# This is a one-time script - DrJ
# 2/2014
# explained nicely in http://www.karlrupp.net/en/computer/nat_tutorial
# and seems to even work!
 
# flush old iptables stuff. Need to specify nat table specifically
iptables -t nat -F
iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
 
# make it permanent
#/usr/sbin/service iptables save
 
# list it
iptables -t nat -L
 
# persist it:
 
iptables-save &gt; /etc/iptables.conf
 
# note that in /etc/network/interfaces we added this line to read in
# iptables.conf upon reboot:
# pre-up iptables-restore &lt; /etc/iptables.conf

and ran it:

$ cd; sudo ./iptables-NAT

It outputs this:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
 
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
 
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
 
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  anywhere             anywhere

Now run our source ping:

$ ping -I 10.31.42.11 8.8.8.8

PING 8.8.8.8 (8.8.8.8) from 10.31.42.11 : 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_req=1 ttl=44 time=76.3 ms
64 bytes from 8.8.8.8: icmp_req=2 ttl=44 time=71.4 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 71.471/73.922/76.373/2.451 ms

Voila! It worked. We now are NATing our source address of 10.31.42.11 to our wlan IP of 192.168.0.90 before sending the packet along to the next router. But as skeptics, we’d like to see some more empirical proof that all that is really happening. You can. Here’s how using tcpdump.

The thing is that tcpdump doesn’t have full access to pre-processing, it only has “post processing” access. So we need to hook up the Pi to a device we are going to route traffic for, in my case the Sony Blueray player. I don’t have much control, but I can do a network diagnostics which I know tries to reach the configured DNS server (because it complained that it could not reach the DNS server 8.8.8.8 during one of my early tests. So tee up tcpdump to monitor all interfaces’ traffic to 8.8.8.8 like this:

$ sudo tcpdump -n -i any host 8.8.8.8

Then run ping (or a DNS query) to 8.8.8.8 on that device, and voila, this is the result:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
20:22:09.041922 IP 10.31.42.13.54693 &gt; 8.8.8.8.53: 5+ A? m.root-servers.net. (36)
20:22:09.042144 IP 192.168.0.90.54693 &gt; 8.8.8.8.53: 5+ A? m.root-servers.net. (36)
20:22:09.163618 IP 8.8.8.8.53 &gt; 192.168.0.90.54693: 5 1/0/0 A 202.12.27.33 (52)
20:22:09.163777 IP 8.8.8.8.53 &gt; 10.31.42.13.54693: 5 1/0/0 A 202.12.27.33 (52)

The -i any switch told it to listen on all interfaces. Unfortunately the output doesn’t show which interface, but you can easily deduce it. the traffic comes in from my Blueray player at 10.31.42.13 on eth0, and then leaves the Pi on wlan0 having had its source IP translated to 192.168.0.90. And the reverse happens to the response from 8.8.8.8. In fact I see from this tcpdump output that the Blueray player does not do a PING, it does an actual DNS query to 8.8.8.8. Makes sense since it is a DNS server.

Be persistent
But a reboot will undo this nice NATing we have achieved. Try it. So we have to find a way to make our iptables entry persist across reboots. I’m sure there are many ways to do this. I liked this one. You add this line to the end of your interfaces file:

pre-up iptables-restore &lt; /etc/iptables.conf

And of course we already anticipated this manner of proceeding in our iptables-NAT file when we included this line at the bottom:

iptables-save &gt; /etc/iptables.conf

Now reboot again, and try that source ping as the first command you issue. Now it should work.

Ready for the real test
Now I can hook up my Blueray player with some confidence that at least the networking should be working. Assign its IP (10.31.42.13 as mentioned above), its gateway (the Pi’s eth0:0 virtual IP, namely 10.31.42.11 in our example), a DNS server (8.8.8.8, of course!) and see if:

– we can ping 10.31.42.13 from the Pi
– the Blueray player’s network tester shows OK (there aren’t many fine debugging utilities, just this single “network test”)

If the Blueray player can’t reach the configured DNS server then it its test fails.

Why this big long explanation for something that no one else in the world wants to do?
All this sounds like a very, very specific application that doesn’t apply to anyone else. But once you understand some of the core networking principles involved that I’ve touched on here, you will begin to realize that this can be very easily generalized to something much more applicable and powerful: a full-blown replacement for a standard wireless router such as my TP-LINK nano router. After all, we’ve set up almost all the utilities and facilities that you get from a standard router (NATing and routing), perhaps with one glaring exception: a DHCP service. I personally didn’t want one, but nothing prevents you from setting that up on the wired side. I give some suggestions below on how to do that in the next section.

A few words on a DHCP service
I did get that running on the Pi as well, though for an entirely different purpose. I used dnsmasq:

$ sudo apt-get install dnsmasq

and I think if you edit /etc/dnsmasq.conf and put these lines at the bottom:

interface=eth0
dhcp-range=10.31.42.14,10.31.42.254,10h
dhcp-option=3,10.31.42.11

I think you will pretty much have a working DHCP service as well and could use that instead of assigning static IPs.

What I fear about the DHCP service – and I think I have seen this – is that if I put the Pi back to the wired network, its DHCP server competes with the normal router’s DHCP service, and I start to lose connectivity to my devices! So be careful. If devices like your PC start to pick up 10.31.42.x addresses on a network where 192.168.2.x is expected, there could be some connectivity troubles ahead until you disconnect the Pi!

Finally, the full /etc/network/interfaces file

auto lo
auto eth0
auto eth0:0
auto wlan0
 
iface lo inet loopback
# DrJ change: make IP static
# somewhat inspired by http://www.techiecorner.com/486/how-to-setup-static-ip-in-debian/ - DrJ 1/8/13
#iface eth0 inet dhcp
iface eth0 inet static
address 192.168.2.100
#gateway  192.168.2.1
netmask 255.255.255.0
network 192.168.2.0
broadcast 192.168.2.255
 
# for network 3142
iface eth0:0 inet static
address 10.31.42.11
netmask 255.255.255.0
network 10.31.42.0
broadcast 10.31.42.255
 
# following http://antael.blogspot.com/2013/01/wifi-is-live-and-kicking.html
# to see Wifi signal strength and available signals run iwlist wlan0 scan
#allow-hotplug wlan0
iface wlan0 inet static
wpa-ssid 
wpa-psk 
wpa-key_mgmt WPA-PSK
address 192.168.0.90
gateway 192.168.0.254
netmask 255.255.255.0
broadcast 192.168.0.255
# wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf
#iface default inet dhcp
 
# to read in iptable configuration. See ~pi/iptables-NAT
# added - DrJ 3/2/14
pre-up iptables-restore &lt; /etc/iptables.conf

Conclusion
Yes, we got our Pi to act as a router, and it wasn’t too bad. We demonstrated it for this specific application, but also showed how the simple addition of a DHCP service would make this a very general solution.
Did it help with the problem at hand – getting smoother streaming on the Sony Blueray player? Well, actually, yes, it did seem to help.

References and related

An idea for temporarily replacing a busted home router with a Raspberry Pi router which uses your hotspot is described in this article.

Getting started on a Pi without a dedicated console is described here.
A look at playing around with real-time video using the Pi’s camera is described here.
A more esoteric project, using the Pi to monitor your home’s Internet/power connection, is presented here.
Turning your Raspberry Pi into a transparent networking bridge is described here.
Interested to run some real networking protocols on your Pi like RIP, OSPF or BGP? I suggest to look into Quagga. Quagga is a networking suite. In full disclosure I haven’t had time or motivation to experiment with it myself (yet).

Categories
Admin Linux

Linux command line tips

Intro
I love the Linux command line. I don’t know how to categorize the tips I have collected so I’m dumping them here for the time being.

The newer netstat
In the old days I used to constantly do

$ netstat -an|grep LISTEN|grep <PORT>

to see what tcp ports I had processes listening on. While that is very helpful, it doesn’t tell you the listening process. A better approach:

$ lsof -i tcp:<PORT>

where is 22, 80 or whatever you want to look for. lsof does show the process.

Repeat last argument

I am constantly doing things like

$ grep <string> <big-file> > /tmp/results

and then I want to have a look at that file but I don’t want to type in

$ more /tmp/results

so what do I do?

I type this to give me the last argument:

$ more !$

My fingers have memorized that pattern and so I do it without conscious thought, and you’re holding down the SHIFT key so it can be typed very quickly.

But sometimes I can’t wait to type even that. I want to collect the grep results into a file in case there were a lot of matches, but I also want to see the matching results on my screen as soon as they are available. What do I do? Use tee, as in:

$ grep <string> <big-filez> |tee /tmp/results|more

Cool, huh?

More Obscure stuff
A cool blog with lots of good, obscure tips, generally more technical than the ones above, is https://die-computerhilfe.de/blog

To be continued…

Conclusion

Categories
Linux Raspberry Pi

Superimposing a grid on your raspivid output

Intro
In my previous post I outlined how to get real-time video from your Raspberry Pi with its camera, and to make it somewhat robust. In the conclusion I mentioned that it would be nice to superimpose (overlay) a grid over that image, and speculated that openCV might be just the tool to do it. Here I demonstrate how I have done it, and what compromises I had to make along the way.

The details
Well, let’s talk about why I didn’t go the openCV route. I began to bring down the source code for raspivid and raspistill, as outlined in this series of blog posts. And I did get it to compile, but it’s a lot of packages to bring down, and then I still needed to add in the openCV stuff. He provided one example of a hacked source file, but for raspistill, and I needed raspivid which is slightly different. Then there was cmake to master – I have no idea never having used it before. And then I would have needed to figure out openCV, which in turn might require programming in C++, which I have only the most basic skills. And then after all that, my fear was that it would slow down the video to the point where we would lose the real-time aspect! So the barriers were many, the risk was great, the reward not that great.

Logic dictates there should be another way
I reasoned as follows. Windows display graphics. Something decides what pixels to display, and this is true for every window, including mplayer. So if you can get control of what decides how to draw pixels in Windows we can draw our grid on the client side in Windows rather than on the encoder side on the Pi. So I looked for a way to superimpose an image using mplayer. Though they don’t use that term, I soon was drawn to what sounded similar, a -vf (video filter) switch with post-processing capability. I don’t know how to bring up the mplayer documentation in Windows, but on the Pi it’s just

$ man mplayer

and you’ll get a whole long listing. Under -vf are different filters, non of which sounded very promising. then I came across geq (general equation). That sounded pretty good to me. I searched for examples on the web and came across this very helpful discussion of how to use it, with examples.

So, off I went. A lot of the stuff I tried initially didn’t work. Then, when it did work, it lost the real-time feature that’s so important to us. Or the convergence to real-time took too long. I finally settled on this string for my mplayer:

mplayer -vf geq=p(X\,Y)*(1-gt(mod(X/SW\,100)\,98))*(1-gt(mod(Y/SH\,100)\,98)) -ontop -fps 27 -vo gl -cache 1024 -geometry 600:50 -noborder -msglevel all=0 -

in combination with these switches on raspivid:

raspivid -n -o - -t 9999999 -rot 180 -w 560 -h 420 -b 1000000 -fps 9

And, voila, my grid of black lines appears at 100 pixel intervals. Convergence is about 30 seconds and we have preserved the real-timeyness of the video!

But it as a series of compromises and tuning that got me there. For my desired 640 x 480 video I could get real-time video at about 7 fps (frame per second). I think my PC, a Dell Insipron 660, just can’t keep up at higher fps. Because when you think about it, it’s got to do calculations for each and every pixel, which must introduce quite some overhead. Perhaps things will go better on PCs that don’t need the -vo gl switch of mplayer which I have to use on my Dell display. So I kept the pixels per second constant and calculated what area I would have to shrink the picture to to increase the fps to a value that gave me sufficient real-timeyness. I decided there was a small but noticeable difference between 7 fps and 9 fps.

So

pixels/second = fps * Area,

so keeping that constant,

7 fps * A7 = 9 fps * A9

A = w*h = w*((3/4)*w)

So after some math you arrive at:

w9 = w7*sqrt(7/9) = 640 * 0.935 ~ 560 pixels

and h9 = w9*3/4 = 420 pixels

And that worked out! So a slightly smaller width gives us fewer pixels to have to calculate, and allows us to converge to real-time and have almost unnoticeable lag.

What we have now
One the Pi the /etc/init.d/raspi-vid now includes this key line:

raspivid -n -o - -t 9999999 -rot 180 -w 560 -h 420 -b 1000000 -fps 9|nc  -l 443

and on my PC the key line in my .bat file now looks like this:

c:\apps\netcat\nc 192.168.0.90 443|c:\apps\smplayer\mplayer\mplayer -vf geq=p(X\,Y)*(1-gt(mod(X/SW\,100)\,98))*(1-gt(mod(Y/SH\,100)\,98)) -ontop -fps 27 -vo gl -cache 1024 -geometry 600:50 -noborder -msglevel all=0 -

For the full versions of the files, and more discussion about the switches I chose, go back to my previous article about screaming streaming on the Pi, and just substitute in these lines in the obvious place. Adjust the IP to your Pi’s IP address.

No tick marks?
My original goal was to innlude tick marks, but I see given the per-pixel calculations required that that’s gonna be a lot more complicated and could only further slow us down (or force us to reduce the picture size further). So for now I think I’ll stop here.

A word on YUV coloring
I am much more comfortable with RGB, but that seems not to be used in Pi video stream. I guess raspivid encodes using YUV. I haven’t mastered the representation of YUV, but here’s a couple words on it anyways! I’m sure it’s related to YCbCr, which is described here. So because groups of pixels share a color, if you change the function above to mod(X/SW\,101),99), for instance, you get alternating green and black grid lines as you go from even to odd pixels. That is my vague understanding at this point. I learned just enough to get my black grid lines but no more…

Unsolved Mystery
Although the approach outlined above does generally work and can be real-time, I find that it also gets laggy when i leave the room and there is no motion. I’m not sure why. Then I introduce motion and it converges again to real-time. I don’t think this behaviour was so noticeable before I added the grid lines, but I need more tests.

Conclusion
We’ve shown how to overlay a black grid on the video output of our Raspberry Pi, while keeping the stream real-time with almost unnoticeable lag.

Conclusion
We have managed to overlay a black grid on our video using built-in functionality of mplayer. It appreciably slows things down. so your mileage may vary depending on your hardware.

References
The original Screaming Streaming on the Raspberry Pi article.

Categories
Admin IT Operational Excellence Linux Network Technologies Raspberry Pi

Screaming Streaming on the Raspberry Pi

Intro
The Raspberry Pi plus camera is just irresistible fun. But I had a strong motivation to get it to work the way I wanted it to as well: a First robotics team that was planning on using it for vision for the drive team. So of course those of us working on it wanted to offer something with a real-time view of the field with a fast refresh rate and good (though not necessarily perfect) reliability. Was it all possible? Before starting I didn’t know. In fact I started the season in January not knowing the team would want to use a Raspberry Pi, much less that there was a camera for it! But we were determined to push through the obstacles and share my love of the Pi with students. Eventually we found a way.

The details
Well, we sure made a lot of missteps along the way, that’s why I’m excited to write this article to help others avoid some of the pain points. It needs to be fleshed out some more, but this post will be expanded to become a litany of what didn’t work – and that list is pretty long! All of it borrowed from well-meaning people on various Internet sites.

The essence of the solution is the quick start page – I always search for Raspberry pi camera quick start to find it – which basically has the right idea, but isn’t fleshed out enough. So raspivid + nc + a PC with netcat (nc) and mplayer will do the trick. Below I provide a tutorial on how to get it all to work.

Additional requirement
Remember I wanted to make this almost fool-proof. So I wanted the Pi to be like a passive device that doesn’t need more than a one-time configuration. Power-up and she’s got to be ready. Cut power and re-power, it better be ready once more. No remote shell logins, no touching it. That’s what happens when it’s on the robot – it suddenly gets powered up before the match.

Here is the startup script I created that does just that. I put it in /etc/init.d/raspi-vid:

#! /bin/sh
# /etc/init.d/raspi-vid
# 2/2014
 
# The following part always gets executed.
echo "This part always gets executed"
 
# The following part carries out specific functions depending on arguments.
case "$1" in
  start)
    echo "Starting raspi-vid"
# -n means don't show preview on console; -rot 180 to make image right-side-up
# run a loop because this command dies unless it can connect to a listener
    while /bin/true; do
# if acting as client do this. Probably it's better to act as server however
# try IPs of the production PC, test PC and home PC
#      for IP in 10.31.42.5 10.31.42.6 192.168.2.2; do
#        raspivid -n -o - -t 9999999 -rot 180 -w 640 -h 480 -b 800000 -fps 15|nc $IP 80
#      done
#
# act as super-simple server listening on port 443 using nc
# -n means don't show preview on console; -rot 180 to make image right-side-up
# -b (bitrate) of 1000000 (~ 1 mbit) seems adequate for our 640x480 video image
# so is -fps 20 (20 frames per second)
# To view output fire up mplayer on a PC. I personally use this command on my PC:
# c:\apps\netcat\nc 192.168.2.100 443|c:\apps\smplayer\mplayer\mplayer -ontop -fps 60 -vo gl -cache 1024 -geometry 600:50 -noborder -msglevel all=0 -
      raspivid -n -o - -t 9999999 -rot 180 -w 640 -h 480 -b 1000000 -fps 20|nc  -l 443
# this nc server craps out after each connection, so just start up the next server automatically...
      sleep 1;
    done
    echo "raspi-vid is alive"
    ;;
  stop)
    echo "Stopping rasip-vid"
    pkill 'raspi-?vid'
    echo "raspi-vid is dead"
    ;;
  *)
    echo "Usage: /etc/init.d/rasip-vid {start|stop}"
    exit 1
    ;;
esac
 
exit 0

I made it run on system startup thusly:

$ cd /etc/init.d; sudo chmod +x raspi-vid; sudo update-rc.d raspi-vid defaults

Of course I needed those extra packages, mplayer and netcat:

$ sudo apt-get install mplayer netcat

Actually you don’t really need mplayer, but I frequently used it simply to study the man pages which I never did figure out how to bring up on the Windows installation.

On the PC I needed mplayer and netcat to be installed. At first I resisted doing this, but in the end I caved. I couldn’t meet all my requirements without some special software on the PC, which is unfortunate but OK in our circumstances.

I also bought a spare camera to play with my Pi at home. It’s about $25 from newark.com, though the shipping is another $11! If you’re an Amazon Prime member that’s a better bet – about $31 when I looked the other day. Wish I had seen that earlier!

I guess I used the links provided by the quick start page for netcat and mplayer, but I forget. As I was experimenting, I also installed smplayer. In fact I ended up using the mplayer provided by smplayer. That may not be necessary, however.

A word of caution about smplayer
smplayer, if taken from the wrong source (see references for correct source), will want to modify your browser toolbar and install adware. Be sure to do the Expert install and uncheck everything. Even so it might install some annoying game which can be uninstalled later.

Lack of background
I admit, I am no Windows developer! So this is going to be crude…
I relied on my memory of some basics I picked up over the years, plus analogies to bash shell programming, where possible.

I kept tweaking a batch file on my desktop. So I associated notepad to my Send To menu. Briefly, you type

shell:sendto

where it says Search programs and files after clicking the Start button. Then drag a copy of notepad from c:\windows\notepad into the window that popped up.

Now we can edit our .bat file to our heart’s content.

So I created a mplayer.bat file and saved it to my desktop. Here are its contents.

if not "%minimized%"=="" goto :minimized
set minimized=true
start /min cmd /C "%~dpnx0"
goto :EOF
:minimized
rem Anything after here will run in a minimized window
REM DrJ 2/2014
rem 
rem very simple mplayer batch file to play output from a Raspberry Pi video stream
rem
rem Use the following line to set up a server
REM c:\apps\netcat\nc -L -p 80|c:\apps\smplayer\mplayer\mplayer -fps 30 -vo gl -cache 1024 -msglevel all=0 -

rem Set up as client with this line...
rem put in loop because I want it to start up whenever there is something listening on port 80 on the server
 
:loop

 
rem this way we are acting as a client - this is more how you'd expect and want things to work
c:\apps\netcat\nc 192.168.2.102 443|c:\apps\smplayer\mplayer\mplayer -ontop -fps 60 -vo gl -cache 1024 -geometry 600:50 -noborder -msglevel all=0 -

rem stupid trick to sleep for about a second. Boy windows shell is lacking...
ping 127.0.0.1 -n 2 -w 1000 > NUL
 
goto loop

A couple notes about what is specific to my installation. I like to install programs to c:\apps so I know I installed them by hand. So that’s why smplayer and netcat were put there. Of course 192.168.2.102 is my Pi’s IP address on my home network. In this post I describe how to set a static IP address for your Pi. We also found it useful to have the CMD Window minimize itself after starting up and running in the background, so the I discovered that the lines on the top allow that to happen.

The results
With the infinite loops I programmed either Pi or mplayer.bat can be launched first – there is no necessary and single order to do things in. So it is a more robust solution than that outlined in the quick start guide.
Most of my other approaches suffered from lag – delay in displaying a live event. Some other suggested approaches had quite large lag in fact. The lag from the approach I’ve outlined above is only 0.2 s. So it feels real-time. It’s great. Below I outline a novel method (novel to me anyways) of measuring lag precisely.
Many of my other approaches also suffered from a low refresh rate. You’d specify some decent number of frames per second, but in actual fact you’d get 1 -2 fps! That made for choppy and laggy viewing. With the approach above there is a full 20 frames per second so you get the feel of true motion. OK, fast motions are blurred a bit, but it’s much better than what you get with any solution involving raspistill: frame updates every 0.6 s and nothing you do can speed it up!
Many Internet video examples showed off high-resolution images. I had a different requirement. I had to keep the bandwidth usage tamped down and I actually wanted a smaller image, not larger because the robot driver has a dashboard to look at.
I chose an unconventional port, tcp port 443, for the communication because that is an allowed port in the competition. The port has to match up in raspi-vid and mplayer.bat. Change it to your own desired value.

Limitations
Well, this is a one-client at a time solution, for starters! did I mention that nc makes for a lousy server?
Even with the infinite looping, things do get jammed up. You get into situation where you need to kill the mplayer CMD window to get things going again.
I would like to have gotten the lag down even further, but haven’t had time to look into it.
Begin a video amateur I am going to make up my own terms! This solution exhibits a phenomenon I call convergence. What that means is that once the mplayer window pops up, which takes a few seconds, what it’s displaying shows a big lag – about 10 seconds. But then it speeds along through the buffered frames and converges with real-time. This convergence takes slightly more than 10 seconds. So if you need instant-on and real-time, you’re not getting it with this solution!

What no one told us
I think we were all so excited to get this little camera for the Pi no one bothers to talk about the actual optical properties of the thing! And maybe they should. because even if it is supposedly based on a cellphone camera, I don’t know which cellphone, certainly not the one from my Samsung Galaxy S3. The thing is (and I admit someone else first pointed this out to me) that it has a really small field-of-view. I measured it as spreading out only 8.5″ at a 15″ distance – that works out to only 31.6 degrees! See what I mean? And I don’t believe there are any tricks or switches to make that larger – that’s dictated by the optics of the lens. This narrow field-of-view may make it unsuitable for use as security camera or many other projects, so bear that in mind. If I put my Samsung next to it and look at the same view its field of view is noticeably larger, perhaps closer to 45 degrees.

Special Insights
At some point I realized that the getting started guide put things very awkwardly in making the PC the server and the Pi the client. You normally want things the other way around, like it would be for an ethernet camera! So my special insight was to realize that nc could be used in the reverse way they had documented it to switch client/server roles. nc is still a lousy “server,” if you can call it that, but hey, the price is right.

Fighting lag
To address the convergence problem mentioned above I chose a frame rate much higher on the viewer than on the camera. The higher this ratio the faster convergence occurs. So I have a 3:1 ratio: 60 fps on mplayer and 20 fps on raspivid. The PC does not seem to strain from the small bit of extra cpu cycles this may require. I think if you have an exact fps match you never get convergence, so this small detail alone could convince you that raspivid is always laggy when in fact it is more under your control than you realized.

Even though with the video quality such as it is there probably is no real difference between 10 fps and 20 fps, I chose 20 fps to reduce lag. After all, 10 fps means an image only every 100 msec, so on average by itself it introduces a lag of half that, 50 msec. Might as well minimize that by increasing the fps to make this a negligble contributor to lag.

Measuring lag
Take a smartphone with a stopwatch app which displays large numbers. Put that screen close up to the Pi camera. Arrange it so that it is next to your PC monitor so both the smartphone and the monitor are in your field of view simultaneously. Get mplayer.bat running on your PC and move the video window close to the edge of the monitor by the smartphone.

Now you can see both the smartphone screen as well as the video of the smartphone screen running the stopwatch (I use Swiss Army Knife) so you can glance at both simultaneously and quantify the lag. But it’s hard to look at both rapidly moving images at the same time, right? So what you do is get a second camera and take a picture of the two screens! We did this Saturday and found the difference between the two to be 0.2 s. To be more scientific several measurements ought to be taken and results avergaed and hundredths of seconds perhaps should be displayed (though I’m not sure a still picture could capture that as anything other than a blur).

mplayer strangeness on Dell Inspiron desktop
I first tried mplayer on an HP laptop and it worked great. It was a completely different story on my Dell Inspiron 660 home desktop however. There that same mplayer command produced this result:

...
VO: [directx] 640x480 => 640x480 Packed YUY2
FATAL: Cannot initialize video driver.
 
FATAL: Could not initialize video filters (-vf) or video output (-vo).
 
 
Exiting... (End of file)

So this was worrisome. I happened on the hint to try -vo gl and yup, it worked. Supposedly it makes for slower video so maybe on PCs where this trick is not required lag could be reduced.

mplayer personal preferences
I liked the idea of a window without a border (-noborder option) – so the only way to close it out is to kill the CMD window, which helps keep them in sync. Running two CMD windows doesn’t produce such good results!

I also wanted the window to first pop-up in the upper right corner of the screen, hence the -geometry 600:50

And I wanted the video screen to always be on top of other windows, hence the -ontop switch.

I decided the messages about cache were annoying and unimportant, hence the message suppression provided by the -msglevel all=0 switch.

Simultaneously recording and live streaming
I haven’t played with this too much, but I think the unix tee command works for this purpose. So you would take your raspivid line and make it something like:

raspivid -n -o – -t 9999999 -rot 180 -w 640 -h 480 -b 1000000 -fps 20|tee /home/pi/video-`date +%Y%h%d-%H%M`|nc -l 443

and you should get a nice date-and-time-stamped output file while still streaming live to your mplayer! Tee is an under-appreciated command…

Conclusion
I have tinkered with the Pi until I got its camera display to be screaming fast on my PC. I’ve shown how to do this and described some limitations.

Next Act?
I’m contemplating superimposing a grid with tick marks over the displayed video. This will help the robot driver establish their position relative to fixed elements on the field. This may be possible by integrating, for instance, openCV, for which there is some guidance out there. But I fear the real-time-ness may greatly suffer. I’ll post if I make any significant progress!
Update: I did get it to work, and the lag was an issue as suspected. Read about it here.

References and related
First Robotics is currently in season as I write this. The competition this year is Aerial Assist. More on that is at their web site, http://www3.usfirst.org/roboticsprograms/frc
Raspberry Pi camera quick start is a great place to get started for newbies.
Setting one or more static IP addresses on your Pi is documented here.
How not to set up your Pi for real-time video will be documented here.
How to get started on your Pi without a dedicated monitor is described here.
Finally, how to overlay a grid onto your video output (Yes, I succeeded to do it!) is documented here.
Correct source for smplayer for Windows.

Categories
Admin Apache Linux

Recording Host Header in the apache access log

Intro
Guess I’ve made it pretty clear in previous posts that Apache documentation is horrible in my opinion. So the only practical way to learn something is to configure by example. In this post I show how to record the Host header contained in an HTTP request in your Apache log.

The details
Why might you want to do this? Simple, if you have multiple hosts using one access log in common. For instance I have johnstechtalk.com and drjohnstechtalk.com using the same log, which I view as a convenience for myself. But now I want to know if I’m getting my money’s worth out of johnstechtalk.com, which I don’t see as the main URL, but I I use it to to type it into the browser location bar and get directed onto my site – fewer letters.

So I assume you know where to find the log definitions. You start with that as a base and create a custom-defined access log name. These two lines, taken from my actual config file, apache2.conf, show this:

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" \"%{Host}i\"" DrJformat

Then I have my virtual server in a separate file containing a reference to that custom format:

#CustomLog ${APACHE_LOG_DIR}/../drjohns/access.log combined
CustomLog ${APACHE_LOG_DIR}/../drjohns/access.log DrJformat

The ${APACHE_LOG_DIR} is an environment variable defined in envvars in my implementation, which may be unconventional. you can replace it with a hard-wired directory name if that suits you better.

There is some confusion out there on the Internet. Host as used in this post refers as I have said to the value contained in the HTTP Host Request header. It is not the hostname of the client.

Here are some recorded access resulting from this format early this morning:

108.163.185.34 - - [08/Jan/2014:02:21:32 -0500] "GET /blog/2012/02/tuning-apache-as-a-redirect-engine/ HTTP/1.1" 200 11659 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36" "drjohnstechtalk.com"
5.10.83.22 - - [08/Jan/2014:02:21:56 -0500] "GET /blog/2013/03/generate-pronounceable-passwords/ HTTP/1.1" 200 8253 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)" "drjohnstechtalk.com"
220.181.108.91 - - [08/Jan/2014:02:23:41 -0500] "GET / HTTP/1.1" 301 246 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" "vmanswer.com"
192.187.98.164 - - [08/Jan/2014:02:25:00 -0500] "GET /blog/2012/02/running-cgi-scripts-from-any-directory-with-apache/ HTTP/1.0" 200 32338 "http://drjohnstechtalk.com/blog/2012/02/running-cgi-scripts-from-any-directory-with-apache/" "Opera/9.80 (Windows NT 5.1; MRA 6.0 (build 5831)) Presto/2.12.388 Version/12.10" "drjohnstechtalk.com"

While most lines contain drjohnstechtalk.com, note that the next-to-last line has the host vmanswer.com, which is another domain one I bought and associated with my site to try it out.

Conclusion
We have shown how to record the contents of the Host header in an Apache access log.

Related rants against apache
Creating a maintenance page with Apache web server
Turning Apache into a Redirect Factory
Running CGI Scripts from any Directory with Apache

Categories
Admin Linux

The IT Detective Agency: Can someone really see what we’re doing in our X sessions?

Intro
We’ve been audited again. My most faithful followers will recall the very popular and educational article on SSL ciphers that cane out of our previous audit. So I guess audits are a good thing – helps us extend our learning.

This time we got dinged on that most ancient of protocols, X Windows. So this article is aimed at all those out there who, like me, know enough about X11 to get it more-or-less working, but not enough to claim power user status. The X cognescenti will find this article redundant with other material already widely available. Whatever. Sometimes I will post an article as though it were my own personal journal – and I like to have all my learning in one easy-to-find place.

The details
The findings amount to this: anyone in our Intranet can take a screen shot of what the people using Exceed are currently seeing. The nice tool (Nessus) actually provided such a screen shot to back up its claim, so this wasn’t a hypothetical. At Drjohn’s we believe in open source, but we do have our secrets and confidential information, so we don’t want everyone to have this type of access.

Here is some of the verbatim wording:

Synopsis
The remote X server accepts TCP connections.
 
Description
The remote X server accepts remote TCP connections. It is possible for an attacker to grab a screenshot of the remote host.
 
Solution
Restrict access to this port by using the 'xhost' command. If the X client/server facility is not used, disable TCP connections to the X server entirely.
 
Risk Factor
Critical
 
CVSS Base Score
10.0 (CVSS2#AV:N/AC:L/Au:N/C:C/I:C/A:C)
 
References
CVE CVE-1999-0526
...
Hosts
drjms.drjs.com (tcp/6002)
It was possible to gather the following screenshot of the remote computer.
...

So in my capacity as old Unix hand I was asked to verify the findings. This turned out to be dead easy. Here are the steps:

– pick a random Linux machine which has xwd installed
> xwd -debug -display drjms.drjs.com:2 -root -out drjms.xwd
> cat drjms.xwd|xwdtopnm|pnmtopng > drjms.png

The PNG file indeed showed lots of interesting stuff from a screen capture of the user’s X server – amazing.

I should mention that tcp port 600 maps to X Server display 0, 6001 to 1, 6002, to 2, etc. That’s why I set my display to drj…com:2 since port 6002 was mentioned in the findings as vulnerable.

My advice: don’t use

> xhost +

or whatever is the equivalent to that in Exceed onDemand.

Guilty
Now I have to admit to using xhost + in just this way in my own past – so convenient! Now that I see how dead easy it makes it to get a screenshot – in fact I tested the whole thing against my own XServer first – I will forego the convenience.

Conclusion
This is the danger in knowing something and some things, but not enough!

References
But I still stand by use of xhost + in the privacy of your home network, as for instance I show it in my Raspberry Pi without a monitor acticle.

I picked off that command set from this interesting article: https://www.linuxquestions.org/questions/linux-general-1/commanding-the-x-server-to-take-a-png-screenshot-through-ssh-459009/

Categories
Admin Linux

Setting up my Galaxy S3 for remote host access

Intro
I just got a Samsung Galaxy S3 last week. Before long I was wondering how I might use it to access my cloud server if indeed it was at all possible.

The Details
Looking at my other Android devices I decided to install Terminal Emulator. That’s a cute application, providing shell access to the underlying OS of your phone. But it’s fairly useless. You get a shell, but your account id, 10155, has essentially no privileges, and you can’t do much more than ls, cd, ps and top. You don’t have enough privileges to look into interesting directories. So you can’t do anything interesting. There’s also no native ssh so you can’t connect to another host.

To ssh to my Amazon cloud server I got the app ConnectBot. This app shows promise. I was able to connect to my server. I read some of the introductory screens which gave some helpful tips. So I quickly learned how to resize the window. I found 80×39 is a good size in portrait orientation. Yes, the font is tiny, but it is legible. Getting an elusive Esc or Ctrl character isn’t bad at all, just tap the top half of the screen.

So running constantly refreshing screens like top worked out fine.

vi was a problem. It used multi-colors in displaying my code. Comments, in dark blue, are not legible to me. In fact using vi at all on this device with a soft keyboard is quite unnatural. It might be better to use a curses-based editor like pico, though I haven’t yet tried it. But w/ vi, I found I could get rid of the multi-colors by setting the TERM environment variable to vt100. It had been screen screen. Once that was done:

> export TERM=vt100

vi displayed all characters in white, and background in black – quite legible.

Conclusion
It’s a strange world where you can administer a virtual server on a device that fits in the palm of your hand, and achieve fairly powerful effects.

Being a resourceful person, I had alternatives to reach my server. There is a web-based terminal emulator which works surprisingly well. See this post for a description.

connectBot is a native ssh remote terminal app and is actually useable as such on a Samsung Galaxy S3, if your eyes are still good! Pay attention to a just a few usage tips and you’ll be in full control of your server.

References
See this post for the world’s most natural, unobtrusive ringtone.

Categories
Admin Linux

The IT Detective Agency: Teraterm’s colors washed out

Intro
Some things we do as IT folk are just embarrassingly stupid in retrospect. This is such a story. I present it in keeping with my overall theme of throwing stuff out there in the hopes that some of it helps someone else facing the same situation.

The details
I love teraterm (from logmett.com). Teraterm plus screen (as in /usr/bin/screen) makes for a great combination when you live and die by the command line.

Actually I have been told I only use a small fraction of teraterm’s capabilities. It is programmable, apparently. I’m a very basic user.

So I had the self-assigned task to switch out a DNS server from an older Solaris OS to Linux. I completed the task months ago and I noticed one small side-effect: certain commands had the effect of washing out the font to just about the same color as the background. For the record my text is black (R,G,B) = (0,0,0) with Attribute Normal and my background is beige (255,255,183). When it’s behaving normally it looks very pleasant and is easy on the eyes.

I noticed when I ran man pages the text was all washed out – just a slightly brighter yellow against the beige background, same story when I ran vi. Comments such as text following # were washed out.

This was the case if I used a docking station. Using the native laptop display, the text was washed out, but not as severely so I could just make it out by straining my eyes.

I played with font color and background color in Teraterm, but didn’t really get anywhere, so I learned to cope. I learned that if I piped the man page to more the text was all-black and I didn’t really lose any functionality. In vi I learned that if I deleted the whitespace before the #, the whole comment became visible, unless it started a line. Kludgy, but it worked and hardly slowed me down – this is after all just one of many, many hosts I was focussed on.

Then it came time to migrate the second and last Solaris DNS server to Linux and I noticed the same thing happening on the new Linux server. What the…?

Previously I wasn’t really even sure when the washed-out problem occurred. This time I had no doubt that it was fine until the OS switch.

That in turn points to some difference in the environment, especially because on my many other Linux sessions I did not have this problem.

> env

shows the environment. By comparing where it was working to where it was not, I zeroed in on this environment variable: TERM.

TERM=vt100

where it wasn’t working

and

TERM=screen

where it was.

I set TERM=screen:

> export TERM=screen

and immediately noticed the display working when running vi. Even multiple colors are displayed.

But actually, hmm, the man pages are still washed out, e.g.,

> man -s1 ls

shows NAME, SYNOPSIS and DESCRIPTION are all yellowed out, as well as all switches! That makes it really difficult to decipher.

Oh, well. This mystery is not completely solved.

My point was going to be that in Solaris the TERM=vt100 made sense – things worked better – and so it was in my .bashrc file. In Linux (SLES) it didn’t make so much sense. No setting for TERM seems to be necessary as the value screen gets auto-defined somehow if you’re using screen.

What I had done was copy my .bashrc file from Solaris to Linux not really thinking about it. That’s what did me in on these two servers.

If I get around to resolving the man pages I’ll revise this post…

2020 update

Still plagued by this issue of washed out colors, I rolled up my sleeves and got it done. Turns out you have to set the Bold font settings separately.  I’m trying settings like in this picture.

References
Teraterm used to be available from logmett.com, (2020 update) but is no longer. I’m looking for a link… Here it is: https://osdn.net/projects/ttssh2/releases/

Conclusion
Problems with washed-out colors using teraterm plus screen are resolved. Once again, this was mostly a self-inflicted problem.