Category: Network Technologies

Network Technologies

Categories
Admin IT Operational Excellence Linux Network Technologies Raspberry Pi

Screaming Streaming on the Raspberry Pi

Intro
The Raspberry Pi plus camera is just irresistible fun. But I had a strong motivation to get it to work the way I wanted it to as well: a First robotics team that was planning on using it for vision for the drive team. So of course those of us working on it wanted to offer something with a real-time view of the field with a fast refresh rate and good (though not necessarily perfect) reliability. Was it all possible? Before starting I didn’t know. In fact I started the season in January not knowing the team would want to use a Raspberry Pi, much less that there was a camera for it! But we were determined to push through the obstacles and share my love of the Pi with students. Eventually we found a way.

The details
Well, we sure made a lot of missteps along the way, that’s why I’m excited to write this article to help others avoid some of the pain points. It needs to be fleshed out some more, but this post will be expanded to become a litany of what didn’t work – and that list is pretty long! All of it borrowed from well-meaning people on various Internet sites.

The essence of the solution is the quick start page – I always search for Raspberry pi camera quick start to find it – which basically has the right idea, but isn’t fleshed out enough. So raspivid + nc + a PC with netcat (nc) and mplayer will do the trick. Below I provide a tutorial on how to get it all to work.

Additional requirement
Remember I wanted to make this almost fool-proof. So I wanted the Pi to be like a passive device that doesn’t need more than a one-time configuration. Power-up and she’s got to be ready. Cut power and re-power, it better be ready once more. No remote shell logins, no touching it. That’s what happens when it’s on the robot – it suddenly gets powered up before the match.

Here is the startup script I created that does just that. I put it in /etc/init.d/raspi-vid:

#! /bin/sh
# /etc/init.d/raspi-vid
# 2/2014
 
# The following part always gets executed.
echo "This part always gets executed"
 
# The following part carries out specific functions depending on arguments.
case "$1" in
  start)
    echo "Starting raspi-vid"
# -n means don't show preview on console; -rot 180 to make image right-side-up
# run a loop because this command dies unless it can connect to a listener
    while /bin/true; do
# if acting as client do this. Probably it's better to act as server however
# try IPs of the production PC, test PC and home PC
#      for IP in 10.31.42.5 10.31.42.6 192.168.2.2; do
#        raspivid -n -o - -t 9999999 -rot 180 -w 640 -h 480 -b 800000 -fps 15|nc $IP 80
#      done
#
# act as super-simple server listening on port 443 using nc
# -n means don't show preview on console; -rot 180 to make image right-side-up
# -b (bitrate) of 1000000 (~ 1 mbit) seems adequate for our 640x480 video image
# so is -fps 20 (20 frames per second)
# To view output fire up mplayer on a PC. I personally use this command on my PC:
# c:\apps\netcat\nc 192.168.2.100 443|c:\apps\smplayer\mplayer\mplayer -ontop -fps 60 -vo gl -cache 1024 -geometry 600:50 -noborder -msglevel all=0 -
      raspivid -n -o - -t 9999999 -rot 180 -w 640 -h 480 -b 1000000 -fps 20|nc  -l 443
# this nc server craps out after each connection, so just start up the next server automatically...
      sleep 1;
    done
    echo "raspi-vid is alive"
    ;;
  stop)
    echo "Stopping rasip-vid"
    pkill 'raspi-?vid'
    echo "raspi-vid is dead"
    ;;
  *)
    echo "Usage: /etc/init.d/rasip-vid {start|stop}"
    exit 1
    ;;
esac
 
exit 0

I made it run on system startup thusly:

$ cd /etc/init.d; sudo chmod +x raspi-vid; sudo update-rc.d raspi-vid defaults

Of course I needed those extra packages, mplayer and netcat:

$ sudo apt-get install mplayer netcat

Actually you don’t really need mplayer, but I frequently used it simply to study the man pages which I never did figure out how to bring up on the Windows installation.

On the PC I needed mplayer and netcat to be installed. At first I resisted doing this, but in the end I caved. I couldn’t meet all my requirements without some special software on the PC, which is unfortunate but OK in our circumstances.

I also bought a spare camera to play with my Pi at home. It’s about $25 from newark.com, though the shipping is another $11! If you’re an Amazon Prime member that’s a better bet – about $31 when I looked the other day. Wish I had seen that earlier!

I guess I used the links provided by the quick start page for netcat and mplayer, but I forget. As I was experimenting, I also installed smplayer. In fact I ended up using the mplayer provided by smplayer. That may not be necessary, however.

A word of caution about smplayer
smplayer, if taken from the wrong source (see references for correct source), will want to modify your browser toolbar and install adware. Be sure to do the Expert install and uncheck everything. Even so it might install some annoying game which can be uninstalled later.

Lack of background
I admit, I am no Windows developer! So this is going to be crude…
I relied on my memory of some basics I picked up over the years, plus analogies to bash shell programming, where possible.

I kept tweaking a batch file on my desktop. So I associated notepad to my Send To menu. Briefly, you type

shell:sendto

where it says Search programs and files after clicking the Start button. Then drag a copy of notepad from c:\windows\notepad into the window that popped up.

Now we can edit our .bat file to our heart’s content.

So I created a mplayer.bat file and saved it to my desktop. Here are its contents.

if not "%minimized%"=="" goto :minimized
set minimized=true
start /min cmd /C "%~dpnx0"
goto :EOF
:minimized
rem Anything after here will run in a minimized window
REM DrJ 2/2014
rem 
rem very simple mplayer batch file to play output from a Raspberry Pi video stream
rem
rem Use the following line to set up a server
REM c:\apps\netcat\nc -L -p 80|c:\apps\smplayer\mplayer\mplayer -fps 30 -vo gl -cache 1024 -msglevel all=0 -

rem Set up as client with this line...
rem put in loop because I want it to start up whenever there is something listening on port 80 on the server
 
:loop

 
rem this way we are acting as a client - this is more how you'd expect and want things to work
c:\apps\netcat\nc 192.168.2.102 443|c:\apps\smplayer\mplayer\mplayer -ontop -fps 60 -vo gl -cache 1024 -geometry 600:50 -noborder -msglevel all=0 -

rem stupid trick to sleep for about a second. Boy windows shell is lacking...
ping 127.0.0.1 -n 2 -w 1000 > NUL
 
goto loop

A couple notes about what is specific to my installation. I like to install programs to c:\apps so I know I installed them by hand. So that’s why smplayer and netcat were put there. Of course 192.168.2.102 is my Pi’s IP address on my home network. In this post I describe how to set a static IP address for your Pi. We also found it useful to have the CMD Window minimize itself after starting up and running in the background, so the I discovered that the lines on the top allow that to happen.

The results
With the infinite loops I programmed either Pi or mplayer.bat can be launched first – there is no necessary and single order to do things in. So it is a more robust solution than that outlined in the quick start guide.
Most of my other approaches suffered from lag – delay in displaying a live event. Some other suggested approaches had quite large lag in fact. The lag from the approach I’ve outlined above is only 0.2 s. So it feels real-time. It’s great. Below I outline a novel method (novel to me anyways) of measuring lag precisely.
Many of my other approaches also suffered from a low refresh rate. You’d specify some decent number of frames per second, but in actual fact you’d get 1 -2 fps! That made for choppy and laggy viewing. With the approach above there is a full 20 frames per second so you get the feel of true motion. OK, fast motions are blurred a bit, but it’s much better than what you get with any solution involving raspistill: frame updates every 0.6 s and nothing you do can speed it up!
Many Internet video examples showed off high-resolution images. I had a different requirement. I had to keep the bandwidth usage tamped down and I actually wanted a smaller image, not larger because the robot driver has a dashboard to look at.
I chose an unconventional port, tcp port 443, for the communication because that is an allowed port in the competition. The port has to match up in raspi-vid and mplayer.bat. Change it to your own desired value.

Limitations
Well, this is a one-client at a time solution, for starters! did I mention that nc makes for a lousy server?
Even with the infinite looping, things do get jammed up. You get into situation where you need to kill the mplayer CMD window to get things going again.
I would like to have gotten the lag down even further, but haven’t had time to look into it.
Begin a video amateur I am going to make up my own terms! This solution exhibits a phenomenon I call convergence. What that means is that once the mplayer window pops up, which takes a few seconds, what it’s displaying shows a big lag – about 10 seconds. But then it speeds along through the buffered frames and converges with real-time. This convergence takes slightly more than 10 seconds. So if you need instant-on and real-time, you’re not getting it with this solution!

What no one told us
I think we were all so excited to get this little camera for the Pi no one bothers to talk about the actual optical properties of the thing! And maybe they should. because even if it is supposedly based on a cellphone camera, I don’t know which cellphone, certainly not the one from my Samsung Galaxy S3. The thing is (and I admit someone else first pointed this out to me) that it has a really small field-of-view. I measured it as spreading out only 8.5″ at a 15″ distance – that works out to only 31.6 degrees! See what I mean? And I don’t believe there are any tricks or switches to make that larger – that’s dictated by the optics of the lens. This narrow field-of-view may make it unsuitable for use as security camera or many other projects, so bear that in mind. If I put my Samsung next to it and look at the same view its field of view is noticeably larger, perhaps closer to 45 degrees.

Special Insights
At some point I realized that the getting started guide put things very awkwardly in making the PC the server and the Pi the client. You normally want things the other way around, like it would be for an ethernet camera! So my special insight was to realize that nc could be used in the reverse way they had documented it to switch client/server roles. nc is still a lousy “server,” if you can call it that, but hey, the price is right.

Fighting lag
To address the convergence problem mentioned above I chose a frame rate much higher on the viewer than on the camera. The higher this ratio the faster convergence occurs. So I have a 3:1 ratio: 60 fps on mplayer and 20 fps on raspivid. The PC does not seem to strain from the small bit of extra cpu cycles this may require. I think if you have an exact fps match you never get convergence, so this small detail alone could convince you that raspivid is always laggy when in fact it is more under your control than you realized.

Even though with the video quality such as it is there probably is no real difference between 10 fps and 20 fps, I chose 20 fps to reduce lag. After all, 10 fps means an image only every 100 msec, so on average by itself it introduces a lag of half that, 50 msec. Might as well minimize that by increasing the fps to make this a negligble contributor to lag.

Measuring lag
Take a smartphone with a stopwatch app which displays large numbers. Put that screen close up to the Pi camera. Arrange it so that it is next to your PC monitor so both the smartphone and the monitor are in your field of view simultaneously. Get mplayer.bat running on your PC and move the video window close to the edge of the monitor by the smartphone.

Now you can see both the smartphone screen as well as the video of the smartphone screen running the stopwatch (I use Swiss Army Knife) so you can glance at both simultaneously and quantify the lag. But it’s hard to look at both rapidly moving images at the same time, right? So what you do is get a second camera and take a picture of the two screens! We did this Saturday and found the difference between the two to be 0.2 s. To be more scientific several measurements ought to be taken and results avergaed and hundredths of seconds perhaps should be displayed (though I’m not sure a still picture could capture that as anything other than a blur).

mplayer strangeness on Dell Inspiron desktop
I first tried mplayer on an HP laptop and it worked great. It was a completely different story on my Dell Inspiron 660 home desktop however. There that same mplayer command produced this result:

...
VO: [directx] 640x480 => 640x480 Packed YUY2
FATAL: Cannot initialize video driver.
 
FATAL: Could not initialize video filters (-vf) or video output (-vo).
 
 
Exiting... (End of file)

So this was worrisome. I happened on the hint to try -vo gl and yup, it worked. Supposedly it makes for slower video so maybe on PCs where this trick is not required lag could be reduced.

mplayer personal preferences
I liked the idea of a window without a border (-noborder option) – so the only way to close it out is to kill the CMD window, which helps keep them in sync. Running two CMD windows doesn’t produce such good results!

I also wanted the window to first pop-up in the upper right corner of the screen, hence the -geometry 600:50

And I wanted the video screen to always be on top of other windows, hence the -ontop switch.

I decided the messages about cache were annoying and unimportant, hence the message suppression provided by the -msglevel all=0 switch.

Simultaneously recording and live streaming
I haven’t played with this too much, but I think the unix tee command works for this purpose. So you would take your raspivid line and make it something like:

raspivid -n -o – -t 9999999 -rot 180 -w 640 -h 480 -b 1000000 -fps 20|tee /home/pi/video-`date +%Y%h%d-%H%M`|nc -l 443

and you should get a nice date-and-time-stamped output file while still streaming live to your mplayer! Tee is an under-appreciated command…

Conclusion
I have tinkered with the Pi until I got its camera display to be screaming fast on my PC. I’ve shown how to do this and described some limitations.

Next Act?
I’m contemplating superimposing a grid with tick marks over the displayed video. This will help the robot driver establish their position relative to fixed elements on the field. This may be possible by integrating, for instance, openCV, for which there is some guidance out there. But I fear the real-time-ness may greatly suffer. I’ll post if I make any significant progress!
Update: I did get it to work, and the lag was an issue as suspected. Read about it here.

References and related
First Robotics is currently in season as I write this. The competition this year is Aerial Assist. More on that is at their web site, http://www3.usfirst.org/roboticsprograms/frc
Raspberry Pi camera quick start is a great place to get started for newbies.
Setting one or more static IP addresses on your Pi is documented here.
How not to set up your Pi for real-time video will be documented here.
How to get started on your Pi without a dedicated monitor is described here.
Finally, how to overlay a grid onto your video output (Yes, I succeeded to do it!) is documented here.
Correct source for smplayer for Windows.

Categories
Network Technologies Raspberry Pi

Making the Raspberry Pi camera look like an Axis ethernet camera

Intro
I can’t add much to this excellent guide:

http://blog.miguelgrinberg.com/post/how-to-build-and-run-mjpg-streamer-on-the-raspberry-pi

except a few customizations and suggestions.

I think we will be able to manage to make the Raspberry Pi + its camera behave like an Axis ethernet camera. This can be useful for First Robotics. But I haven’t proven it out yet, I’m just anticipating it can be done.

I will also mention there is a better way to get real-time true motion video (see the references) and there are sure a lot of ways to not even come close – I know because I tried a bunch of dead-ends before I hit on a good way to do this! I will try to share some of my failures so others can avoid things like vlc, motion, raspi-still, etc.

The details
The Axis camera (I think its model 206) sends output via MJPG (motion JPEG). The Raspberry Pi camera can be made to do the same, with a little tweak here and there.

For instance the mjpg-streamer’s default port is 8080, but you can change it to 80, just like the Axis camera.

Instead of

$ LD_LIBRARY_PATH=/usr/local/lib mjpg_streamer -i “input_file.so -f /tmp/stream -n pic.jpg” -o “output_http.so -w /usr/local/www”

do this:

$ LD_LIBRARY_PATH=/usr/local/lib mjpg_streamer -i “input_file.so -f /run/shm -n pic.jpg” -o “output_http.so -p 80 -w /usr/local/www”

You’d better make sure you don’t have an apache server or something else listening on port 80, however.

Our enemy – lag
Although this command provides some helpful insights into the efficient running of raspi-still:

$ raspistill –nopreview -w 640 -h 480 -q 5 -o /run/shm/pic.jpg -tl 100 -t 9999999 -th 0:0:0 &

it is not sufficient by itself to eliminate all lag, unfortunately. I think the -q switch is a big help, however. In my testing lag seems to be under a second. So, ok, but nothing to write home about. But it’s easy to make it worse than that….

I settled on this testing methodology to get more precise results about lag and frames per second (fps). I held my smartphone with its stopwatch app running next to the computer screen, with the Pi camera close and pointed at the phone. So in my field of view could see the actual phone plus the phone image from the Pi on the laptop. This test was very helpful in illuminating what is going on in fact.

No matter how many fps I requested (e.g., 10 fps by setting -tl 100) the best I can do is a frame every 0.6 seconds (1.6 fps). Because of the stopwatch app I know this pretty precisely! The other interesting thing is that contrary to what i thought prior to doing this more quantitative test, the lag actually isn’t all that bad! It’s maybe 0.2 s. What made the lag seem larger is that you often get “unlucky” and your motion seems delayed because there are so few frames per second repainting the screen.

To be continued…

Other bad approaches
Suuposedly, i was assured, true motion video can be achieved following this recipe. It uses a package called motion. There is no use of either raspistill or raspivid, which is probably a good thing. I have to yet try this out – I just learned about it. Apparently it also solves the lag problem, at least at a low frame rate. This turned out to be one of those dead ends for me. Yes it more-or-less works, but not in real-time and not providing smooth motion.

References
I finally achieved true motion video and documented it in great detail in this post.

Categories
Admin Network Technologies Raspberry Pi

Basic networking: creating a virtual IP in Debian Linux

Intro
A quick Internet search showed a couple top-level matches that didn’t quite work for me, so I’m documenting how I got my multiple IP assignments on one interface to stick. This was work done for my Raspberry Pi, but it should apply to any Debian Linux system.

The details
This was work done for my Raspberry Pi, but it should apply to any Debian Linux host. I made my file /etc/network/interfaces look as follows:

auto lo
auto eth0
auto eth0:0
 
iface lo inet loopback
# DrJ change: make IP static
# somewhat inspired by http://www.techiecorner.com/486/how-to-setup-static-ip-in-debian/ - DrJ 1/8/13
#iface eth0 inet dhcp
iface eth0 inet static
address 192.168.2.100
gateway  192.168.2.1
netmask 255.255.255.0
network 192.168.2.0
broadcast 192.168.2.255
 
# virtual IP on eth0
iface eth0:0 inet static
address 10.31.42.11
netmask 255.255.255.0
network 10.31.42.0
broadcast 10.31.42.255

I think the key statement which is missing on some people’s examples are the lines at the top of the file:

auto lo
auto eth0
auto eth0:0

When I didn’t have those I was finding that my primary IP was defined upon reboot, but not my virtual IP, although the virtual IP could be dynamically created with a simple

sudo ifup eth0:0

Still, I wanted it to survive a reboot and adding the auto lines did the trick.

Conclusion
A few of the pages you will find on the Internet may give incomplete information on how to configure virtual IPs in Debian Linux. The approach outlined above should work. Additional virtual IPs would just require sections like eth0:1, eth0:2, etc modelled after what was done for eth0:0

References
I present some basic information on one way to get started on the Pi without an external monitor (yes, it can be done) here!
If you think you like networking, you will learn a lot of useful tips in this posting which describes how to turn your Raspberry Pi into a full-blown router.

Categories
Network Technologies

How we got a little extra oomph from our firewall cluster, and why this trick no longer works for us

Intro
I was a running some Checkpoint firewalls in a cluster. In fact it’s been that way for years and years. At some point you get comfortable and forget to challenge and understand how it was set up. In this case re-examining the setup rewarded us with temporary survival as we were able to offload the primary member. Read on for the details…

The details
This firewall cluster included an active/standby pairing – a Nokia cluster with no state sync. The active firewall, an older model, was often hitting 99% or even 100% cpu usage on a daily basis. Dropped packets were correlated with these cpu spikes, and time-sensitive protocols, especially SIP used by IP phones, suffered mightily. Call quality often degraded, or the call was altogether dropped.

Some other relevant facts in this case: these firewalls were not doing NAT, they were acting more like routers with a firewall function. There are a handful of key servers behind them, like a VPN concentrator, a proxy, a Juniper ISG VPN concentrator, etc. On the external side was an Internet router, also under our control.

So the breakthrough was in revisiting what makes them active/passive in the first place. We weren’t relying on Checkpoint clustering. We used VRRP, defined through a Voyager setup. Then we set up our routing on all protected devices to use these VRRP IPs for their default routes. It all worked great until more and more usage crept in and then complaints started rolling in.

Upgrading costs $$ and the procurement cycle takes some time. What to do immediately, if anything?

The loudest complaints were from users of the Juniper ISG SSL VPN concentrators, who ran VOIP over those connections. What I realized (which of course is obvious in hindsight), is that this device could have all its traffic routed to the standby firewall where there was no cpu load whatsoever, and leave everything else on the active firewall.

How we did it
This was accomplished by adjusting the default route of the ISG to use the physical IP of the standby firewall, as opposed to the VRRP IP. Then, to avoid asymmetric routing, a host route was defined on the Internet router for this ISG, using as gateway the physical external IP of the standby firewall (again as opposed to the external VRRP IP.)

How it worked
It worked like a charm. We were well below our Internet link capacity, after all. So the master firewall was really the chokepoint for this voice traffic. Once we got it onto this unused firewall all the complaints stopped.

This is of course just a stop-gap measure because of course now we have no redundancy if we lose one of the firewalls. But meanwhile we’ve bought some time and kept the work-from-home users running smoothly. The master firewall still hits 99% cpu, but not quite as frequently. It’s difficult to find a true root cause, but an upgrade is definitely in order. Acceleration is already in place.

Why it won’t work for you – Checkpoint Cluster
Fast-forward five years and I tried this same trick which has served me well over the years. No worky. Why? Well these days we’ve switched to use of a Checkpoint Cluster with SYNC. In a Checkpoint cluster the secondary firewall will not forward traffic. In fact a firewall guy was the first to inform me of that. I didn’t believe him so I went ahead and configured it anyway. Sure enough, it simply didn’t work.

So for us, this trick has played itself out. But we used it multiple times during the five years it was available to us.

Conclusion
By re-visiting some old design principles were we able to get a little more mileage out of our firewalls and buy ourselves some time until we can do a planned upgrade.

Categories
Network Technologies

The IT Detective Agency: Why our forwarding vserver doesn’t route

Intro
F5 BigiP appliances are very versatile networking appliances. But sometimes you gotta know what you are doing!

The details
We set up a load-balanced Radius service using the same subnet for the radius servers as the load balancer itself. Setting up this service is moderately tricky. You have to set the default route of the Radius servers to be the load balancer, and on the load balancer SNAT (which I prefer to translate as “source NAT,” though technically it is “secure NAT”) and NAT should be disabled. And there are two services, AAA and audit (UDP ports 1812 and 1813).

So everything’s a bit different when all you'[re used to is creating load-balanced pools for web servers.

So with this setup an incoming packet comes in, its source is preserved, but its destination is NAT’d to the radius server by the load balancer. In the response, the source is the radius server. That gets NAT’d to the IP of the load-balanced service. So there are two stages for incoming request packets (pre- and -post NAT) and two for the responses. Here’s a trace which shows all this:

12:10:41.259073 IP drj-wlc-nausresea055-01.drj.com.filenet-rpc > radius.drj.com.radius: RADIUS, Access Request (1), id: 0x30 length: 260
12:10:41.259086 IP drj-wlc-nausresea055-01.drj.com.filenet-rpc > wusandradaa01.drjad.drj.net.radius: RADIUS, Access Request (1), id: 0x30 length: 260
12:10:41.259735 IP wusandradaa01.drjad.drj.net.radius > drj-wlc-nausresea055-01.drj.com.filenet-rpc: RADIUS, Access Reject (3), id: 0x30 length: 44
12:10:41.259745 IP radius.drj.com.radius > drj-wlc-nausresea055-01.drj.com.filenet-rpc: RADIUS, Access Reject (3), id: 0x30 length: 44

So all is good, right? Except that now we have a default route from the Radius server to the load balancer and so all response traffic is going through the load balancer, even things not related to Radius, such as a packets from an RDP session.

So we defined a forwarding_vserver to make the BigIP act as a router:

A forwarding vserver is a virtual server of type Forwarding (IP). In the bigip.conf file it looks like this:

virtual forwarding_vserver {
   ip forward
   destination any:any
   mask 0.0.0.0
   profiles route_friendly_fastL4 {}
}

But it doesn’t work! Packets from the Radius server come to the load balancer, and then they get source NAT’d to the floating self-IP of the load balancer. That’s no good. In TCP your response packets have to come from the IP you connected to! For simple PINGs you kind of get away with it, but with a warning. In TCP your PC will send a RST (reset) packet every time it gets a response packet with the wrong source IP, even if the other information is correct.

The solution
With the help of someone who understands snat auto-maps better than I do (evidently), I got the tip that I have a global snat-automap enabled, which is doing this. That’s how I’ve always run these LTMs (Local Traffic Managers). I had forgotten why or how I did it. Well the snat-automap pretty mcuh applies to all my other load-balanced services so I can’t simply chuck it. And I don’t have another subnet handy for use so I can’t simply exclude one of my vlans. They suggested that it could be turned off on my forwarding_vserver with an irule! Who would have figured? So I created a very simple irule:

# Turn off snat, i.e., for us in our forwarding_vserver
# inspired by https://devcentral.f5.com/wiki/iRules.snat.ashx
# DrJ, 11/2013
when CLIENT_ACCEPTED {
         snat none
}

and applied it to my forwarding_vserver, which now looks like this:

virtual forwarding_vserver {
   ip forward
   destination any:any
   mask 0.0.0.0
   rules snat-none
   profiles route_friendly_fastL4 {}
}

And voila, the LTM now routes those packets correctly without any address translation! And the Radius service still does its translations as desired.

Case closed!

Conclusion
We learned a little about F5 BigIPs today. The frustrating thing about the documentation is that they don’t really cover actual use cases so they introduce configuration settings without fully explaining why you would want to use them.

For the curious, the forwarding_vserver is accomodating an asymmetric routing situation. Incoming RDP (remote desktop protocol) packets get sent directly from a Cisco router to the Radius server. It’s just the response packets that flow from the Radius server, to the LTM, to the Cisco router.

References and related
In this post I show why a basic virtual server might not be working – a kind of rookie mistake we’ve all probably made at some point!
This post shows some non-trivial iRule examples.

Categories
Admin Network Technologies

Are we in Brazil? We are NOT in Brazil.

But Google thinks we are:

$ curl -i http://www.google.com/

returns:

HTTP/1.1 302 Found
Location: http://www.google.com.br/?gws_rd=cr&ei=nrA4UoPZB8fb4AP3loGYAg
Cache-Control: private
Content-Type: text/html; charset=UTF-8
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Date: Tue, 17 Sep 2013 19:42:22 GMT
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Alternate-Protocol: 80:quic
Content-Length: 262
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
Set-Cookie: PREF=ID=9ed4d9e91767f4ce:FF=0:TM=1379446942:LM=1379446942:S=O9jxL9sXRCc0kF-E; expires=Thu, 17-Sep-2015 19:42:22 GMT; path=/; domain=.google.com
Set-Cookie: NID=67=BJA1pj-o2tu-IDI9KS5MjQvoKPJoY8x3uREAmeGCItGKxxfovFqdjPhvBaHUHcISFLVcTWSmCXwiTAuVhF4DIYCbPcuubfBBEGYaNy2wgveeyvGj35xTzM4Oo-yCLaDe; expires=Wed, 19-Mar-2014 19:42:22 GMT; path=/; domain=.google.com; HttpOnly
 
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.com.br/?gws_rd=cr&amp;ei=nrA4UoPZB8fb4AP3loGYAg">here</A>.
</BODY></HTML>

and MSN is similar:

$ curl -i http://www.msn.com/

HTTP/1.1 302 Found
Cache-Control: no-cache, no-store
Pragma: no-cache
Content-Type: application/xhtml+xml; charset=utf-8
Location: http://br.msn.com/?rd=1&ucc=BR&dcc=BR&opt=0
P3P: CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND"
X-AspNet-Version: 4.0.30319
S: BN2SCH030301048
Edge-control: no-store
Date: Tue, 17 Sep 2013 20:02:42 GMT
Content-Length: 172
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
Set-Cookie: MC1=V=3&GUID=7173a7e9bb5444fc88d52c3d302f0cdf; domain=.msn.com; expires=Thu, 17-Sep-2015 20:02:42 GMT; path=/
Set-Cookie: MC1=V=3&GUID=7173a7e9bb5444fc88d52c3d302f0cdf; domain=.msn.com; expires=Thu, 17-Sep-2015 20:02:42 GMT; path=/
Set-Cookie: brdSample=89; domain=.msn.com; expires=Thu, 17-Sep-2015 20:02:42 GMT; path=/
blah, blah
 
 
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="http://br.msn.com/?rd=1&amp;ucc=BR&amp;dcc=BR&amp;opt=0">here</a>.</h2>
</body></html>

Now a request from the IP Next door to it, on the same subnet, produces an entirely different result:

$ curl -i http://www.google.com/

Date: Tue, 17 Sep 2013 19:48:11 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more in
fo."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Alternate-Protocol: 80:quic
Transfer-Encoding: chunked
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
Set-Cookie: PREF=ID=0e280a6247dbe89f:FF=0:TM=1379447291:LM=1379447291:S=Ws_EoUItgwgQMFLL; expires=Thu, 17-Sep-2015 19:48:11
 GMT; path=/; domain=.google.com
Set-Cookie: NID=67=MEOnbrr2pgKz8MVKEzIQ2v9f4nkR0o5FXJxbeBRk3GZg6GsfKNRZm9UEHTzcuRF5A6D2kXKa4N6FQnP88fkVLrgDokdOlXvX1Oba2JzC
koZ0K0PiACYSiTPCru5eH9C3; expires=Wed, 19-Mar-2014 19:48:11 GMT; path=/; domain=.google.com; HttpOnly
 
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage"><head><meta content="Search the world's information,
 including webpages, images, videos and more. Google has many special features
...

This is a problem when you do searches in which you expect to get localized results, like a search for dinner. Believe me, you see completely different things if Google thinks your IP is in Brazil.

And some services simply shut themselves off. Pandora simply does not work, for instance.

What is going on?

It seems some Geo-location service has incorrectly tagged our IP as Brazil. Not sure how to fix that…

Google does have an IP location change form, but they say it will take a month for them to make the change. That hardly seems like Internet time! https://support.google.com/websearch/contact/ip And what about the others? I don’t see any simple fix for Pandora for instance.

Possible solution?
Someone suggested that the proxy has cached Google’s home page. This is one thing I didn’t test for. However, these proxies are very sophisticated and know to refresh popular pages frequently so I very much doubt that was the cause of the problem.

Workarounds
Google present a link Google.com in the lower right corner of the page which you can click on to pop you to the English-language version of the site, but it’s still not location-aware and treats you more as an English traveler in Brazil when you search for a generic term like restaurant.

You can also go to http://www.google.com/ncr. I personally haven’t tried it, but that’s a tip I just received.

Oct 30 update
Well, finally, after filling out the IP change form twice, and of course keeping all Brazilian users off that proxy, Google has finally deemed fit to declare our proxy really is in the US after all.

It happens again…
Some light usage by a fraction of the user base and I find all proxies are in Brazil yet again. I fill out the form March 19th 2015 and a couple times after that. Now it is May 11th and Google still has not acted. This time I have not kept Brazil users off the proxies, because really this is ridiculous. When users complain I urge them to use duckduckgo.com for its better privacy protections. June 24th update. I’ve been checking every week just about. I’ve filled out Google’s own form about four times in total. So finally after three months, Google set us back to US Enlgish. Without a word to us, of course. Incredible.

Google has gotten big and unresponsive.

Conclusion
Use duckduckgo.com.

References and related
A good web site to learn where your IP is without too much malvertising that you often get with these kinds of things is: https://ipinfo.io/

Categories
Admin Network Technologies

Extended Passive Mode FTP through Checkpoint Firewall

Intro
The vast majority of time there is no problem doing an FTP to a server behind a firewall protected by Checkpoint’s Firewall-1. But occasionally there is.

The details
The problem I am about to document I think will only occur on a server that has multiple interfaces. I have seen it occur on multiple operating systems, so that doesn’t seem to matter. On the other hand, I have also not seen it on other similar systems, a point which I don’t fully yet understand.

Nevertheless, a work-around is always appreciated, so I provide what I found here, to complete my extensive documentation of problems I’ve encountered and resolved.

Here is a snippet from the FTP session showing the problem:

ftp> cd uploadDirectory
250 CWD command successful.
ftp> put smallfile.txt
local: smallfile.txt remote: smallfile.txt
229 Entering Extended Passive Mode (|||36945|)
200 EPRT command successful.
421 Service not available, remote server timed out. Connection closed

And here is the solution:

Enter epsv4 after logon and before any other commands are issued. Problem fixed!

Conclusion
We have shown a way to fix a firewall-related problem that manifests itself during extended passive mode FTPs. Some more research should be done to understand under what circumstances this problem should be expected, but it seems to occur with a Checkpoint Firewall-1 firewall and an FTP server with multiple interfaces.

Categories
Admin Network Technologies

The IT Detective Agency: trouble with wireless at home

Intro
I don’t usually have the luxury of writing about a mystery I’ve solved right out of my own home, but there finally is one that I got got to the bottom of recently – poor WiFi performance.

The details
Considering that I deal with this stuff for a living, I have a thread-bare setup at home. After my company-issued router’s WiFi began to work unreliably, I resuscitated an old Linksys wireless router, WRK54G V2. Superficially it seemed to work. But we weren’t very demanding of it.

It eventually seemed to be the case, as visitors mentioned, that streaming videos does not work through wireless. This was hard for me to check, with my broken-down, aging equipment. I have a desktop which always freezes and crashes is you play any Youtube video. And a Netbook which kind of worked better, but its peculiarity is that its ethernet interface doesn’t work. Wirelessly, its version of Flash was too old and insecure for Firefox, and attempts to update Flash using WiFi in turn were unsuccessful.

In general the Linksys router, as I eventually realized, seemed to initially serve up large downloads ok, but then at some point during the download, things begin to crawl and you are left with a download that proceeds at 10 kbit/s or something ridiculously slow like that.

Providing mixed evidence is a Sony BlueRay player. using WiFi it could sort of manage to show a HuluPlus TV episode. You might have to be patient at times while it’s loading, but we did get through a full episode of Grey’s Anatomy recently.

After more complaints I decided enough is enough. It seemed as though my WiFi was the most likely suspect, sifting through the mixed evidence. I perhaps waited so long because who’d think they’d be dealing with two bad WiFi routers from two totally different vendors?

So hedging my bets, I didn’t go all out with a new Gbit router. I reached back in time a little and got a refurbished Cisco 1200E wireless-N router. It was only $28 from Amazon. But before buying it, I read the comments and got one idea about routers: sometimes they need to be rebooted!

This is pretty funny, really, because it is probably apparent to any homeowner, and here I am, a specialist, missing this point. You see with Cisco enterprise-class gear you almost never have to reboot to fix a problem. These things run uninterrupted for not only weeks and months at a time, years at a time is also not at all uncommon. Same for some Unix servers. So from my perspective rebooting is something for consumer devices running Microsoft OSes!

So, before rebooting the Linksys to see if that would cure it, I ran a Ping to Google’s DNS server (very easy to remember its IP) from a CDM window:

> ping -t 8.8.8.8

I didn’t preserve the output, but it wasn’t pretty. It was something like this:

Pinging 8.8.8.8 with 32 bytes of data:
Reply from 8.8.8.8: bytes=32 time=51ms TTL=56
Reply from 8.8.8.8: bytes=32 time=369ms TTL=56
Reply from 8.8.8.8: bytes=32 time=51ms TTL=56
Reply from 8.8.8.8: bytes=32 time=1204ms TTL=56
Reply from 8.8.8.8: bytes=32 time=284ms TTL=56
...

51 msec – fine. But round-trip times much greater than that? That’s not right.

So I hopefully reboot the Linksys router and re-run the test on the Netbook:

Pinging 8.8.8.8 with 32 bytes of data:
Reply from 8.8.8.8: bytes=32 time=51ms TTL=56
Reply from 8.8.8.8: bytes=32 time=51ms TTL=56
Reply from 8.8.8.8: bytes=32 time=51ms TTL=56
Reply from 8.8.8.8: bytes=32 time=50ms TTL=56
Reply from 8.8.8.8: bytes=32 time=51ms TTL=56
...

Much more consistent.

Try a Youtube video from Firefox. Nope, need to update Flash. Update Flash. Nope – download times out and kicks me out.

So I’ve accomplished nothing in rebooting in terms of results that matter.

That’s when I decided to check out of Amazon with that refurbished router.

Aside about Wireless-N
Given my ancient equipment, I was concerned that Wireles-N routers might not be compatible with my wireless radios, which would only support G. Is it backwards compatible? Yes. Some quick research showed that and my own experience confirmed it.

Conclusion
The setup of the router was pretty straightforward although it froze at some point just after I set the wireless password. It helps to have done this a zillion times before. At that point I observed what my default gateway was and hit it as a web site URL. Guessed the admin password incorrectly a zillion times, until I tried the wireless password as the admin password, and, wham – I was in and happily configuring away…

More importantly, I went to that Netbook, updated Flash. No problems. Ran a Youtube video. No problems. Ran a speedtest.net test (which wouldn’t even run before this). Numbers look as good as my wired connection: 6 mbit download, 0.6 mbit upload.

Last test is to see where the speed maxes out within my home network. I plan to hit my Raspberry Pi web server to test this and will provide results as soon as they are available.

Conclusion to the conclusion
So I really was cursed with two bad wireless routers. Sometimes using 10-year-old equipment is really not worth the $30 saved in deferred spending. Read product reviews on Amazon to get hints about real issues others have faced.
To be continued…

Categories
Apache Linux Network Technologies Perl Raspberry Pi

Getting started on my Raspberry Pi

Intro
The Raspberry Pi computer is an awesome idea. Its performance is surprisingly good as well, as I will show below. Available packages? Not so impressive. I share some old X-windows tricks which will allow you to bring up the GUI without ever using the HDMI port.

The details
My Methodology
I was too lazy to set up an HDMI console plus keyboard and mouse. I’m more a server guy anyways so I’m more interested in what I can accomplish from a command prompt. And this also makes getting started that much easier. I had burned the Raspbian Wheezy image to a super-fast SD card (more on that below) the day that my Pi came in the mail. I attached power and ethernet, booted it up, guessed the IP it acquired by running some PINGs, did an ssh using the pi/raspberry user and Bingo! I was in. It couldn’t be easier. How I tested GUI applications without a console is explained further down below.

First Impressions
It feels fast.

Packages
Not much seems to be there by default – no apache, not many X utilities. There is a lame X browser called x-www-browser. I thought this is Debian, right? So we can just start downloading Debian packages, like Firefox. Wrong! It doesn’t work that way. There’s no Firefox, Safari, Chrome or Opera! It does come pre-loaded with curl, however, ha, ha.

No, the Raspbian FAQ explains why this is. It’s rather complicated. I guess the compiler works though I haven’t tested it yet. So I suppose you could compile packages from their source code.

The x-terminal-emulator is pretty decent, however.

If it comes with a web server, I didn’t notice. So I quickly checked for the availability of apache. It’s available. Then installed it:

> sudo apt-get install apache2

That worked out well. It installed it and the packages it depended on and even launched it, and it all felt fairly peppy. See the suggested fix further down if this gives you errors. The default HTML DOCROOT is /var/www. I accessed it locally:

> curl localhost

And a welcome message displayed. A good start.

Where’s the rest of my 16 GB SD card gone to?

Original disk layout:

pi@raspberrypi:~$ df -k
Filesystem     1K-blocks    Used Available Use% Mounted on
rootfs           1804128 1492908    219572  88% /
/dev/root        1804128 1492908    219572  88% /
devtmpfs          224436       0    224436   0% /dev
tmpfs              44900     204     44696   1% /run
tmpfs               5120       0      5120   0% /run/lock
tmpfs              89780       0     89780   0% /run/shm
/dev/mmcblk0p1     57288   16872     40416  30% /boot

Layout after raspi-config:

pi@raspberrypi:~$ df -k
Filesystem     1K-blocks    Used Available Use% Mounted on
rootfs          15251960 1494852  12982544  11% /
/dev/root       15251960 1494852  12982544  11% /
devtmpfs          224436       0    224436   0% /dev
tmpfs              44900     196     44704   1% /run
tmpfs               5120       0      5120   0% /run/lock
tmpfs              89780       0     89780   0% /run/shm
/dev/mmcblk0p1     57288   16872     40416  30% /boot

Whew! That was easy. All 16 GB accounted for and actively used.

Was it worth it to buy that UHS SD card?
I didn’t want a sluggish server, so I paid a couple bucks more and bought a 16 GB SD UHS (ultra high speed) card for my “disk,” not knowing whether or not the Pi had the muscle to put it to work.

A quick aside about SD cards
I did a quickie self-education on this topic. Most SD cards are rated by class, so a class 4 SD card can do 4 MB/sec I/O, and a class 10 card can do at least 10 MB/sec. Faster still are the UHS SD cards. My Sandisk, which only cost about $19, is rated for 45 MB/sec I/O. A great write-up on this topic specifically for Raspberry Pi is: Raspberry Pi SD Card Speed Test – Raspberry Pi

diskSpec.pl benchmark (higher numbers are better)
1333 file creation/destruction operations per second – Raspberry Pi with UHS SD card
6666 file creation/destruction operations per second – EBS volume on small image running CentOS in Amazon cloud
26000 file creation/destruction operations per second – high-end HP server (G7 DL380) running SLES 11

I think I provided the source for this simple Perl program I wrote, diskSpec.pl. It creates a file, writes a random number into it, then deletes it – that all counts as one operation. Here it is:

#!/usr/bin/perl
# DrJ, 1/2000
# Test disk I/O
$DIR = $ARGV[0];
chdir($DIR);
$t0 = time();
while(1) {
  $ran = rand();
  open(FILE,"&gt;$ran") || die "Cannot open file $ran in directory $DIR!!\n";
  print FILE $ran;
  close(FILE);
  unlink($ran);
  $cnt++;
  if ($cnt % 20000 == 0) {
    $rate = $cnt / (time() - $t0) ;
    print "File creation/desctruction rate: $rate\n";
  }
}

DrJ 2017 Note: The notes below are historical and does not seem to work at all for the Raspberry Pi 3 loaded with NOOBS. In NOOBS you select your OS to install. You can’t ssh to it. I know. I just tried! Even after you install Raspbian Wheezy, you still can’t access it via ssh until you enable the ssh daemon with raspi-condfig.

How to get the GUI working without a console
I have this feeling that many people trying out the Pi won’t have the faintest idea how X windows works, unlike us Unix old-timers. It’s fun to put 20-year-old lessons to work on something new. Like I said I’m lazy and didn’t feel the need to set up an actual console to the thing. I used some old X features to allow me to launch specific X-windows applications that are pre-loaded on the device, and display them on my PC. How?

On a Windows PC you install Cygwin. Then launch the XWin Server. You ssh to your pi. How do you know its IP the first time? Guess! It picks it up via DHCP, so start PINGing around the range where your other devices are numbered. My PC is 192.168.5.12/24, my pi was 192.168.5.16. Maybe you have a bunch of devices responding to PING and are unsure which is which? Your MAC table is your friend. Here’s mine:

C:\Documents and Settings&gt;arp -a
 
Interface: 192.168.5.12 --- 0x2
  Internet Address      Physical Address      Type
  192.168.5.1           00-14-f6-e0-c0-4c     dynamic
  192.168.5.16          b8-27-eb-dd-21-02     dynamic
  192.168.5.99          00-90-a9-bb-3d-76     dynamic

arp displays the MAC table with the IP-to-physical (MAC) address correspondence. So most Pi’s will have a MAC address whose beginning is similar to b8-27-eb. A quick aside. Does the MAC address follow the board (SOC) or the SD Card? The board – I tested this with a friend’s SD Card.

You login with the pi/raspberry.

Then set your DISPLAY environment variable:

> export DISPLAY=192.168.5.12:0

Most of your X applications begin with the letter “x,” so enter

> x<tab><tab>

to see a display of available programs like this:

xapian-config        xdg-screensaver      xkbevd               xpdf.real            xxd
xarchiver            xdg-settings         xkbprint             xprop                xz
xargs                xdpyinfo             xkbvleds             x-session-manager    xzcat
xauth                xdriinfo             xkbwatch             xsubpp               xzcmp
xdg-desktop-icon     xev                  xkill                xtables-multi        xzdiff
xdg-desktop-menu     xfd                  xlsatoms             x-terminal-emulator  xzegrep
xdg-email            xfontsel             xlsclients           xvinfo               xzfgrep
xdg-icon-resource    xinit                xlsfonts             x-window-manager     xzgrep
xdg-mime             xkbbell              xmessage             xwininfo             xzless
xdg-open             xkbcomp              xpdf                 x-www-browser        xzmore

Actually I don’t know how many of these are X. But at least a few are.

Start an xterm in Cygwin. In the xterm window, give permission to the Pi to use it as its Xserver:

> xhost +

Now in the Pi shell (ha, ha), type:

> x-terminal-emulator

and you should see the colorful terminal emulator on your PC in a few seconds. this is a true GUI application. You similarly launch the x-www-browser. Don’t forget to background your X-windows in the Pi shell:

<Ctrl-Z>
> bg

so you can use the one window to launch multiple X windows.

Another example the book Programming the Raspberry Pi has is the Python interactive development environment. I reasoned from the screen shots that idles3 would also be an X application – hey, they don’t have to start with the letter x – and indeed it is!

Want the whole ball of wax, a complete console? I just figured this one out by taking an educated guess:

> x-session-manager

and you will see the complete GUI on your PC! Cool, huh?

Want to get rid of the last thing you backgrounded, like, say, that x-session-mnager which has taken over your PC?! Type

> fg
<Ctrl-C>

and it will be killed.

How to get the GUI working without a console, Method 2
The above steps look a little daunting? Even I don’t want to install cygwin on my new PC. There is an alternative which can suffice for light usage.

On the Pi install a vnc server:

$ sudo apt-get install tightvncserver

Launch it:

$ vncserver

The first time only it will ask you to set up a password. Might as well make it raspberry like everything else we do on the Pi.

Then install a VNC client on your PC (Or Macbook). I use RealVNC.

Launch your VNC client and connect to your Pi’s IP address (which you need to know) + the display number, like this:

192.168.0.100:1

For a Pi at IP 192.168.0.100 in which the vncserver started display 1. Normally it will be display 1, but I guess it might be display 0.

Don’t launch vncserver more than once! You don’t want a bunch of those running and dragging on performance.

Anyways, that’s it! You should see the Pi’s GUI on your PC, but it might seem a wee bit small.

Setting a static IP
If you’re going to use the Pi more as a server like I am, I think it’s a good idea to give it a static IP. What I did is to edit /etc/network/interfaces. Mine now looks like this:

Nagios can be installed! That's pretty cool - it's a sophisticated network monitoring utility.

Get a decent browser
The web browsers that come with the Pi are horrible. Midori? Seriously? I found you can get Firefox, but the downside is that it’s sloooww. But at least it works. The secret is that it’s not called Firefox. Instead:

$ sudo apt-get install iceweasel

Yes, it’s iceweasel, not Firefox, in Debian Linux. Go figure.

My cool transparent case
I recommend to get a case. I got the one with the best reviews. It’s kind of expensive, about $20, but worth it. It’s practically a work-of-art. Clear, the PC board fits snugly. I put it in my pocket and showed it around to my friends, feeling it was well protected, and yet also a sight to behold the first time. I even has a thoughtful light guide so the LEDs look beautiful as their light follows the rectangular opening to open air. I never had this much fun in show-and-tell! I just pulled the Pi wrapped in its case from my shirt pocket and amazed those around me. So go ahead and splurge. Anyways some of the cheaper cases look just that. Here is what I bought:

Helping a friend out with his Pi
So I dutifully take my friend’s Pi home and offer to install a web server. What did I do wrong? Well, duh, I could have just taken his SD Card home and plugged it into my Pi case! That concept takes some getting used to! We all have the same hardware. Our SD cards – our disk – are what make one Pi different from another.

So I followed my own blog post to recall some things. This Pi also had a MAC address beginning with the same six characters.

The apache2 installation did not work out, however. What to do? Well, I eventually read the darn output from running it. It suggests to try this:

> sudo apt-get update

So I ran that, figuring it could do no harm. Then I re-ran

> sudo apt-get install apache2

and this time the install actually worked!

Reading a flash drive
I was curious to see if you could stick a flash drive in the thing and just read it. I didn’t think so since I thought it would be formatted for NTFS. But if you have the GUI running and bring up a file manager, I’ll be darned if it doesn’t just work. I noticed the drive is mounted as /media/Cruzer (my flash drive has the brand name Cruzer).

If you don’t launch the file manager, I think you can still work with it as follows:

$ sudo mkdir /media/Cruzer; sudo mount /dev/sda1 /media/Cruzer

Then when you’re all done and before you remove it:

$ sudo umount /media/Cruzer

So that’s pretty cool. You can create tar archives on the flash drive, plug it into someone else’s Pi and untar it, etc, just like on Windows.

Conclusion
Raspberry Pi is respectable as a computer. It will be a lot of fun to explore for the hobbyist.

References

Raspberry Pi SD Card Speed Test – Raspberry Pi – a great discussion of the various speeds of Micro SD cards and how to measure yours
Go here for my next project – using your Raspberry Pi to monitor your home’s power or Internet connection.
Interested in networking? A lot of useful tips can be found in this posting describing how to turn your Pi into a router.
Realvnc.com distributes realVNC viewers for various platforms.
How about a Raspberry Pi-driven digital photo frame? I describe an approach in this article.
Brief Nagios for Raspberry Pi writeup.

Categories
Admin Network Technologies

The IT Detective Agency: Internet Explorer cannot display https web page, part II

Intro
They say when it rains it pours. As a harassed It support specialist its tempting to lump all similar-looking problems that come across your desk at the same time in the same bucket. This case shows the shallowness of that way of thinking, for that’s exactly what happened in this case. It occurred exactly when this case was occurring in which a user had an issue displaying a secure web site. That other case was described here.

The details
I was doing some work for a company when they came to me with a problem one user at HQ was having accessing an https web site. Everything else worked fine. The same web page worked fine from other sites.

Since I had helped them set up their proxy services I was keen to prove that the proxy wasn’t at fault. The user removed the proxy settings – still the error occurred. The desktop support got involved. I suggested a whole battery of tests because this was a weird one.
– what if you access via proxy?
– what if you take that laptop and access it from a VPN connection?
– what if you access another site which uses the same certificate-issuer?
– what if you access the host, but using http?
– can you PING the server?
– can you telnet to that server on port 443?
– what if you access the webpage by IP address?
– what if accessed via Firefox?
– etc.
The error, by the way, was

Internet Explorer cannot display the webpage

What you can try:
Diagnose Connection Problems

The answers came back like this:
– what if you access via proxy? It works!
– what if you take that laptop and access it from a VPN connection? It works.
– what if you access another site which uses the same certificate-issuer? It works.
– what if you access the host, but using http? It works.
– can you PING the server? Yes.
– can you telnet to that server on port 443? Yes
– what if you access the webpage by IP address? It does not work.
– Firefox? I don’t remember the answer to this.

Most of that thinking behind those tests is pretty obvious. Why so many tests? The networking group is kind of crotchety and understaffed, so they really wanted to eliminate all other possibilities first. And at one point I thought it could have been a desktop issue. Why would the network allow some of these packets through but not others when it doesn’t run a firewall? The desktop did have A-V and local firewall, after all.

So I didn’t have proof or even a good idea. Time to take out the big guns. We ran a trace on the server with tcpdump while the error occurred. The server was busy so we had to be pretty specific:

> tcpdump -s 1580 -w /tmp/drjcap.cap -i any host 10.19.79.216 and port 443

What do we see? We see the initial handshake go through just fine. Then a client Hello, then a server Hello, followed by a Certificate sent from the server to the client. The Certificate packet kept getting re-sent because there was no ACK to it from the client. So it’s beginning to smell like a dropped packet somewhere. I was hot on the trail but I decided I needed even stronger proof. We arranged to do simultaneous traces on both PC and server to compare the two results.

On the PC we had to install Wireshark. Actually I think Microsoft also has a utility to do traces but I’ve never used it. What did we find? Proof that there was indeed a dropped packet. That server certificate? Never received by the PC (which is acting as the SSL client). But why?

I had noticed one other funny thing about the trace comparisons. The server hello packet left as a packet of length 1518 bytes, but arrived, apparently, as two packets, one of length 778 bytes, the other of 770 bytes, i.e., it was fragmented. That should be OK. I don’t think the don’t fragment flag was set (have to check this). But it got me to wondering. Because the server certificate packet was also on the large side – 1460 bytes.

Regardless, I had enough ammo to go to the networking group with, which I did. It was like, “Oh yeah, our Telecom provider (let’s call it “CU”) implemented GETVPN around that time.” And further discussion revealed suspected other problems related to this change with MTU, etc. In fact they dredged up this nice description of the pitfalls that await GET VPN implementations from the Cisco deployment guide:

3.8 Designing Around MTU Issues Because of additional IPsec overhead added to each packet, MTU related issues are very common in IPsec deployments, and MTU size becomes a very important design consideration. If MTU value is not carefully selected by either predefining the MTU value on the end hosts or by dynamically setting it using PMTU discovery, the network performance will be impacted because of fragmentation and reassembly. In the worst case, the user applications will not work because network devices might not be able to handle the large packets and are unable to fragment them because of the df-bit setting. Some of the scenarios which can adversely affect traffic in a GET VPN environment and applicable mitigation techniques are discussed below. LAN MTU of 1500 – WAN MTU 44xx (MPLS) In this scenario, even after adding the 50-60 byte overhead, MTU size is much less than the MTU of the WAN. The MTU does not affect GET VPN traffic in any shape or form. LAN MTU of 1500 – WAN MTU 1500 In this scenario, when IPsec overhead is added to the maximum packet size the LAN can handle (i.e. 1500 bytes) the resulting packet size becomes greater than the MTU of the WAN. The following techniques could help reduce the MTU size to a value that the WAN infrastructure can actually handle. Manually setting a lower MTU on the hosts By manually setting the host MTU to 1400 bytes, IP packets coming in on the LAN segment will always have 100 extra bytes for encryption overhead. This is the easiest solution to the MTU issues but is harder to deploy because the MTU needs to be tweaked on all the hosts. TCP Traffic Configure ip tcp adjst-mss 1360 on GM LAN interface. This command will ensure that resulting IP packet on the LAN segment is less than 1400 bytes thereby providing 100 bytes for any overhead. If the maximum MTU is lowered by other links in the core (e.g. some other type of tunneling such as GRE is used in the core), the adjust-mss value can be lowered further. This value only affects TCP traffic and has no bearing on the UDP traffic. 3.8.1.1 Host compliant with PMTU discovery For non-TCP traffic, for a 1500 packet with DF bit set, the GM drops the packet and send ICMP message back to sender notifying it to adjust the MTU. If sender and the application is PMTU compliant, this will result in a packet size which can successfully be handled by WAN. For example, if a GM receives a 1500 byte IP packet with the df-bit set and encryption overhead is 60 bytes, GM will notify the sender to reduce the MTU size to 1440 bytes. Sender will comply with the request and the resulting WAN packets will be exactly 1500 bytes

I don’t claim to understand all those scenarios, but it shouts pretty loudly. Watch out for problems with packets of size 1400 bytes or larger!

Why would they want GET VPN? To encrypt the HQ communication over the WAN. So the idea is laudable, but the execution lacking.

Case: understood but not yet closed! The fix is not yet in…

To be continued…