Intro
I’ve done a few things to do some vision processing with OpenCV on a Raspberry Pi 3. I am a rank amateur so my meager efforts will not be of much help to anyone else. My idea is that maybe this could be used on an FRC First Robotics team’s robot. Hence I will be getting into some tangential areas where I am more comfortable.
Even though this is a work in progress I wanted to get some of it down before I forget what I’ve done so far!
Tangential Stuff
Disable WiFi
You shouldn’t have peripheral devices with WiFi enabled. Raspeberry Pi 3 comes with built-in WiFi. Here’s how to turn it off.
Add the following line to your /boot/config.txt file:
dtoverlay=pi3‐disable‐wifi
Reboot.
If it worked you should only see the loopback and eth0 interefaces in response to the ip link command, something like this:
$ ip link
1: lo:
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0:
link/ether b8:27:eb:3f:92:f3 brd ff:ff:ff:ff:ff:ff
Hardcode an IP address the simple-minded way
On a lark I decided to try the old-fashioned method I first used on Sun Solaris, or was it even Dec Ultrix? That is, ifconfig. I thought it was to be deprecated but it works well enough for my purpose.
So something like
$ sudo ifconfig eth0 192.168.1.160
does the job, as long as the network interface is up and connected.
Autolaunch a VNC Server so we can haul the camera image back to the driver station
$ vncserver &hypher;geometry 640×480 ‐Authentication=VncAuth :1
Launch our python-based opencv program and send output to VNC virtual display
$ export DISPLAY=:1
$ /home/pi/.virtualenvs/cv/bin/python green.py > /tmp/green.log 2>&1 &
The above was just illustrative. What I actually have is a single script, launcher.sh which puts it all together. Here it is.
#!/bin/sh # DrJ sleep 2 # set a hard-wired IP - this will have to change!!! sudo ifconfig eth0 192.168.1.160 # launch small virtual vncserver on DISPLAY 1 vncserver -Authentication=VncAuth :1 # launch UDP server $HOME/server.py > /tmp/server.log 2>&1 & # run virtual env cd $HOME # don't need virtualenv if we use this version of python... #. /home/pi/.profile #workon cv # # now launch our python video capture program # export DISPLAY=:1 /home/pi/.virtualenvs/cv/bin/python green.py > /tmp/green.log 2>&1 & |
OpenCV (open computer Vision)
opencv is a bear and you have to really work to get it onto a Pi 3. There is no apt-get install opencv. You have to download and compile the thing. There are many steps and few accurate documentation sources on the Internet as of this writing (January 2018).
I think this guide by Adrian is the best guide:
However I believe I still ran into trouble and I needed this cmake command in stead of the one he provides:
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D INSTALL_C_EXAMPLES=OFF \
-D ENABLE_PRECOMPILED_HEADERS=OFF \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib-3.1.0/modules \
-D BUILD_EXAMPLES=ON .. |
I also replaced opencv references to version 3.0.0 with 3.1.0.
I also don’t think I got make -j4 to work. Just plain make.
An interesting getting started tutorial on images, opencv, and python:
Simplifying launch of VNC Viewer
I wrote a simple-minded DOS script which launches UltraVNC with a password. So with a double-click it should work).
Here’s a Dos .bat file to launch ultravnc viewer by double-clicking on it.
if not "%minimized%"=="" goto :minimized set minimized=true start /min cmd /C "%~dpnx0" goto :EOF :minimized c:\apps\ultravnc\vncviewer -password raspberry 192.168.1.160:1 |
I’m sure there’s a better way but I don’t know it.
The setup
We have a USB camera plugged into the Pi.
A green disc LED light.
A green filter over the camera lens.
A target with two parallel strips of retro-reflective tape we are trying to suss out from everything else.
Some sliders to control the sensitivity of our color matching.
The request to analyze the video in opencv as well as display it on the driver station.
Have opencv calculate the pixel distance (“correction”) from image center of the “target” (the two parallel strips).
Send this correction via a UDP server to any client who wants to know the correction.
Here is our current python program green.py which does these things.
import Tkinter as tk from threading import Thread,Event from multiprocessing import Array from ctypes import c_int32 import cv2 import numpy as np import sys #from Tkinter import * #cap = cv2.VideoCapture(0) global x global f x = 1 y = 1 f = "green.txt" class CaptureController(tk.Frame): NSLIDERS = 7 def __init__(self,parent): tk.Frame.__init__(self) self.parent = parent # create a synchronised array that other threads will read from self.ar = Array(c_int32,self.NSLIDERS) # create NSLIDERS Scale widgets self.sliders = [] for ii in range(self.NSLIDERS): # through the command parameter we ensure that the widget updates the sync'd array s = tk.Scale(self, from_=0, to=255, length=650, orient=tk.HORIZONTAL, command=lambda pos,ii=ii:self.update_slider(ii,pos)) if ii == 0: s.set(0) #green min elif ii == 1: s.set(0) elif ii == 2: s.set(250) elif ii == 3: s.set(3) #green max elif ii == 4: s.set(255) elif ii == 5: s.set(255) elif ii == 6: s.set(249) #way down below s.pack() self.sliders.append(s) # Define a quit button and quit event to help gracefully shut down threads tk.Button(self,text="Quit",command=self.quit).pack() self._quit = Event() self.capture_thread = None # This function is called when each Scale widget is moved def update_slider(self,idx,pos): self.ar[idx] = c_int32(int(pos)) # This function launches a thread to do video capture def start_capture(self): self._quit.clear() # Create and launch a thread that will run the video_capture function # self.capture_thread = Thread(cap = cv2.VideoCapture(0), args=(self.ar,self._quit)) self.capture_thread = Thread(target=video_capture, args=(self.ar,self._quit)) self.capture_thread.daemon = True self.capture_thread.start() def quit(self): self._quit.set() try: self.capture_thread.join() except TypeError: pass self.parent.destroy() # This function simply loops over and over, printing the contents of the array to screen def video_capture(ar,quit): print ar[:] cap = cv2.VideoCapture(0) Xerror = 0 Yerror = 0 XerrorStr = '0' YerrorStr = '0' while not quit.is_set(): # the slider values are all readily available through the indexes of ar # i.e. w1 = ar[0] # w2 = ar[1] # etc. # Take each frame _, frame = cap.read() # Convert BGR to HSV hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV) # define range of blue color in HSV lower_green = np.array([ar[0],ar[1],ar[2]]) upper_green = np.array([ar[3],ar[4],ar[5]]) # Threshold the HSV image to get only green colors mask = cv2.inRange(hsv, lower_green, upper_green) # Bitwise-AND mask and original image res = cv2.bitwise_and(frame,frame, mask= mask) cv2.imshow('frame', frame) # cv2.imshow('mask',mask) # cv2.imshow('res',res) #------------------------------------------------------------------ img = cv2.blur(mask,(5,5)) #filter (blur) image to reduce errors cv2.imshow('img',img) ret,thresh = cv2.threshold(img,127,255,0) im2,contours,hierarchy = cv2.findContours(thresh, cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE) print 'number of contours==640x480==================== ', len(contours) target=0 if len(contours) > 0: numbercontours = len(contours) while numbercontours > 0: numbercontours = numbercontours -1 # contours start at 0 cnt = contours[numbercontours] #this is getting the first contour found, could look at 1,2,3 etc x,y,w,h = cv2.boundingRect(cnt) # #---line below has the limits of the area of the target----------------------- # #if w * h > 4200 and w * h < 100000: #area of capture must exceed to exit loop if h > 30 and w < h/3: #area of capture must exceed to exit loop print ' X Y W H AREA Xc Yc xEr yEr' Xerror = (-1) * (320 - (x+(w/2))) XerrorStr = str(Xerror) Yerror = 240 - (y+(h/2)) YerrorStr = str(Yerror) print x,y,w,h,(w*h),'___',(x+(w/2)),(y+(h/2)),'____',Xerror,Yerror break #------- draw horizontal and vertical center lines below cv2.line(img,(320,0),(320,480),(135,0,0),5) cv2.line(img,(0,240),(640,240),(135,0,0),5) displaySTR = XerrorStr + ' ' + YerrorStr font = cv2.FONT_HERSHEY_SIMPLEX cv2.putText(img,displaySTR,(10,30), font, .75,(255,255,255),2,cv2.LINE_AA) cv2.imshow('img',img) # wrtie to file for our server' sys.stdout = open(f,"w") print 'H,V:',Xerror,Yerror sys.stdout = sys.__stdout__ target=1 # #-------------------------------------------------------------------- if target==0: # no target found. print non-physical values out to a file sys.stdout = open(f,"w") print 'H,V:',1000,1000 sys.stdout = sys.__stdout__ k = cv2.waitKey(1) & 0xFF #parameter is wait in millseconds if k == 27: # esc key on keboard cap.release() cv2.destroyAllWindows() break if __name__ == "__main__": root = tk.Tk() selectors = CaptureController(root) selectors.pack() # q = tk.Label(root, text=str(x)) # q.pack() selectors.start_capture() root.mainloop() |
Well, that was a big program by my standards.
Here’s the UDP server that goes with it. I call it server.py.
#!/usr/bin/env python # inspired by https://gist.github.com/Manouchehri/67b53ecdc767919dddf3ec4ea8098b20 # first we get client connection, then we read data frmo file. This order is important so we get the latest, freshest data! import socket import re sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) server_address = '0.0.0.0' server_port = 5005 server = (server_address, server_port) sock.bind(server) print("Listening on " + server_address + ":" + str(server_port)) while True: # read up to 32 bytes from client payload, client_address = sock.recvfrom(32) print("Request from client: " + payload) # get correction from file while True: with open('green.txt','r') as myfile: data=myfile.read() #H,V: 9 -14 data = data.split(":") if len(data) == 2: break sent = sock.sendto(data[1], client_address) |
For development testing I wrote a UDP client to go along with that server. I called it recvudp.py.
#!/usr/bin/env python import socket UDP_IP = "127.0.0.1" UDP_PORT = 5005 print "UDP target IP:", UDP_IP print "UDP target port:", UDP_PORT sock = socket.socket(socket.AF_INET, # Internet socket.SOCK_DGRAM) # UDP # need to send one newline minimum to receive server's message... MESSAGE = "correction"; sock.sendto(MESSAGE, (UDP_IP, UDP_PORT)) # get data data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes print "received message:", data |
Problems
Lag is bad. Probably 1.5 seconds or so.
Video is green, but then we designed it that way.
Bandwidth consumption of VNC is way too high. We’re supposed to be under 7 mbps and it is closer to 12 mbps right now.
Probably won’t work under the bright lights or an arena or gym.
Sliders should be labelled.
Have to turn a pixel correction into an angle.
Have to suppress initial warning about ssh default password.
To be improved, hopefully…