2021 Intro
This post promises more than it actually delivers, ha, ha. It is squarely aimed at the more mid to advanced RPi enthusiast. Most who read this will get discouraged and look for another solution. I did the same in fact and I will review my failures with alternatives.
The essence of this aproach is screen automation with a very nice tool called xdotool. For me it works. It will definitely, 100% require some tweaks for anyone else. This is not run a few installs, copy this code and you’re good to go. But if you have the patience, you wil be rewarded with either fully or at least semi-automated video uploads to YouTube from your Raspberry Pi.
One caveat. Please obey YouTube’s terms of service. In other words, don’t abuse this! As soon as someone starts using this method in an aggressive or abusive fashion, we will all lose this capability. They have crack security experts and could squelch this approach in a heartbeat.
I actually don’t have all the peices in place for myself, but I have enough cool stuff that I wanted to begin to share my findings.
One beautiful thing about what I’m going to show is that you get to see the cursor moving about the screen in response to your automated commands – you see exactly what it;s doing, which screens it’s clicking through, etc. So if there’s an issue – say YouTube changes its layout – you’ll most likely be able to know how to adapt.
What’s wrong with using YouTube’s api?
Plenty. It used to be feasible. It certainly would make all our lives a lot easier. But YouTube is not a charity. They have squeezed out the little guy by making the barrier to entry so high that it’s really only available to highly determined IT folks. It’s just too difficult to figure out all the neeed screens, etc, and all the help guides refer to older api versions where things were different. YouTube clamped down in July 2020 on who or what can use their api. There’s a lot of old HowTos pre-dating that that will just lead you to dead-ends. So, go ahead, I dare you to stop reading this and use the api and report back. Maybe yuo manage to create a project, great, and an api key, great, and even to assign your api key the correct YouTube specific permissions – all great, and associate crednetials – super, and finally borrow someone’s code to upload a video – been there, done that. That video will be listed private. So then you try to root around to see what you have to do to make it public. Ah, a project review. Great. You were only in test mode. So no your confronted with this form. If they cared about the little guy there would be a radio button – “I only wish to upload a few videos a week for a small cadre of users, spare me the bureaucracy,” and that’d be it. But, no… Are you applying for a quota? Huh? I just want to upload a video and have it marked as unlisted. Some users remarked they filled out the form, never got their project reviewed and never heard back. Maybe they’re the exception, I don’t know. It’s just over the top for me so I give up.
OK. So, maybe YoutubeUploader?
Nope. Doesn’t work. It’s based on the old stuff.
OK. What about that guy’s api-less Node.js uploader?
Maybe. I could not get it to work on RPi. But I didn’t try super hard. I just like rolling my own, frankly. My approach is much more transparent. At least this approach inspired me to imagine the approach I am about to share. Because I believe the Node.js guy is just doing screen scraping but you can’t even see the screens.
Or simply do a Livestream?
Agreed. Livestreaming is quite straightforward by comparison with what I developed. My blog post about one click livestreaming covers it. But I have not had good results with reliability. As often as it works, it doesn’t work. With this new approach I’m going to try to create separate steps so that if anything goes wrong, an individual step can be re-run. Another advantage of separating steps is that a recording can be done “in the field” and without WiFi access. Remember an RPi 3 works great for hours with a decent portable USB battery that’s normally used for phones. Then the resulting recording can be converted to video and uploaded once the RPi is back to its usual WiFi SSID.
Preliminary upload, October 2021
In the video below the right screen is a terminal window showing what the script is doing. It needs some tweaking, and the YouTube window gets stuck so it’s not showing some of the screens. But it’s already totally awesome – and it worked!
Code for the above
#!/bin/sh
# automate upload of YouTube videos
#
# define some functions
randomsleep(){
# sleep random amount between 1.5 to 2.5 seconds
t10=$(shuf -n1 -i 15-25)
t=$(echo $t10/10|bc -l)
sleep $t
}
drjtool(){
randomsleep
xdotool $1 $2 $3
randomsleep
}
echo Start video upload
echo set display to main display
export DISPLAY=:0
# launch chromium
echo launch chromium
chromium --kiosk https://studio.youtube.com/ > /dev/null 2>&1 &
sleep 20
echo move to CREATE button
drjtool mousemove 579 19
echo click on CREATE button
drjtool click 1
echo move to Upload videos
drjtool mousemove 577 34
echo click Upload videos
drjtool click 1
echo move to SELECT FILES
drjtool mousemove 305 266
echo click on SELECT FILES
drjtool click 1
echo move mouse to Open button
drjtool mousemove 600 396
echo click open and pause a bit for video upload
drjtool click 1
sleep 20
echo "mouse to NEXT button (accept defaults)"
drjtool mousemove 558 386
echo click on NEXT
drjtool click 1
echo move to radio button No it is not made for kids
drjtool mousemove 117 284
echo click radio button
drjtool click 1
echo back to NEXT button
drjtool mousemove 551 384
echo click NEXT
drjtool click 1
echo 'click NEXT again (then says no copyright issues found)'
drjtool click 1
echo click NEXT again
drjtool click 1
echo move to Unlisted visibility radio button
# [note that public would be drjtool mousemove 142 235, private is 142 181]
drjtool mousemove 142 208
echo click Unlisted
drjtool click 1
echo move to copy icon
drjtool mousemove 532 249
echo echo copy URL to clipboard
drjtool click 1
echo move to Save
drjtool mousemove 551 384
echo click Save
drjtool click 1
echo move to CLOSE
drjtool mousemove 434 270
echo click close
drjtool click 1
echo video URL
xsel -b|tee clipboard
echo kill chromium browser
sleep 5
echo kill chromium
kill -9 %1
url=$(cat clipboard|xargs -0 echo)
echo url is $url
I call the script auto-test.sh, just to give it a name.
Ingredients
- RPi 3 or RPi 4
- Raspberry OS with full GUI and autologin set up
- tiger VNC, i.e., the package tigervnc-scraping-server
- chromium browser – but I think that comes with the GUI install
- xdotool (apt-get install xdotool)
- xsel (apt-get install xsel)
- YouTube account
- crontab entries – see below
- you do not need an HDMI display, except for the OS setup
- a vncviewer such as Real VNC
Idea
Don’t use the GUI for anything else!
Crontab (do a crontab -e to get into your crontab) should contain these lines:
@reboot sleep 15; /home/pi/recordswitch.sh > recordswitch.log 2>&1
# launch vnc server on display 1
@reboot sleep 65;x0vncserver -localhost no -passwordfile ~/.vnc/passwd -display :0 > x0vncserver.log 2>&1
Work with chromium the first time by hand. As I recall you should:
- Create a directory like 00uploads – so it appears highest in the list
- put a single video in 00uploads
- Do an upload by hand (to help chromium remember to choose this upload directory)
- launch chromium browser
- log into your YouTube account at https://studio.youtube.com
- shrink the browser until its size is 50% (Ctrl-Shift – about four times)
- Don’t add other tabs and stuff to Chromium
Then subsequent launches of chromium should remember a bunch of these settings, specifically, your login info, the shrunken size, the upload directory, and maybe the (lack of) other tabs.
The beauty of this approach is that it is more transparent than the alternatives. You see exactly what your program is doing. You can issue the xdotool commands by hand to, e.g., change up the coordinates a little bit. Or even enter a video title.
So getting back to the idea, the automation idea is to finish a video somehow, then move it to the 00uploads dircetory, invoke this uploader program, then either move it to a uploaded directory or some such.
Imagine the versatility if I used my remote controller for RPi to map one button for audio recording, and a second button for automating video upload! Well, when I find the time that’s what I plan to do. I will make a separate post where the recording and uploading are shown – more or less the culmination of all the pieces.
Oh, and back to the idea again, I wanted to share the unlisted link with band members. So, you see how it is basically in the result of xsel -b since xsel copied the clipboard which contained the YouTube URL for this video we just uploaded? I have to fix up the parsing because some junk characters are getting included, but I plan to email that link to myself first, where I will do a brief manual check, and then forward it to the rest of the band. so, again, it’s really cool that we could even think to pull that off with this simplistic approach.
Techniques developed for this project
Lots.
I “discovered” – in the sense that Columbus discovered America – xdotool as an amazing X Windows screen automation tool. I knew of autohotkey for Windows so inquired what was like it for X Windows. I further learned that xdotool is generally broken when it comes to use with traditional VNC servers such as the native tightvncserver. It simply doesn’t work. But Tiger VNC is a scraping server so it like shares your console screen and makes it available via VNC protocol. That’s required because to develop this approach you have to see what you’re doing. All those coordinates? it comes from experimentation.
I also learned how to embed a YouTube video in my blog post. In fact this is the very first video I made for a blog post. So I did a screen recording for the first time with screen recorder for Windows.
I landed on the idea of a side-by-side video showing my terminal running the automated script in one window and the effect it is having on the chromium browser in the other window running on the RPi.
I put a wrapper around xdotool to make things cleaner. (But it’s not done yet.)
I changed to two-factor authentication to see if it made a difference. It did not. It still remembers the authentication, thankfully, at least for a few days. I wonder for how long though. Hmm.
Kiosk mode. By launching chromium with kiosk mode it not only gives us more screen real estate to work in, it in principle should also permit you to interact with chromium in a regular fashion and still have it come up in a known, fixed position, which is an absolute requirement of this approach. All the buttons have to have the same coordinates from invocation to invocation.
I also developed ffmpeg-based converters which take wav files and converts them to mp3’s (a nice compact format. wav files are space hogs), and another which takes mp3 files and adds a gray screen and converts them to flv ([Adobe] Flash Video, I guess – a compact video file format which YouTube accepts).
I also learned the ffmpeg command to tell exactly how long a recording is.
I also learned how to turn off blocking in ffmpeg so that its constantly writing packets and thus not losing audio data at the end when stopped.
I came up with the idea of randomizing the sleep time between clicks to make it seem more human-like.
And mostly for the purpose of demonstrations, though it also greatly helps in debugging, I introduced around two seconds of sleep both before and after a command is issued. That really makes things a lot clearer.
The results of the clipboard, xsel -b, contains null bytes. I had trouble parsing it to pull out just the url, but finally landed on using xargs -0 which is designed to parse null-delimited strings. And it worked! This was a late edition and did not make it into the video, but is in the provided script above towards the bottom.
ffmpeg chokes on too-complicated filenames. Who knew? I had files containing colon (:) and dash (-) characters which work perfectly fine in linux, but ffmpeg was interpreting part of the filename as a command-line argument it appeared. The way out of that mess was to introduce file: in front of the filename.
I’ll probably put my ffmpeg tricks into my next post because I want to keep this one lean and focused on this one upload automation topic.
References and related
This post was preceded by my post on how to stream live to YouTube with a click of a button on a Raspberry Pi.