AKA: Das Blinkenlights

Date: 2017 — now

Status: Steady Green

I sometimes envy those that work in hardware. To be able to build something that one can hold and touch. It’s something you really cannot do with software. And yeah, I dabbled a little with Arduino, setting up sketches that would run on prebuilt shields, but I never went beyond the point of building something that, however trivial or crappy, I could call my own.

Except for this one time.

And I admit that this thing is pretty trivial and crappy: little more than some controllable LEDs. But the ultimate irony is that it turned out to be quite useful for a bunch of software projects. 

The Hardware

I built this Arduino shield a while ago, probably something like 2013. It’s really not that complicated: just a bunch of LEDs wired up in series to a bunch of resistors atop an Arduino prototyping shield. The LEDs can be divided up into two groups of three, with each group having a red, amber, and green LED, arrange much like two sets of traffic lights. I’m using the analogue pins of the Arduino, making it possible to dim the LEDs (well, “dimmed”: the analogue pins are little more than a square pulse with an adjustable duty cycle).

I can’t remember why I built this shield originally: it might had something to do with train signals, or maybe they were intended as indicators right out of the box. But after briefly using them for their original purpose, it sat on my desk for a while before I started using them as indicator lights. Their first use was for some tool that would monitor the download and transcode of videos. This would take between 45–60 minutes, and it was good to be able to start the job, leave the room, and get the current progress without having to wake the screen while I pass by the door. The red LED will slowly pulse while the download was in progress, then the yellow LED will start flashing when transcoding begins. Once everything is done, the green LED will be lit (or the red LED, which will indicate an error). The Arduino sketch had a bunch of predefined patterns, encoded as strings. Each character would indicate an intensity, with “a” being the brightness, and “z” being the dimmest (I believe the space or dot meant “off”). Each LED could be set to a different pattern, which was done via commands sent over the RS-232 connection. I think the code driving this connection was baked into the download-and-transcode tool itself. The Arduino would reset whenever the RS-232 connection is formed, and just letting it do this when the tool started up meant that I didn’t need to worry about connection state (it didn’t make the code portable though).

Watching Webpack

Eventually this tool fell out of use, and for the long time this board sat in my drawer. Projects came and went, until one came along with a problem that was perfect for this device. I was working on a HTML web-app and I was switching between the code and a web-browser, while Webpack was watching for changes. Because I only had a single screen, the terminal was always out of sight — behind either the code editor or the web-browser — and the version of Webpack I was using would stop watching when it encountered an error (a Go application was serving the files, and Webpack was simply deploying the bundle assets to a public folder, so even though Webpack would stop working, the actual web-server will continue running). Not seeing these errors, I’d fall into the trap into thinking that I was changing things, and getting confused as to why I wasn’t seeing them in the browser. I could go for a minute or two like this before I found out that Webpack died because of an earlier error and my changes were not getting deployed at all. So I dug this device out, built a very simple Go CLI tool and daemon that would talk to it, and hacked it into the Webpack config. When a Webpack build started, it would light up the amber LED. If the build was successful, the green LED would light up; if not, the red LED would.

This proved to be super useful, and took out the guesswork of knowing when a change was deployed. As long as the green LED was lit, it’s good to go, but as soon as amber becomes red, I know I’ll have to check for errors and get it green once more.

The sketch and daemon software is a lot simpler than what this device used to do. Now, instead of individual patterns of intensity, the daemon — which is itself controlled by a CLI tool — would communicate to the device using a very simple protocol that would either turn LEDs on or off. Some of the protocol details, taken from the Arduino sketch, are included below:

/*
 * ledstatus - simple led indicators
 * 
 * SERIAL FORMAT
 * 
 * Commands take the form:  <cmd> <pars>... NL.  Any more than
 * 8 bytes (1 command, 7 parameters) will be ignored.
 * 
 * Responses from the device will take the form: <status> <par>... NL
 * 
 */

// Commands
\#define CMD_NOP       0x0
\#define CMD_PING      'p'   // a ping, which should simply respond with RES_OK
\#define CMD_TURN_ON   'o'   // 'o' <addr>   :: turn on the leds at these addresses
\#define CMD_TURN_OFF  'f'   // 'f' <addr>   :: turn off the leds at these addresses

// Response
\#define RES_OK        '1'

\#define PIN_ADDR_G1   (1 << 0)
\#define PIN_ADDR_Y1   (1 << 1)
\#define PIN_ADDR_R1   (1 << 2)
\#define PIN_ADDR_G2   (1 << 3)
\#define PIN_ADDR_Y2   (1 << 4)
\#define PIN_ADDR_R2   (1 << 5)

But in a way the simplicity actually helps here. Because it’s now a command and daemon, I could use it in anything else where I’d like to show progress without having to see the screen. Just now, for example, I’m working on a Go project that uses Air to rebuild and restart whenever I change a template. The cycle is slightly longer than a simple Webpack run, and I tend to reload the browser window too soon. So waiting for this device to go  from amber to green cuts down on these early reloads.

So thats the Build Indicators. The project is steady, and I have no desire to do anything significant, like modify the hardware. But if I were to work on it again, I think I’d like it to add variable intensity back, and extend the command language to let the user upload customer patterns. But for the moment, it’s doing it’s job just fine.

Date: 2021 – 2022

Status: Paused

First project I’ll talk about is Broadtail. I think I talked about this one before, or at least I posted screenshot of it. I started work on this in 2021. The pandemic was still raging, and much of my downtime was watching YouTube videos. We were coming up to a federal election, and I was getting frustrated with seeing YouTube ads from political parties that offend me. This was before YouTube Premium so there was no real way to avoid these ads. Or was there?

A Frontend For youtube-dl

I had some experience with youtube-dl in the past, downloading and saving videos that I hoped to watch later. I recently discovered that YouTube also published RSS feeds for channels and playlist. So I was wondering if it was possible to build something that could use both of these. My goal was to have something that would allow me to subscribe to YouTube channels via RSS, download videos using youtube-dl, and watch them via Plex on my TV. This was to be deployed on a Intel Nuc that I was using as a home server, and be accessible via the web-browser.

I deceided to get the YouTuble downloading feature built first. I started a new Go project and got something up and running reasonably quickly. It was a good excuse to get back to vanilla Go web development, using http.Handle and Go templates, instead of relying on frameworks like Buffalo (don’t get me wrong, I still like Buffalo, but it is quite heavy handed).

It was also an excuse to try out StormDB, which is an embedded NoSQL data store. The technology behind it is quite good — it’s used B-Trees memory mapped files under the cover — and I tend to use it for other things as well. It proved to be quite usable, apart from not allowing multiple read/writers at the same time, which made deployments difficult.

But the backend code was the easy part. What I lack was any sense of web design. That’s one good thing about a framework like Buffalo: it comes with a usable style framework out of the box (Bootstrap). If I were to go my own way, I’d have to start from scratch.

The other side of that coin though, is that it would give me the freedom to go for something that’s slightly off-beat. So I went for an aesthetics that reminded me of early 2000 web-design: san-serif font, all grey lines, dull pastels colours, small controls and widgets (I stopped short at gradients and table-based layouts).

This version also included a hand-rolled job manager that I used for a bunch of other things. It’s… fine. I wouldn’t use it for anything “real”, but it had a way of managing job lifecycles, updating progress, and the ability to cancel a running job. So for that, it was good enough.

Finally, it needed a name. At the time, I was giving all projects bird-like codename, since I can’t come up with names that I liked. I eventually settled on Broadtail, which was a reference to broadtail parrots, like the rosella.

RSS Subscriptions

It didn’t take long after I got this up and running before I realised I needed the RSS subscription feature. So that was the next thing I added.

That way it worked was pretty straight forward. One would setup a subscription to a YouTube channel or playlist. Broadtail will then poll that RSS subscription every 15 minutes or so, and show new videos on the homepage. Clicking that video item would bring up details and an option to download it.

Each RSS subscription had an associated target directory. Downloading an ad-hoc video would just dump it in a configured directory but I wanted to make it possible to organise downloads from feeds in a more structured way. This wasn’t perfect though: I can’t remember the reason but I had some trouble with this, and most videos just ended up in the download directory by default (it may have had to do with making directories or video permissions).

Only the feed polling was automatic at this stage. I was not interested in having all shows downloaded, as that would eat up on bandwidth and disk storage. So users still had to choose which videos they wanted to download. The list of recent feed items were available from the home-screen so they were able to just do so from there.

I also wanted to keep abreast with what jobs were currently running, so the home-screen also had the list of running job.

The progress-bar was powered by a web-socket backed by a goroutine on the server side, which meant realtime updates. Clicking the job would also show you the live output of the youtube-dl command, making it easy to troubleshoot any jobs that failed. Jobs could be cancelled at any time, but one annoying thing that was missing was the ability to retry failed job. If a download failed, you had to spin up a new job from scratch. This meant clearing out the old job from the file-system and finding the video ID again from wherever you found it.

If you were interested in a video but were not quite ready to download it right away, you could “favourite” it by clicking the star. This was available in every list that showed a video, and was a nightmare to code up, since I was keeping references to where the video came from, such as a feed or a quick look. Keeping atop of all the possible references were become difficult with the non-relational StormDB, and the code that handled this became quite dodgy (the biggest issue was dealing with favourites from feeds that were deleted).

Rules & WWDC Videos

The basics were working out quite well, but it was all so manual. Plus, going from video publication to having something to watch was not timely. The RSS feed from YouTube was always several hours out of date, and downloading whole videos took quite a while (it may not have been realtime, but it was pretty close).

So one of the later things I added was a feature I called “Rules”. These were automations that would run when the RSS feed was polled, and would automatically download videos that met certain criteria (you could also hide them or mark them as downloaded). I quite enjoy building these sorts of complex features, where the user is able to do configure sophisticated automatic tasks, so this was a fun thing to code up. And it worked: video downloads would start when they become available and would usually be in Plex when I want to watch it (it was also possible to ping Plex to update the library once the download was finished). It wasn’t perfect though: not retrying failed downloads did plague it a little. But it was good enough.

This was near the end of my use of Broadtail. Soon after adding Rules, I got onto the YouTube Premium bandwagon, which hid the ads and removed the need for Broadtail as a whole. It was a good thing too, as the Plex Android app had this annoying habit of causing the Chrome Cast to hang, and the only way to recover from this was to reboot the device.

So I eventually returned to just using YouTube, and Broadtail was eventually abandoned.

Although, not completely. One last thing I did was extend Broadtail’s video download capabilities to include Apple WWDC Videos. This was treated as a special kind of “feed” which, when polled, would scrap the WWDC video website. I was a little uncomfortable doing this, and I knew when videos were published, they wouldn’t change. So this “feed” was never polled and the user had to refresh it automatically.

Without the means to stream them using AirPlay, downloading them and making them available in Plex was the only way I knew of watching them on my TV, which is how I prefer to watch them.

So that’s what Broadtail is primarily used for now. It’s no longer running as a daemon: I just boot it up when I want to download new videos. And although it’s only a few years old, it’s starting to show signs of decay, with the biggest issue being youtube-dl slowly being abandoned.

So it’s unlikely that I’ll put any serious efforts into this now. But if I did, there are a few things I’d like to see:

• Authentication added with username/password
• Retry failed video downloads.
• The ability to download YouTube videos in audio only (all these “podcasts” that are only available as YouTube videos… 😒)
• The ability to handle the lifecycle of videos a little better than it does now. It’s already doing this for errors: when a download fails, the video is deleted. But it would be nice if it did things like automatically delete videos 30 days after downloading them. This would require more control over the “video store” though.

So, that’s Broadtail.