Old Projects
- Having multiple catalogues with different access roles.
- Feature flags, which disabled certain features for users.
- Generating a QR code for an API token for easy sign in on the mobile app.
- Authentication added with username/password
- Retry failed video downloads.
- The ability to download YouTube videos in audio only (all these “podcasts” that are only available as YouTube videos… 😒)
- The ability to handle the lifecycle of videos a little better than it does now. It’s already doing this for errors: when a download fails, the video is deleted. But it would be nice if it did things like automatically delete videos 30 days after downloading them. This would require more control over the “video store” though.
Small Calculator Commands
This page documents the extra commands from Small Calculator. These were taken from source code, pretty much as is, but styled to suite the web, and any spelling mistakes fixed. These were retrievable from the application itself by typing “help” follow by the command.
Available Commands
The list of available commands are as follows
BLOCK <statements> Executes a block of statements
HELP [topic] Display help on topic
DEFFNC <function> Defines a new function
ECHO <text> Displays text on the line
ECHOEXPR <cmd> Executes a command and displays the result
EXEC <file> Executes a file of commands
FUNCTIONS Displays all predefined functions
IF <pred> Does a command on condition
RETURN <val> Sets the return value
RETURNEXPR <cmd> Sets the return value to the result of <cmd>
Type "HELP <command>" to see infomation on a command
BLOCK
BLOCK {<cmd1>} {<cmd2>} ...
Executes a block of commands. The commands can be any statement including other block statements.
DEFFNC
DEFFNC <fnname>(<parameters>) = <command>
Defines a new function. The function name can only consist of letters
and numbers. Only a maximum of 4 parameters can be used in the
parameter list. Parameters are required to be referred to using
$
Example:
deffnc test(x) = $x + 2
-- Adds two to any number
deffnc sign(k) = if ($k < 0} {-1} {if {$k > 0} {1} {0}}
-- Returns -1 if k is negative, 1 if k is positive and
0 if k is 0.
Functions can be recursive if using the “if” command.
ECHO
ECHO <string>
Displays a string on the console.
ECHOEXPR
ECHOEXPR <command>
Executes a command and displays the result on the console.
EXEC
EXEC <filename>
Executes a file of commands. Lines starting with “;” are considered comments. Lines ending with "" are considered incomplete and the next line is appended (after trimming) to the end of that line.
FUNCTIONS
functions
Displays all predefined functions. No user functions are included.
IF
IF {<cond>} {<truepart>} {<falsepart>}
If the result of
HELP
HELP [topic]
Displays a help topic on the console window. Use “HELP
RETURN
RETURN <val>
Sets the return value to
RETURNEXPR
RETURNEXPR <cmd>
Sets the return value to the return value of
Small Calculator
Date: Unknown, but probably around 2005
Status: Retired
Give me Delphi 7, a terminal control, and an expression parser, and of course I’m going to build a silly little REPL program.
I can’t really remember why I though this was worth spending time on, but I was always interested in little languages (still am), and I guess I though having a desk calculator that used one was worth having. I was using a parser library I found on Torry’s Delphi Pages (the best site at the time to get free controls for Delphi) for something else, and after getting a control which simulated a terminal, I wrote a very simple REPL loop which used the two.
And credit to the expression parser developer: it was pretty decent. It supported assignments and quite a number of functions. Very capable for powering a desk calculator.
For a while the app was simply that. But, as with most things like this, I got the itch to extend it a little. I started by added a few extra commands. Simple things, like one that would echo something to the screen. All quite innocent, if a little unnecessary. But it soon grew to things like if statements, blocks using curly brackets, and function definitions.
It even extended to small batch scripts, like the one below. The full set of commands is listed here.
x := 2
y := 3
if {x = y} {echo 5} \
{echo 232}
return
These never went anywhere beyond a few tests. The extra commands was not really enough to be useful, and they were all pretty awful. I was already using a parser library so I didn’t want to spend any time extending it. As a result, many of these extensions were little more than things that scanned and spliced strings together. It was more of a macro language rather than anything else.
Even with the expression parser the program didn’t see a great deal of use. I was working on the replacement at the time which would eventually be much more capable, and as soon as that was ready, this program fell out of use.
Even so, it was still quite a quirky little program to make a bit of an impression.
Alto
Date: 2020 — present
Status: Rockin'
The year was 2020. The pandemic was just beginning and I was stuck at home, not being able to do much of anything. Worse, rumours came around that Google was shutting down Google Play Music, my music player of choice. They were going to force everyone onto their streaming service instead. Oh, they may have a place for all the music you’ve downloaded (or written) yourself, but not in the first version. Maybe they’ll get to it later.
“Well, fuck that”, I said to myself. “I’ve been battered around by Google shutting down things and forcing migrations onto other things one to many times. I’m going build my own music app.”
And so I built my own music app: Alto.
Why “Alto”? Well, the name goes back to when I was in secondary school, when I was learning the viola, which uses the alto clef for it’s written music. I said to myself at the time that if I were ever to build a music player, it’ll have something referencing the viola. The alto clef seemed like the best thing to use. Plus, it stands out amongst the other music apps that tend to use other notation symbols like notes or the treble clef.
The idea for Alto was pretty straight forward: a music player and catalogue that’ll manage and stream music from an S3 bucket. No tracking, no “promotions” or “recommendations”, no bullshit UI that’s impossible to navigate. Only the music I’m interested in, played the way I want, and a dead simple UI that puts the album front and centre. The music player that was meant for me.
The Web Catalogue
The ultimate version that would come to be would consist of two parts: an Android mobile app, and a web-app. I’ll talk about the mobile app in the next post.
The web-app can be used as a player, but is ultimately be responsible for managing the collection.
The web-app was built using Buffalo, a rapid web development much like
Rails for Go. It’s basically a simple server-side rendered web-app. The
frontend consists mainly of Bootstrap plus some Stimulus and vanilla
JavaScript to handle the interactive elements.
It’s also using Turbo to prevent unloading the current page when
moving to a new one. This means clicking around the site will not stop
playback of the current track, a very nice feature (and doubly so when
you consider that this isn’t a single-page app).
Much of the UI is dedicated to managing the catalogue but there is also an integrated player, which can be invoked by clicking the play button. The player itself can be bought up at any time by pressing “P”. There’s no scrubber but there are seek by 30 second buttons which do the job. Each of the controls in this player have associated shortcut keys that are always available, even with the player hidden.
The collection is managed within a PostgreSQL database and referenced files stored on an S3 bucket. S3 was chosen to ensure that if I were ever to stop work on this project, or the database were to be corrupted, I wouldn’t loose my music. It does mean that I’ve got an ongoing cost for running this service, but based on the amount of music I’m keeping and my music listening patterns, the monthly bill for S3 is around 50¢-60¢ AUD, plus $12.00 US for the web-app server.
The catalogue model was made as simple as possible. The main construct is the Album, which would consist of zero or more Tracks. Albums had things like title, artist, cover images, etc. but these are nothing more than just properties of an album.
Tracks could be added one at a time via the frontend, or uploaded from a Zip file pulled from a URL (useful for songs bought on Bandcamp). The catalogue tries it’s best to avoid uploading media via the web-server, either opting to pull it in from the backend or upload it to S3 directly. This was a deliberate choice to reduce the amount of network it uses, but it does mean going through some strange loops. For example, when uploading a single track via the frontend, it would upload it directly to S3, then download it from S3 on the backend so that it can set catalogue metadata from the file itself (ID3 tags, MP3 length, etc). This is pretty convoluted and doesn’t even work half the time, and if I were making this again, I’d probably just bite the bullet and allow large uploads via the frontend.
Media
The media model is a little more complicated. Media — audio files, cover images, etc — all belonged to a Repository, which is essentially a reference to an S3 bucket, although it could also reference things like a HTTP domain. A repository does have some other configuration such as how to name the uploaded media files. I could’ve used something like a UUID, but I wanted to keep the names human readable as much as I could, so that if this project were to shutdown, I would still be able to access the files from S3 myself.
An Album or Track is linked to a Media record through what’s called a Media Reference. Each of these references has a “rel” property (short for “relevance”) which describes what the media is for: an audio file for a track, a cover image for an album, etc. There also exists a classifier which was to allow an Album of Track to use multiple Media records of a particular relevance. For example, some albums, released in different regions, had different album covers, with one being slightly darker than the other. The classifier could be use to switch between the two, based on whether Dark Mode was enabled. At least that was the theory: it never really got used for that.
Governing the link between albums, tracks and media was a simple resolution algorithm that supported things like inheritance. For example, tracks could have their own album cover, but if one didn’t exist, they would inherit it from the album. This went beyond just album covers: it would be possible, theoretically at least, to have the audio associated with the album and have the tracks reference a different cover image (I never tried this).
Finally, there are Playlists. These are pretty standard: just a collection of references to other tracks, plus some ordering information. Playlists and playlist items are essentially links and cannot have Media References themselves. There are some downsides to this: the biggest one being that Playlists do not have album cover art, which is something I’ll need to fix. Playlists can also have metadata items.
Speaking of metadata items.
Metadata Items
A number of objects can also have metadata, which could be use to attach extra attributes. The goal was to make this generic enough for end users to use it for whatever, with Alto having a few predefined names it uses for it’s own purpose.
There were actually two kinds of metadata record. The first was an
arbitrary JSON structure that can be set for each Track. Only one such
name was reserved, called heading, which was used to define track
groups within albums. I do have plans for adding more attributes, such
as things dedicated to Eurovision tracks (year and country for example).
The second, and slightly older, type of metadata was for larger and more generic bits of information. I initially envisioned this to be used to store things like additional artist names, but since each metadata item was a separate row, that felt like overkill and why I added the JSON object (not sure why I thought this, PostgreSQL is quite a capable database). The main use of this is to store chapter markers. Chapters are a pretty simple format, just a bunch of lines with a timestamp and name separated by an equals sign:
0=Intro
139.5=Outlaw
229=Crises
467.5=The Watcher and the Tower
599=Interlude
774.5=Sequencer
1111=Coda
I suppose I should find a way to extract chapters from the MP3 file itself but for the moment I’m just setting them manually. I am also using it to store things like lyrics. I should say a few words as to how Media Refs and Metadata Items actually reference other things in the model. They use a notion of “object_type” and “object_id” which describes what the object is (album, track, playlist item, etc.) and object ID referencing the actual object itself. Because this is quite generic, I can’t rely on ON DELETE CASCADE to clean these up, so I opted for database triggers to remove media refs and metadata items when the base object type is removed.
CREATE FUNCTION delete_dependencies_of_object() RETURNS trigger AS $$
BEGIN
DELETE FROM media_references WHERE object_id = OLD.id;
DELETE FROM metadata_items WHERE object_id = OLD.id;
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER delete_dependencies_of_tracks
AFTER DELETE ON tracks
FOR EACH ROW EXECUTE PROCEDURE delete_dependencies_of_object();
CREATE TRIGGER delete_dependencies_of_albums
AFTER DELETE ON albums
FOR EACH ROW EXECUTE PROCEDURE delete_dependencies_of_object();
So far I haven’t had any issues with this, although I haven’t done a lot of snooping around the database to confirm these records are properly being cleaned up.
Grand Plans
I did have grand plans for this catalogue at one point: releasing it as open source, maybe making this a service that others could use. So there were a few things added which are unfinished and half baked. Some examples:
None of these really went anywhere, and if I were to rebuild this, I’d probably pull them out. As things stand now, it does need a bit of a refresh: upgrading Go packages, dealing with Node packages (ugh, it’s always a pain trying to update JS packages and Webpack). But since around late 2020, it’s been serving quite well as my primary music player.
Message Simulator Client
Years: 2017 — 2020
Status: Gone
I once worked at a company that was responsible for sending SMS messages via an API. Think one time passwords when you log into websites, before time-based OTP apps were a thing. And yeah, this did involve some “marketing” messages, although we were pretty strict about outright spam or phishing messages.
Anyway, since sending messages costed us money, we had a simulator setup in our non-prod environments which we used for testing. The features were pretty minimal: basically get the list of messages sent through the API, send a status message back, and simulating a reply from the receiver. The messages were kept in memory and there was no UI: everything was done via a web frontend generated from a Swagger spec.
Being somewhat bored one day, and getting frustrated with the clunky web frontend, I’d though I had a go at making a MacOS client for this thing. After finding my way around Xcode and AppKit, I managed to get something that was usable.
It was not a sophisticated app in the least. It only consisted of a toolbar, a client area, and a right sidebar.
The toolbar allowed switching the environment to connect to, such as Dev or Test. I believe the environments were hard-coded, and if I wanted to add a new one, I’d had to change the Swift code. There was a button to clear the messages from the simulator, and one to refresh the list of messages. There was also a very simple search, which simply did an in-memory substring match, but was good enough for finding unique IDs.
The client area consisted of the message ID and message body, and that’s it. Not included were source and target numbers, plus a few other things. These were occasionally useful to me, but not enough to justify the effort it would take to add them to the UI (I was more likely to use the message body anyway).
The right sidebar consisted of the message “details”, which was just the message ID and message content. There were also sections for sending a particular status message, or sending a reply to the selected message.
I always had grand plans for more features, but I couldn’t justify the time. And eventually I decided to leave, and the project was wiped once I returned my laptop.
Despite how bare-bones it was, it was still useful, and something I used most days if I had to work with the simulator. And given that it was my first attempt at a native Mac app, I was reasonably proud of it. So I think it deserves a place in the archives.
Broadtail
Date: 2021 – 2022
Status: Paused
First project I’ll talk about is Broadtail. I think I talked about this one before, or at least I posted screenshot of it. I started work on this in 2021. The pandemic was still raging, and much of my downtime was watching YouTube videos. We were coming up to a federal election, and I was getting frustrated with seeing YouTube ads from political parties that offend me. This was before YouTube Premium so there was no real way to avoid these ads. Or was there?
A Frontend For youtube-dl
I had some experience with youtube-dl in the past, downloading and saving videos that I hoped to watch later. I recently discovered that YouTube also published RSS feeds for channels and playlist. So I was wondering if it was possible to build something that could use both of these. My goal was to have something that would allow me to subscribe to YouTube channels via RSS, download videos using youtube-dl, and watch them via Plex on my TV. This was to be deployed on a Intel Nuc that I was using as a home server, and be accessible via the web-browser.
I deceided to get the YouTuble downloading feature built first. I started a new Go project and got something up and running reasonably quickly. It was a good excuse to get back to vanilla Go web development, using http.Handle and Go templates, instead of relying on frameworks like Buffalo (don’t get me wrong, I still like Buffalo, but it is quite heavy handed).
It was also an excuse to try out StormDB, which is an embedded NoSQL data store. The technology behind it is quite good — it’s used B-Trees memory mapped files under the cover — and I tend to use it for other things as well. It proved to be quite usable, apart from not allowing multiple read/writers at the same time, which made deployments difficult.
But the backend code was the easy part. What I lack was any sense of web design. That’s one good thing about a framework like Buffalo: it comes with a usable style framework out of the box (Bootstrap). If I were to go my own way, I’d have to start from scratch.
The other side of that coin though, is that it would give me the freedom to go for something that’s slightly off-beat. So I went for an aesthetics that reminded me of early 2000 web-design: san-serif font, all grey lines, dull pastels colours, small controls and widgets (I stopped short at gradients and table-based layouts).
This version also included a hand-rolled job manager that I used for a bunch of other things. It’s… fine. I wouldn’t use it for anything “real”, but it had a way of managing job lifecycles, updating progress, and the ability to cancel a running job. So for that, it was good enough.
Finally, it needed a name. At the time, I was giving all projects bird-like codename, since I can’t come up with names that I liked. I eventually settled on Broadtail, which was a reference to broadtail parrots, like the rosella.
RSS Subscriptions
It didn’t take long after I got this up and running before I realised I needed the RSS subscription feature. So that was the next thing I added.
That way it worked was pretty straight forward. One would setup a subscription to a YouTube channel or playlist. Broadtail will then poll that RSS subscription every 15 minutes or so, and show new videos on the homepage. Clicking that video item would bring up details and an option to download it.
Each RSS subscription had an associated target directory. Downloading an ad-hoc video would just dump it in a configured directory but I wanted to make it possible to organise downloads from feeds in a more structured way. This wasn’t perfect though: I can’t remember the reason but I had some trouble with this, and most videos just ended up in the download directory by default (it may have had to do with making directories or video permissions).
Only the feed polling was automatic at this stage. I was not interested in having all shows downloaded, as that would eat up on bandwidth and disk storage. So users still had to choose which videos they wanted to download. The list of recent feed items were available from the home-screen so they were able to just do so from there.
I also wanted to keep abreast with what jobs were currently running, so the home-screen also had the list of running job.
The progress-bar was powered by a web-socket backed by a goroutine on the server side, which meant realtime updates. Clicking the job would also show you the live output of the youtube-dl command, making it easy to troubleshoot any jobs that failed. Jobs could be cancelled at any time, but one annoying thing that was missing was the ability to retry failed job. If a download failed, you had to spin up a new job from scratch. This meant clearing out the old job from the file-system and finding the video ID again from wherever you found it.
If you were interested in a video but were not quite ready to download it right away, you could “favourite” it by clicking the star. This was available in every list that showed a video, and was a nightmare to code up, since I was keeping references to where the video came from, such as a feed or a quick look. Keeping atop of all the possible references were become difficult with the non-relational StormDB, and the code that handled this became quite dodgy (the biggest issue was dealing with favourites from feeds that were deleted).
Rules & WWDC Videos
The basics were working out quite well, but it was all so manual. Plus, going from video publication to having something to watch was not timely. The RSS feed from YouTube was always several hours out of date, and downloading whole videos took quite a while (it may not have been realtime, but it was pretty close).
So one of the later things I added was a feature I called “Rules”. These were automations that would run when the RSS feed was polled, and would automatically download videos that met certain criteria (you could also hide them or mark them as downloaded). I quite enjoy building these sorts of complex features, where the user is able to do configure sophisticated automatic tasks, so this was a fun thing to code up. And it worked: video downloads would start when they become available and would usually be in Plex when I want to watch it (it was also possible to ping Plex to update the library once the download was finished). It wasn’t perfect though: not retrying failed downloads did plague it a little. But it was good enough.
This was near the end of my use of Broadtail. Soon after adding Rules, I got onto the YouTube Premium bandwagon, which hid the ads and removed the need for Broadtail as a whole. It was a good thing too, as the Plex Android app had this annoying habit of causing the Chrome Cast to hang, and the only way to recover from this was to reboot the device.
So I eventually returned to just using YouTube, and Broadtail was eventually abandoned.
Although, not completely. One last thing I did was extend Broadtail’s video download capabilities to include Apple WWDC Videos. This was treated as a special kind of “feed” which, when polled, would scrap the WWDC video website. I was a little uncomfortable doing this, and I knew when videos were published, they wouldn’t change. So this “feed” was never polled and the user had to refresh it automatically.
Without the means to stream them using AirPlay, downloading them and making them available in Plex was the only way I knew of watching them on my TV, which is how I prefer to watch them.
So that’s what Broadtail is primarily used for now. It’s no longer running as a daemon: I just boot it up when I want to download new videos. And although it’s only a few years old, it’s starting to show signs of decay, with the biggest issue being youtube-dl slowly being abandoned.
So it’s unlikely that I’ll put any serious efforts into this now. But if I did, there are a few things I’d like to see:
So, that’s Broadtail.