My Impressions of GitHub Codespaces
The GitHub Universe 2021 conference started a few days ago and one of the features touted in the day one keynote was GitHub Codespaces. This is a development environment that is accessible from within your web browser. Itβs based on VSCode, which is a popular and well-designed IDE that is already written in JavaScript1, and also provides access to a Linux shell running in the cloud, allowing you to do various things like build and test your code.
After playing around with Codespaces during the beta, and seeing the feature develop into something that is available for everyone, I’d thought I’d say a few words on how I use it and what I think of it.
First, I should probably say that I’m not a heavy Codespaces user. The keynote seemed to be touting the features of Codespaces that makes it useful as a dev environment for collaborative coding. This is something I have no personal use for since it’s mainly just me working on repos I use with Codespaces, so I haven’t really explored these features myself.
The main use I have for Codespaces is making changes to repos on a machine that is not my own. There are times when I need to add a new feature or fix a bug on a tool I use, and for various reasons I cannot (or choose not to) setup a dev environment on the machine I’m working on to make the change. A case like this would have me look at one of the alternatives, or even make use of the GitHub web-editor. The web-editor works but doesn’t really offer much when it comes to building and testing your changes (I’ll talk more about the alternatives later).
I should say that this doesn’t happen very often. Most of my time I’m on my own machine, and I have no need for a web-based dev environment as I can just use the environment I have. But when that’s not possible, being able to spin up and make the change in Codespaces, complete with a Linux environment which you have sudo access to, is quite useful.
Codespaces is also pretty nice in terms of a coding environment. This is no real surprise since itβs based on VSCode, but compared to the alternatives, the little things like keystroke performance and doing things quickly in the editor make a huge difference. Iβve tried a bunch of alternatives in the past like Cloud9 and CodeAnywhere, and Codespaces is by far the most polished.
Another advantage Codespaces have over the comparison is that it seems suited to throw-away development environments. It might be possible to keep an environment around for an extended period of time, but I tend to spin up temporary workspace when I need them. The alternatives tend to prefer a more long-lived environment, which involves a lot setup for something that is kept around. This feels like splitting your dev environments in two, and I always feel the need to select one as the “definitive workspace” for that particular project going forward. I don’t feel that with Codespaces: I can quickly spin up a new environment, which has most of what I need already installed, and make the change I need with the full knowledge that once I no longer need it, it will be teared down (pro-tip: always push your changes to origin once you’re done making your changes in Codespaces). It helps that spinning up a new environment is quite fast.
So, that’s my impression of GitHub Codespaces. I’m not sure who has access to it: you may need to be on a paid plan, for example. But if it’s enable for your account, and you find yourself needing a temporary, cloud-based dev environment to do your work in, I’d suggest giving it a try.
-
Itβs actually TypeScript ↩︎
Alto Catalogue Update
I’ve really tied myself up in knots here. I’m spending some time working on Alto Catalogue, trying to streamline the process of uploading individual tracks into a new album. This is a workflow that is absolutely not user friendly at the moment, and the only way I’ve gotten tracks into the catalogue is to run a hacked-together tool to upload the tracks from the command line. The reason why I’m addressing this now is that it’s slightly embarrassing to have this open-source project without having a nice way of doing something that, by all accounts, is quite fundamental (a good hint for when you’re facing this is when it comes time to write the end-user documentation: if you can’t explain how to do something in a way that doesn’t include the word “hack”, “complicated”, or “unsupported”, then something is missing).
So I’m trying to close this feature gap, but it’s proving to be more complicated than I expected. The main issue relates ID3 tags and how media is arrange in the repository. Previous versions of the catalogue actually did have a way of uploading track media to the repository, which is essentially an S3 bucket. The way this work is that the catalogue will issue the browser a pre-signed Put URL, and the browser could upload the track media directly to S3. But in order to get a pre-signed URL, you need to know the object key, which is a bit like a file path. The old upload flow had the user enter the object key manually in the upload form.
This worked but I had some real issues with it. The first is that I’d like the objects within the S3 bucket to be organised in a nice way, for example “artist/album/tracknum-title.mp3”. I’m hoping that this S3 bucket will be my definitive music collection and I don’t want just some random IDs that are completely indecipherable when I browse the objects in the S3 bucket. That way, if I were ever to shutdown the catalogue down or loose all the metadata, I’d still be able to navigate my collection via the object keys alone.
The second was that this approach did not take into account the track metadata. Track metadata is managed in a PostgreSQL database and had to be entered in manually; yes, this included the track duration. The only reason I used the hacked together tool to upload tracks was that it was a tool I was already using to set ID3 tags on MP3 files, and it was trivial to add a HTTP client to do the upload from there. Obviously, asking users to run a separate tool to do their track uploads is not going to fly.
So I’m hoping to improve this. The ideal flow would be that the user will simply select an MP3 from their file system. When they click upload, the following things will happen:
- The ID3 tags of the MP3 will be read.
- That metadata will be used to determine the location of the object in S3.
- A pre-signed URL will be generated and sent to the browser to upload the file.
- The file is uploaded to S3.
- A new track record is created with the same metadata.
The libraries I’m using to read the ID3 tags and track duration requires the track media to be available as a file on the local file system (I assume this is for random access). Simply uploading the track media to the local file system would be the easiest approach, since it would allow me to read the metadata, upload the media to the repository on the backend, and setup the track metadata all in a single transaction. But I have some reservations about allowing large uploads to the server, and most of the existing infrastructure already makes use of pre-signed URLs. So the first run at this feature involved uploading the file to S3 and then downloading it on the server backend to read the metadata.
But you see the problem here: in order to generate a pre-signed URL to upload the object to S3, I need to know the location of the media, which I want to derive from the track metadata. So if I don’t want uploads to go straight to the file system, I need the object to already be in S3 in order to work out the best location of where to put the object in S3.
So I’m wondering what the best ways to fix this would be. My current thing is this series of events:
- Create a pre-signed URL to a temporary location in the S3 bucket.
- Allow the user to Upload the media directly to that location in the S3 bucket.
- On the server, download that media object to get the metadata and duration.
- From that, derive the objects location and move the object within S3, something I’m guessing should be relatively easy if the objects are in the same bucket.
- Create a new track record from the metadata.
The alternative is biting the bullet and allowing track uploads directly to the file system. That will simplify the crazy workflow above but means that I’ll need to configure the server for large uploads. This is not entirely without precedence though: there is a feature for uploading tracks in a zip file downloaded from a URL which uses the local file system. So there’s not a whole lot stopping me from doing this altogether.
The third approach might be looking for a JavaScript library to read the ID3 tags. This is not great as I’d need to get the location from the server anyway, as the metadata-derive object location is configured on a per repository basis. It also means I’ll be mixing up different ways to get metadata.
In any case, not a great set of options here.
Today is starting out reasonably quietly. All the big development work is done and the only tasks left are the little annoying things, the “bottom of the backlog barrel” if you will. Still, won’t be long before the next major feature comes our way.
A Tool That (Almost) Solved My Step Function Woes
I reached the end of my tether at work today on the task I was working on. The nature of the task involved crafting an AWS Step Function with several steps. Each step on the critical path contained some error handling, and several of them contained some cleanup logic, that had to be called by a bunch of other steps. This cleanup sequence is relatively complicated, and I’ve raised a number of PR’s to my colleagues which have come back with requests for change.
I’ll admit that I may have been a little sloppy with this change. But what’s not helping matters is the representation of Step Functions as a finite state machine written in YAML. I wrote about my issue with YAML used as some weird general purpose programming language, and this applies here as well. But a contributing factor is the level at which I’m defining this Step Function.fearing the sunk cost of the work I’ve done so far, I figured I’d just make the adjustments as they came. But now my patients has run out and I figured it was time to try a different approach.
The way to build an AWS Step Function is to define a finite state machine (FSM), which is basically a set of states linked together to form a graph. Doing this manually with a handful of states is relatively simple, but when you start to consider proper error handling and clean-up logic, such as making changes to DynamoDB records that need to be changed or reversed when something goes wrong, it can get quite complicated. The thing is that this is something that computers are able to do for a long time. A Turing complete program is a subset of a finite state machine, so if it’s possible to represent something that is Turing complete in a regular programming language, it should be possible to do likewise for a FSM with a subset of the language.
So I spent some time today trying to do that. My idea was to build a pre-processor which will take a Step Function definition encoded in the form that looks like a regular programming language, and translate it to the YAML FSM structure that AWS actually uses. Spoiler alert: I only got about half way through, but the seeds for something that could be used down the line are there, so it wasn’t a total waste of time.
The Design
There’s really nothing special about this. At this very early stage, the step function is simply written as if it was a regular programming language. The tool will read this, produce a graph representing the FSM, and generate the YAML file. An example of how this language looks is given below:
pass(name = "Pass1")
pass(name = "Pass2")
Here, we have a simple step function with two states, with one that will run after the other. Running it will produce the following YAML file:
Comment: This is a comment that is actually hard coded
StartAt: Pass1
States:
Pass1:
Type: Pass
Next: Pass2
Pass2:
Type: Pass
End: true
In this scenario, we’re using the Pass state, which will simply succeed, so this step function is pretty useless. But it does give a good demonstration as to what the eventual goal of the preprocessor is, which is to automatically do the wiring between the various states so that you don’t have to. If I were to put Pass2
above Pass1
, it will update the generated YAML file to reflect that, so that Pass2
will have Next: Pass1
, and Pass1
will have End: true
.
The usefulness of the tool comes when we start considering failure modes. These can be expressed as normal “try-catch” blocks, a lot like may languages that exist today:
try {
pass(name = "DoSomething")
} catch {
pass(name = "SomethingFailed")
} finally {
pass(name = "DoAlways")
}
This will produce the following Step Function YAML file:
Comment: This is a comment that is actually hard coded
StartAt: DoSomething
States:
DoSomething:
Type: Pass
Catch:
- ErrorEquals: ["States.ALL"]
Next: SomethingFailed
Next: DoAlways
SomethingFailed:
Type: Pass
Next: DoAlways
DoAlways:
Type: Pass
End: true
Once again, this is a relatively simple graph at this stage. But imagine this growing to several nested try-catch blocks, each one with slightly different error handling and cleanup logic. Manually keeping the various states wired correctly would be quite difficult, and it only makes sense to offload it to a tool to do this for us.
The Implementation
A few notes about how the tool itself was build:
- It was hacked together in Go, which is my language of choice.
- The parser was built using Participle, which is an excellent library for building parsers from annotated structs. If you work with Go and ever considered building a DSL, I would highly recommend this library.
- The YAML document is serialised using this YAML library.
The tool was a complete hack job so I don’t want to go too much into the design. But the general approach is the following:
- First the “program” is parsed from stdin and translated into an AST.
- This AST is then converted into an internal representation of the Step Function. This is effectively the graph, with all the steps linked together via pointers.
- The graph is then traversed and converted into a series of structs which, when serialised, will produce the YAML document that can be loaded into AWS.
The tool does not do everything. For example, choices are not supported, and only a few tasks that were specific to my problem were actually built.
So how did it go? Well, OK, I’ll come clean. I got as far as adding support for try-catch statements, but I never actually used it for the task I was working on. The task was nearing it’s completion and even though fixing the issues were annoying, it was less effort than regenerating the whole step function from scratch. But it’s likely that more step functions would be built in the future, most likely in the manner that we’ve been building them now: with error handling and clean-up states. So this tool might actually come in useful yet.
Thinking over the NaNoWriMo short story I’m working on, there seems to be a lot scenes with characters just sitting around and talking. It’s in danger of turning into a Star Wars prequel. π
Really enjoying Succession season 3 at the moment, but I’m afraid I’m one of those viewers that gets a little lost in all the business jargon. Fortunately, the Succession podcast really helps here, breaking down what actually happened in the episode.
Thought I’d give omg.lol a try this morning. My profile page. Looks a lot nicer than what I could come up with.
For the last month, I was having trouble installing software on my work laptop because I thought I lost admin privileges. But it turns out it was just because I was using my username in the password prompt, instead of my full name.
Why, Apple? Why did you not take my username?
Feeds In Broadtail
My quest to watch YouTube without using YouTube got a little closer recently with the addition of feeds in Broadtail. This uses the YouTube RSS feed endpoint to list videos recently added to a channel or playlist.

There are a bunch of channels that I watch regularly but I’m very hesitant to subscribe to them within YouTube itself (sorry YouTubers, but I choose not to smash that bell icon). I’m generally quite hesitant to give any signal to YouTube about my watching habits, feeding their machine learning models even more information about myself. But I do want to know when new videos are available, so that I can get them into Plex once they’re released. There is where feeds come in handy.

Also improved is the display of video metadata when selecting a feed item or entering a video ID in the quick look bar. Previously this would immediately start a download of the video, but I prefer knowing more about the video first. These downloads aren’t free, and they usually take many hours to get. Better to know more about them before committing to it.

Incidentally, I think this mode of watching has a slight benefit. There are days when I spend the whole evening binging YouTube, not so much following the algorithm but looking at the various channels I’m interested in for videos that I haven’t seen yet. Waiting several hours for a video download feels a little more measured, and less likely to send me down the YouTube rabbit hole. I’m sure there will still be evenings when I do nothing else other than watch TV, but hopefully that’s more of a choice rather than an accident.
I think this is enough on Broadtail for the time being. It’s more or less functional for what I want to do with it. Time to move onto something else.
Itβs a bit strange how the stopwatch on the NaNoWriMo site doesnβt automatically fill in start and end times. It knows when I start the stopwatch. It knows when I stop the stopwatch. And yet, I still need to enter these times myself.
Some days I wonder why I ever considered being a software developer. π
It’s said that it’s a good idea not to blog while your angry. I think that’s good advice. After all, there’s already a lot of anger out there and no-one needs any more.
But I need to vent somewhere. So watch out journal!
YAML and Accidental Programming Language Design
I’m not a huge fan of YAML in general, but I do see it being useful for situations when a structured configuration language is needed. Something for which JSON would normally be used, but where human readability and maintainability is important.
What I don’t like is seeing YAML being used as a way to define a sequence of actions. I complained about Step Functions, but just as annoying is the various CI/CD services that use YAML as the means of defining the build sequence. Encoding a sequence of actions β complete with branches, loops, callouts, and error handling β using a language geared towards configuration is clunky at best, and I’m always curious as to why vendors choose to use it when a language more fit for purpose would be nicer to work with. (This rant is going to be about YAML but using JSON in these situations is just as bad, if not worse. At least YAML has comments).
Obviously being a well known syntax that is approachable and already has a number of available parsers is a huge advantage. But I don’t think that’s the whole story, as there are plenty of JavaScript parsers out there as well. It might be necessary to tweak the runtime a little, which is not easy, but certainly not beyond the skills of Amazon and the like. No, I wonder if the desirability of YAML here is the ease of symbolic manipulation, and something that I’ll call “functionality inflation” (this makes it sound like a bad thing, but I guess another way of putting it is “serving the insatiable needs of the user”).
I’ll touch on latter first, as it’s probably the most sinister one. It all begins with the first release of the feature, which starts off being very simply: just a series of actions that are executed in order. Here too is the moment where you need to define how these sequence of actions are to be encoded. Since the runtime is non-standard (in AWS Step Functions, actions may take up to a year to be completed), and the set of encodable actions is relatively small, it seems like going with a language like JavaScript would be overkill. Better off to to go with something simpler, like YAML. You don’t need to build the parser; you simply need to define how the actions look in terms of arrays and objects, since that’s what a parsed YAML document would give you. You choose a representation β say as an array, with each action being an object β and ship the feature. Easy enough.
The problem is that it never remains this simple. As time goes on, and people continue to use your system, their expectations of the system begin to grow. It eventually comes to the point where the single list of actions is not enough. Now you need to consider actions that should only run when a certain condition is met, or actions that should run when another action fails.
So you extend your YAML language. You add guards to your actions which will only run when the condition is true, or start adding catch clauses to the actions to jump to another one when an error occurs. You begin defining bizarre expression languages on top of your expanded YAML representation that never look quite right, and special fields on actions with act effectively like goto’s. Eventually, you ship that, and for a while your users are happy, but not for long. They want variables, or actions that run multiple times or over a set of items, or the ability to call out to other actions in another workflow so they don’t have to copy-and-paste, and on and on. One day, you wake up and you realised that you’ve accidentally built a Turing complete programming language. The only problem is that it’s written in YAML.
So what’s the alternative? I’d argue something that resembles a proper, Algol-type programming language is a superior method of representing these workflows. There are many benefits of doing so: the logic is clear, one step follows the other as you flow down the page. There are already standard conventions for indicating branching, loops, error handling and call-outs. And there’s a lot less line noise as you’re not mixing your logic in with your data and configuration.
I mentioned JavaScript earlier here, but would using it, or another language, be useful here? Perhaps. They’ve got available runtimes as well, but they may not be easy to work with at the source level. This touches on the second reason why I think YAML is used, which is ease of symbolic manipulation. Given that YAML is just a glorified way to represent arrays and objects, one could argue that the advantages of using YAML is that it’s just as easy to manipulate with machines as it is from a text editor. You could, for example, build a graphical designers on top of your definition language which will manipulate the YAML while preserving any edits done by hand. This is something that’s difficult to do with an existing language like JavaScript or Ruby. Difficult, but not impossible. Plus, there’s nothing stopping vendors from designing a language which is high level enough to be easily manipulatable from machines. It just needs to be easily parsed and unparsed between the code and the AST without any information loss. This can be baked in as a top-level requirement of the language during the design.
In any case, it doesn’t need to be this way. We shouldn’t need to encode our logic using a configuration language. We already have better facilities for doing this: called programming languages. I hope that next time a vendor wishes to build something that has a feature like this, they consider designing one instead of using something not quite fit for purpose like YAML.
I’ve manage to get the first 1,000 words out of my NaNoWriMo-adjacent goal of a 10,000 word short story. I’m sure they’re pretty crappy words at the moment, but at least they’re on the page.
I really like to support independent software developers, but $150.00 for a file diffing tool is a bit much. The tool does a lot, like diffing images, but I don’t need any of that. If they release a text-only diff/merge version for like $30-50, I’d buy it in a heartbeat.
Some Screenshots Of Broadtail
I spent some time this morning doing some styling work on Broadtail, my silly little YouTube video download manager I’m working on.

Now, I think it’s fair to say that I’m not a designer. And these designs look a little dated, but, surprisingly, this is sort of the design I’m going for: centered pages, borders, etc. A bit of a retro, tasteless style that may be ugly, but still usable(-ish).

It’s not quite finished β the colours need a bit of work β but it’s sort of the style I have in my head.
I had the opportunity to go out for breakfast this morning. I was originally going to get it as takeaway, but it was such a lovely morning (if not a bit cold) and with some tables available outside, I figured “what the hell”. It’s nice to be able to do stuff like this again.
Well, it took almost a full 4 months, but I’ve finally got a solar system installed on my house. It’s not functional yet β it needs to go through inspection β but hopefully in a few weeks it should be active and generating power.
I was perusing my archives the other day and I’ve noticed that it’s been two years to the day since my first ever blog post. Also of note was just how infrequently I was posting back then β only 7 posts in 9 months. I’m glad blogging has developed into more of a habit now.
The popups for the admin password in MacOS needs a third button with the label “I wish I had the admin password, and believe me when I say that if I did, I would enter it; but I don’t, so please stop asking me.”