I feel envious of those that are happy to work on one thing β€” project, job, cause β€” for a lifetime. I’ve never felt that myself: after a while, I need to move onto something else. I wonder if it’s the same feeling one gets when they fall in love. Can’t say myself (I’ve never fallen in love either).

πŸ”— Doing Weeknotes

This is probably worth trying. I know for myself that I get lost in the day-to-day activities of work that I loose a sense on what we’ve accomplished during the week. Besides, Confluence has got this blogging feature which never seems to get used.

Dreamt that I was writing a blog post about software design and how it relates to eating vegetables. Not sure the analogy works so I won’t be writing it here. The setting was nice though: I was drafting it on a Swiss train pulling into Geneva where I was to meet up with family at the station.

I’ll bring the lighter fluid. #burn-all-computers-to-the-ground

DALL-E image of a pile of 90's desktop computer in a field set on fire.

(sorry, it’s been one of those days 😼).

This year’s Ides of March started with some stormy internet weather this morning. May have been behind a few production issues we had. Always a fun way to start a Friday. πŸ˜’

Internet weather’s fining up though.

Matt Bircher made an excellent comment about UI design sensibilities:

I get the feeling a non-insignificant contingent of commentators have taken this to the point where they think people who aren’t them are just dumb idiots who can barely figure out how a fork works, let alone how to use a piece of software that does more than one simple thing.

I get this sense as well. Of couse there will be those that arn’t interested in going beyond the basics. But what about those that are? Those that are willing to invest time and effort in your software? Surely they’re worth serving, no?

Oof, not feeling great today. Tired and spaced out. Hope I’m not coming down with something. 😩

Adding A Sidebar To A Tiny Theme Micro.blog

This is now a standalone Micro.blog Plugin called Sidebar For Tiny Theme which adds support for this out of the box. The method documented below no longer works, but I'm keeping it here for posterity reason.

I’d though I’d write a little about how I added a sidebar with recommendations to my Tiny Theme’ed Micro.blog, for anyone else interested in doing likewise. For an example on how this looks, please see this post, or just go to the home page of this site.

I should say that I wrote this in the form of a Micro.blog plugin, just so that I can use a proper text editor. It’s not published at the time of this post, but you can find all the code on Github, and although the steps here are slightly different, they should still work using Micro.blog’s template designer.

I started by defining a new Hugo partial for the sidebar. This means that I can choose which page I want it to appear on without any copy-and-paste. You can do so by adding a new template with the name layouts/partials/sidebar.html, and pasting the following template:

<div class="sidebar">
    <div class="sidebar-cell">
        <header>
            <h1>Recommendations</h1>
        </header>
        <ul class="blogroll">
            {{ range .Site.Data.blogrolls.recommendations }}
                <li><a href="{{ .url }}">{{ .name }}: <span>{{ (urls.Parse .url).Hostname }}</span></a></li>
            {{ else }}
                <p>No recommendations yet.</p>
            {{ end }}
        </ul>
    </div>
</div>

This creates a sidebar with a single cell containing your Micro.blog recommendations. Down the line I’m hoping to add additional cells with things like shoutouts, etc. The styling is not defined for this yet though.

The sidebar is added to the page using Tiny Theme’s microhooks customisation feature. I set the microhook-after-post-list.html hook to the following HTML to include the sidebar on the post list:

{{ partial "sidebar.html" . }}

In theory it should be possible to add it to the other pages just by adding the same HTML snippet to the other microhooks (go for the “after” ones). I haven’t tried it myself though so I’m not sure how this will look.

Finally, there’s the styling. I added the following CSS which will make the page slightly wider and place the sidebar to the right side of the page:

@media (min-width: 776px) {
    body:has(div.sidebar) {
        max-width: 50em;
    }

    div.wrapper:has(div.sidebar) {
        display: grid;
        grid-template-columns: minmax(20em,35em) 15em;
        column-gap: 60px;
    }
}

div.sidebar {
    font-size: 0.9em;
    line-height: 1.8;
}

@media (max-width: 775px) {
    div.sidebar {
        display: none;
    }
}

div.sidebar header {
    margin-bottom: 0;
}

div.sidebar header h1 {
    font-size: 1.0em;
    color: var(--accent1);
}

ul.blogroll {
  padding-inline: 0;
}

ul.blogroll li {  
  list-style-type: none !important;
}

ul.blogroll li a {
  text-decoration: none;
  color: var(--text);
}

ul.blogroll li a span {
  color: var(--accent2);
}

This CSS uses the style variables defined by Tiny Theme so they should match the colour scheme of your blog. A page with a sidebar is also wider than one without it. It doesn’t change with width of pages that don’t have the sidebar (if this isn’t your cup of tea, you can remove the :has(div.sidebar) selector off the body tag) and the sidebar will not appear on small screens, like a phone in portrait orientation. I’m not entirely sure if I like this, and I may eventually make changes. But it’s fine for now.

So that’s how the sidebar was added. More to come as I tinker with this down the line.

Blogroll ported to Micro.blog and placed in a sidebar on the post list screen using Tiny Theme Microhooks. I’ve yet to port the Blogroll page, and may trim some of the recommendations appearing in the sidebar, but not bad for a first pass.

Screenshot of lmika.org with the blogroll recommendations displayed as a sidebar

Spent the last two hours combing through logs trying to find the root cause of a problem, only to solve it by just restarting a random Kubernetes pod out of desperation. If this is not the modern day, software equivalent to bashing the side of your TV to get a picture, I don’t know what is.

🎡 Stationary Loops, by Lee Rosevere

Album cover of Stationary Loops

Bought when I needed a few screenshots of Alto but growing to like it. Quite nice ambient chill out music. Got hints of Tangerine Dream and Jene-Michel Jarre.

Alto

Date: 2020 β€” present

Status: Rockin'

The year was 2020. The pandemic was just beginning and I was stuck at home, not being able to do much of anything. Worse, rumours came around that Google was shutting down Google Play Music, my music player of choice. They were going to force everyone onto their streaming service instead. Oh, they may have a place for all the music you’ve downloaded (or written) yourself, but not in the first version. Maybe they’ll get to it later.

“Well, fuck that”, I said to myself. “I’ve been battered around by Google shutting down things and forcing migrations onto other things one to many times. I’m going build my own music app.”

And so I built my own music app: Alto.

Why “Alto”? Well, the name goes back to when I was in secondary school, when I was learning the viola, which uses the alto clef for it’s written music. I said to myself at the time that if I were ever to build a music player, it’ll have something referencing the viola. The alto clef seemed like the best thing to use. Plus, it stands out amongst the other music apps that tend to use other notation symbols like notes or the treble clef.

The Alto logo, taken mainly from the clef itself.

The idea for Alto was pretty straight forward: a music player and catalogue that’ll manage and stream music from an S3 bucket. No tracking, no “promotions” or “recommendations”, no bullshit UI that’s impossible to navigate. Only the music I’m interested in, played the way I want, and a dead simple UI that puts the album front and centre. The music player that was meant for me.

The Web Catalogue

The ultimate version that would come to be would consist of two parts: an Android mobile app, and a web-app. I’ll talk about the mobile app in the next post.

The web-app can be used as a player, but is ultimately be responsible for managing the collection.

The main album list.

The web-app was built using Buffalo, a rapid web development much like Rails for Go. It’s basically a simple server-side rendered web-app. The frontend consists mainly of Bootstrap plus some Stimulus and vanilla JavaScript to handle the interactive elements.
It’s also using Turbo to prevent unloading the current page when moving to a new one. This means clicking around the site will not stop playback of the current track, a very nice feature (and doubly so when you consider that this isn’t a single-page app).

Much of the UI is dedicated to managing the catalogue but there is also an integrated player, which can be invoked by clicking the play button. The player itself can be bought up at any time by pressing “P”. There’s no scrubber but there are seek by 30 second buttons which do the job. Each of the controls in this player have associated shortcut keys that are always available, even with the player hidden.

Clicking the play button in a track list will start playing it.

The collection is managed within a PostgreSQL database and referenced files stored on an S3 bucket. S3 was chosen to ensure that if I were ever to stop work on this project, or the database were to be corrupted, I wouldn’t loose my music. It does mean that I’ve got an ongoing cost for running this service, but based on the amount of music I’m keeping and my music listening patterns, the monthly bill for S3 is around 50Β’-60Β’ AUD, plus $12.00 US for the web-app server.

The catalogue model was made as simple as possible. The main construct is the Album, which would consist of zero or more Tracks. Albums had things like title, artist, cover images, etc. but these are nothing more than just properties of an album.

Tracks could be added one at a time via the frontend, or uploaded from a Zip file pulled from a URL (useful for songs bought on Bandcamp). The catalogue tries it’s best to avoid uploading media via the web-server, either opting to pull it in from the backend or upload it to S3 directly. This was a deliberate choice to reduce the amount of network it uses, but it does mean going through some strange loops. For example, when uploading a single track via the frontend, it would upload it directly to S3, then download it from S3 on the backend so that it can set catalogue metadata from the file itself (ID3 tags, MP3 length, etc). This is pretty convoluted and doesn’t even work half the time, and if I were making this again, I’d probably just bite the bullet and allow large uploads via the frontend.

Media

The media model is a little more complicated. Media β€” audio files, cover images, etc β€” all belonged to a Repository, which is essentially a reference to an S3 bucket, although it could also reference things like a HTTP domain. A repository does have some other configuration such as how to name the uploaded media files. I could’ve used something like a UUID, but I wanted to keep the names human readable as much as I could, so that if this project were to shutdown, I would still be able to access the files from S3 myself.

Listing all the media managed by the catalogue.

An Album or Track is linked to a Media record through what’s called a Media Reference. Each of these references has a “rel” property (short for “relevance”) which describes what the media is for: an audio file for a track, a cover image for an album, etc. There also exists a classifier which was to allow an Album of Track to use multiple Media records of a particular relevance. For example, some albums, released in different regions, had different album covers, with one being slightly darker than the other. The classifier could be use to switch between the two, based on whether Dark Mode was enabled. At least that was the theory: it never really got used for that.

Governing the link between albums, tracks and media was a simple resolution algorithm that supported things like inheritance. For example, tracks could have their own album cover, but if one didn’t exist, they would inherit it from the album. This went beyond just album covers: it would be possible, theoretically at least, to have the audio associated with the album and have the tracks reference a different cover image (I never tried this).

Finally, there are Playlists. These are pretty standard: just a collection of references to other tracks, plus some ordering information. Playlists and playlist items are essentially links and cannot have Media References themselves. There are some downsides to this: the biggest one being that Playlists do not have album cover art, which is something I’ll need to fix. Playlists can also have metadata items.

Playlists are out of the way from albums, and the UI for them is a little bit underbaked.

Speaking of metadata items.

Metadata Items

A number of objects can also have metadata, which could be use to attach extra attributes. The goal was to make this generic enough for end users to use it for whatever, with Alto having a few predefined names it uses for it’s own purpose.

There were actually two kinds of metadata record. The first was an arbitrary JSON structure that can be set for each Track.Β  Only one such name was reserved, called heading, which was used to define track groups within albums. I do have plans for adding more attributes, such as things dedicated to Eurovision tracks (year and country for example).

Listing metadata items. Very much undeveloped from the UI generated from Buffalo.
Listing a chapter metadata value.

The second, and slightly older, type of metadata was for larger and more generic bits of information. I initially envisioned this to be used to store things like additional artist names, but since each metadata item was a separate row, that felt like overkill and why I added the JSON object (not sure why I thought this, PostgreSQL is quite a capable database). The main use of this is to store chapter markers. Chapters are a pretty simple format, just a bunch of lines with a timestamp and name separated by an equals sign:

0=Intro
139.5=Outlaw
229=Crises
467.5=The Watcher and the Tower
599=Interlude
774.5=Sequencer
1111=Coda

I suppose I should find a way to extract chapters from the MP3 file itself but for the moment I’m just setting them manually. I am also using it to store things like lyrics. I should say a few words as to how Media Refs and Metadata Items actually reference other things in the model. They use a notion of “object_type” and “object_id” which describes what the object is (album, track, playlist item, etc.) and object ID referencing the actual object itself. Because this is quite generic, I can’t rely on ON DELETE CASCADE to clean these up, so I opted for database triggers to remove media refs and metadata items when the base object type is removed.

CREATE FUNCTION delete_dependencies_of_object() RETURNS trigger AS $$
BEGIN
  DELETE FROM media_references WHERE object_id = OLD.id;
  DELETE FROM metadata_items WHERE object_id = OLD.id;
  RETURN NULL;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER delete_dependencies_of_tracks
AFTER DELETE ON tracks
FOR EACH ROW EXECUTE PROCEDURE delete_dependencies_of_object();

CREATE TRIGGER delete_dependencies_of_albums
AFTER DELETE ON albums
FOR EACH ROW EXECUTE PROCEDURE delete_dependencies_of_object();

So far I haven’t had any issues with this, although I haven’t done a lot of snooping around the database to confirm these records are properly being cleaned up.

Grand Plans

I did have grand plans for this catalogue at one point: releasing it as open source, maybe making this a service that others could use. So there were a few things added which are unfinished and half baked. Some examples:

  • Having multiple catalogues with different access roles.
  • Feature flags, which disabled certain features for users.
  • Generating a QR code for an API token for easy sign in on the mobile app.

None of these really went anywhere, and if I were to rebuild this, I’d probably pull them out. As things stand now, it does need a bit of a refresh: upgrading Go packages, dealing with Node packages (ugh, it’s always a pain trying to update JS packages and Webpack). But since around late 2020, it’s been serving quite well as my primary music player.

Hmm, this is a new one. Apparently Telstra is blocking the site to Whisky app for some reason. Going to the website gave me this:

Screenshot of a browser showing a certificate error for the site getwhisky.app

Suspecting that the Whiskey devs just got their certificate missconfigured, I tried viewing the source using curl --insecure. Here’s what I found:

Screenshot of Terminal showing the output of curl insecure, which includes the quote below

We blocked this website because it may contain malicious content and could be harmful to your devices.
You haven’t done anything wrong. We automatically block potentially dangerous websites. We recommend you leave this website.
[…]
Copyright Β© 2023 Telstra

I’ve submitted a request to unblock it just now. I’m suspect this to just be a mistake but I am curious to know how it got blocked in the first place.

I did manage to get Whisky via a VPN.

Photo Bucket Update: Exporting To Zip

Worked a little more on Photo Bucket this week. Added the ability to export the contents of an instance to a Zip file. This consist of both images and metadata.

Screenshot of a finder window showing the contents of the exported Zip file

I’ve went with lines of JSON file for the image metadata. I considered a CSV file briefly, but for optional fields like captions and custom properties, I didn’t like the idea of a lot of empty columns. Better to go with a format that’s a little more flexible, even if it does mean more text per line.

As for the images, I’m hoping the export to consist of the “best quality” version. What that means will depend on the instance. The idea is to have three tiers of image quality managed by the store: “original”, “web”, and “thumbnail”. The “original” version is the untouched version uploaded to the store. The “web” version is re-encoded from the “original” and will be slightly compressed with image metadata tags stripped out. The “thumbnail” version will be a small, highly compressed version suitable for the thumbnail. There is to be a decision algorithm in place to get an image given the desired quality level. For example, if something needed the “best quality” version of an image, and the “original” image is not available, the service will default to the “web” version (the idea is that some of these tiers will be optional depending on the need of the instances).

This is all partially working at the moment, and I’m hoping to rework all this when I replace how stores and images relate to each other (This is what I’m starting on now, and why I built export now since this will be backwards incompatible). So for the moment the export simply consists of the “web” version.

I’ve got unit tests working for this as well. I’m trying a new approach for unit testing in this project. Instead of using mocks, the tests are actually running against a fully instantiated versions of the services. There exists a servicestest package which does all the setup (using temporary directories, etc) and tear down of these services. Each individual unit test gets the services from this package and will run tests against a particular one.

This does mean all the services are available and exercised within the tests, making them less like unit tests and more like integrations tests. But I think I prefer this approach. The fact that the dependent services are covered gives me greater confidence that they’re working. It also means I can move things around without changing mocks or touching the tests.

That’s not to say that I’m not trying to keep each service their own component as much as I can. I’m still trying to follow best practice of component design: passing dependencies in explicitly when the services are created, for example. But setting them all up as a whole in the tests means I can exercise them while they serve the component being tested. And the dependencies are explicit anyway (i.e. no interfaces) so it makes sense keeping it that way for the tests as well. And it’s just easier anyway. 🀷

Anyway, starting rework on images and stores now. Will talk more about this once it’s done.

I’ve being reading several posts from Aaron Francis recently around doing work and talking about it in public. All really good posts on the topic, particularly this one I’ve read a while ago which still resonates with me today.

Anything worth doing is worth doing badly.

G.K. Chesterton

Best quote from this video by Aaron Francis.

It’s always better to go outside if you can, even if it’s just for a bit. You may feel crappy, and feel resistant to do so. And it’s not like going outside will solve the problems you’re facing. But you will feel better, or at least you won’t feel worse.

People say AWS is expensive, but somethings it’s worth it. We’ve been seeing one of our pods crash randomly the last few nights because it’s loosing connection to a self-hosted NATS cluster. I’m pretty sure we wouldn’t be experiencing this if we just used SNS and SQS.

Enjoyed the discussion about AI and independence on Ruminate #179. I agree with Robb and John: websites created by people will definitely have a leg up amongst all the sites that are little more than AI generated “content”. πŸŽ™οΈ

Photo Bucket Update: More On Galleries

Spent a bit more time working on Photo Bucket this last week1, particularly around galleries. They’re progressing quite well. I’m made some strides in getting two big parts of the UI working now: adding and removing images to galleries, and re-ordering gallery items via drag and drop.

I’ll talk about re-ordering first. This was when I had to bite the bullet and start coding up some JavaScript. Usually I’d turn to Stimulus for this but I wanted to give HTML web components a try. And so far, they’ve been working quite well.

The gallery page is generated server-side into the following HTML:

<main>
  <pb-draggable-imageset href="/_admin/galleries/1/items" class="image-grid">
    <pb-draggable-image position="0" item-id="7">
      <a href="/_admin/photos/3">
        <img src="/_admin/img/web/3">
      </a>
    </pb-draggable-image>
        
    <pb-draggable-image position="1" item-id="4">
      <a href="/_admin/photos/4">
        <img src="/_admin/img/web/4">
      </a>
    </pb-draggable-image>
        
    <pb-draggable-image position="2" item-id="8">
      <a href="/_admin/photos/1">
        <img src="/_admin/img/web/1">
      </a>
    </pb-draggable-image>        
  </pb-draggable-imageset>
</main>

Each <pb-draggable-image> node is a direct child of an <pb-draggable-imageset>. The idea is that the user can rearrange any of the <pb-draggable-image> elements within a single <pb-draggable-imageset> amongst themselves. Once the user has moved an image onto to another one, the image will signal its new position by firing a custom event. The containing <pb-draggable-imageset> element is listening to this event and will respond by actually repositioning the child element and sending a JSON message to the backend to perform the move in the database.

A lot of this was based on the MDN documentation for drag and drop and it follows the examples quite closely. I did find a few interesting things though. My first attempt at this was to put it onto the <pb-draggable-image> element, but I wasn’t able to get any drop events when I did. Moving the draggable attribute onto the <a> element seemed to work. I not quite sure why this is. Surely I can’t think of any reason as to why it wouldn’t work. It may had something else, such as how I was initialising the HTTP components.

Speaking of HTML components, there was a time where the custom component’s connectedCallback method was being called before the child <a> elements were present in the DOM. This was because I had the <script> tag in the the HTML head and configured to be evaluated during parsing. Moving it to the end of the body and loading it as a module fixed that issue. Also I found that moving elements around using element.before and element.after would actually call connectedCallback and disconnectedCallback each time, meaning that any event listeners registered within connectedCallback would need to be de-registered, otherwise events would be handled multiple times. This book-keeping was slightly annoying, but it worked.

Finally, there was moving the items with the database. I’m not sure how best to handle this, but I have that method that seems to work. What I’m doing is tracking the position of each “gallery item” using a position field. This field would be 1 for the first item, 2 for the next, and so on for each item in the gallery. The result of fetching items would just order using this field, so as long as they’re distinct, they don’t need to be a sequence incrementing by 1, but I wanted to keep this as much as possible.

The actual move involves two update queries. The first one will update the positions of all the items that are to shift left or right by one to “fill the gap”. The way it does this is that when an item is moved from position X to position Y, the value of position between X and Y would be changed by +1 if X > Y, or by –1 if Y > X. This is effectively the same as setting position X to X + 1, and so on, but done using one UPDATE statement. The second query just sets the position of item X to Y.

So that’s moving gallery items. I’m not sure how confident I am with this approach, but I’ve been testing this, both manually and by writing unit tests. It’s not quite perfect yet: I’m still finding bugs (I found some while coming up with these screencasts). Hopefully, I’ll be able to get to the bottom of them soon.

The second bit of work was to actually add and remove images in the gallery themselves. This, for the moment, is done using a “gallery picker” which is available in the image details. Clicking “Gallery” while viewing an image will show the list of galleries in the system, with toggles on the left. The galleries an image already belongs to is enabled, and the user can choose the galleries they want the image to be in by switching the toggles on and off. These translate to inserts and remove statements behind the scenes.

The toggles are essentially just HTML and CSS, and a bulk of the code was taken from this example, with some tweaks. They look good, but I think I may need to make them slightly smaller for mouse and keyboard.

I do see some downside with this interaction. First, it reverses the traditional idea of adding images to a gallery: instead of doing that, your selecting galleries for an image. I’m not sure if this would be confusing for others (it is modelled on how Google Photos works). Plus, there’s no real way to add images in bulk. Might be that I’ll need to add a way to select images from the “Photos” section and have a dialog like this to add or remove them all from a gallery. I think this would go far in solving both of these issues.

So that’s where things are. Not sure what I’ll work on next, but it may actually be import and export, and the only reason for this is that I screwed up the base model and will need to make some breaking changes to the DB schema. And I want to have a version of export that’s compatible with the original schema that I can deploy to the one and only production instance of Photo Bucket so that I can port the images and captions over to the new schema. More on this in the future, I’m sure.


  1. Apparently I’m more than happy to discuss work in progress, yet when it comes to talking about something I’ve finished, I freeze up. 🀷 ↩︎