Devlog on Leon Mika

Devlog: Imagen

Sun, 26 Apr 2026 11:18:46 +1000

I won’t bore you with any justification on why I actually built this thing. It’s really nothing more than wanting a harness to play with Google’s Nano Banana. Up until now, I’ve been using Blogging Tools for that, but it lacked any ability to request changes to images in a chat-like interface (so called “multi-step flows”), where the context is preserved. So I made Imagen as a replacement.

The landing page, showing current chat sessions and 'assets' (uploaded and generated images).

An example of a chat session, showing the meta-commands, plus messages from the user and model.

Clicking 'New Chat' will show you this: a prompt window along with a way to upload images.

Chat management section, allowing you to archive or delete any chats.

Clicking the gear icon brings you to the admin section, allowing you to revoke passkeys and end your session.

This is a pretty typical chat-based AI model harness. You start a chat requesting an image, maybe uploading one you want modified, click “Send”, and wait for the model to respond. There also exist a bunch of meta commands, entered using the / prefix, that allow you change the model, and retry or undo the last request.

Chats and assets are stored as regular files (assets as regular image files, chats as a Gob-serialised history of the chat) and metadata is managed by a Sqlite3 database. In terms of the tech stack, I’ve settled on a mix that works well for me. A Go backend, using Fiber as the web middleware, and Go templates for server-side rendering. I eschewed Stimulus.js for this one though, deciding to try out HTMX. And it’s a unique way of working. I learnt that to really get the most out of HTMX, you really need to find ways not to use JavaScript to do something.

The chat interface is a good example. While a more traditional frontend would post a message as JSON and rerender the page with a JSON response, posting a chat message here would return a server-side HTML render of the chat interface, which HTMX will swap out. This also include things like the message form itself, which would be rendered disabled from the server template, rather than being controlled by JavaScript. The good thing about that is that if you were to hard-refresh the page, it will appear disabled. No need to run JavaScript on load to detect the state and disable the form: the HTML is the state. As for getting updates while waiting for a message, well that’s just a poll. Nothing too sophisicated here.

As you can probably guess, much of this was hand-rolled. I did use coding agents to assist with making the passkey authentication flow, the meta commands, and the OpenAI API integration. While I probably could’ve done the last two myself, if there was any justification for using coding agents for anything it was probably adding passkeys. The complexity involved with the passkey integration probably justified the use . And this is probably one area of AI-generated code that I’m more likely to trust the model over my own shoddy code.

Devlog: Other Project Updates

Sun, 29 Mar 2026 10:17:00 +1000

A few updates of some other projects I worked on recently.

Webtools saw some love as I needed some tooling made to make the icon easy to include in the Well Read Flutter project itself. Android expects the logo of a specific size, so I “commissioned” an Android Icon Resizer, which will take one or more PNG files, resize them to what Android expects, and prepare them in a ZIP that could be extracted at the route of the res/mipmap directory. It will also produce a small preview of the icon, rendering it in a circle so you can see how it looks on the device. It’s background savvy, layering the icon over the PNG with “background” in the filename.

The Icon Resizer tool.

The background for Well Read is a gradient, so I also added a simple gradient maker to Webtools too. Finally, there was a need to resize the logo without resizing the image canvas size itself so that it looked good in the cut-out circle. This required the commission of the image inner resize tool.

The gradient tool.

It’s true I could’ve used any image editor to do this, but none of them were running and Claude was, so I figured it was easier doing it this way at the time.

I did get my hands a little more dirty with UCL. I was making some changes to Dequoter to allowed users to perform UCL transformations, and I needed a way for the user to reference the current line. UCL has pseudo-variables, so @line was possible here. But I was finishing of a Go template transformation, and much like those templates, I was wondering if a simple dot could work: add . 2, for example. This was a little larger than expected, as it did require changes to the AST, but it was simple enough to knock out in an hour. The host sets the value of dot via the pseudo-variable ., and it even supports direct sub-indexing at the syntax level: add .item.(0) 2.

Oh, and Dequoter now allows actions to specify input values. Here it is being used for a new UCL transform action, which will map each line of the input to the result of a UCL expression.

The action input prompt.

So, quite a busy week few weeks. Not sure what I’ll work on next. I do have an inkling of going back to that Godot project that’s been on the back-burner for about a month.

Devlog: gRPC Client - A Vibe-coded Client for Testing REST/gRPC Endpoints

Sun, 29 Mar 2026 09:56:00 +1000

My current craving of vibe-coding various tools I need to do my job continues, with an attempt to build a REST/gRPC test client.

This is motivated by my distaste with all the other clients I’ve tried. There’ve been a few, and I’ve been unhappy with each one. For one thing, they seem more heavyweight than my needs. I don’t know if this is just how they’re implemented, or it’s because the realm of HTTP request testing is complicated (It’s probably a bit of both).

Another thing is that they’re tend to add features targeting testing teams that manage a central repository of test cases. I get the feeling that’s how most of them try to make money: offer the tool for free to get the foot in the door, then upsell them with a hosted test repository. This leaves half the interface either useless, or just bait for throwing up requests to setup a service account and pay them money.

Finally, some are just awkward to use. The UI is inconsistent and buggy, and just gets in the way. I did wonder if the typical UI for these REST clients is partly to blame for this, but as you will see, it’s not like this project deviates from the mainstream, so I think it’s just comes down to implementation.

Anyway, these frustrations, along with wanting to see what these coding agents are capable of, motivated me to requested Claude Code to make me one. And it did a pretty decent job.

Main client area of gRPC Client.

It’s called gRPC Client — stellar naming, I know — and it’s pretty typical of most other REST/gRPC test clients out there. Request details on the left, the latest response on the right. A list of saved requests, along with a history, in the sidebar, that can be recalled and resent. There’s a reflection browser for gRPC endpoints, and a command for producing an example request for service methods, a feature of Insomnia that I really like.

The right sidebar has three parts, of which the first is the ability to define environment values, which can be included in the request properties via Go templates (this is a Wails app with a Go backend and HTML+JS frontend).

Editing environment values. This modal is only really useful for large values, like JWT tokens, but values can also be modified from the sidebar itself.

One feature of this client is the ability to override HTTP headers at the global level, the second thing in the right sidebar. When testing APIs, I usually have a collection of pre-made requests for endpoints I’m familiar with. Yet I’m always editing the headers when the user or auth token changes. The motivation with this feature was to make the change once, so that I don’t need to do so when I switch saved requests. I originally requested a way to do this for the hostname and port too, but I found this wasn’t as useful for me.

In practice, I ended up simply using environment values and request scripts, so-called “Hooks”, which is the third thing in the right sidebar. This is some JavaScript that executes before each requests, allowing the user to modify the headers and body using the values from the environment. What I would do is set the username as an environment value, and a hook would set the HTTP header, and generate the payload for a JWT token, saving me from getting this from the browser (the endpoints I need to test just need the payload, not the signature, so this simply involvers marshalling a JSON object to Base64).

Editing a hook. The placeholder gives a good example of what is supported.

These are all features of most REST clients out there, but they’re always hidden away and difficult to modify, which means I’m never using them. Placing them in the main client area makes them front and centre, so to speak, and so much easier to change.

One other thing I requested was the use of HuJSON for the request body. It’s a little crazy to think that all these REST/gRPC clients, designed for use by humans, require their users to write perfect JSON for the body, especially when I’m left wanting to comment out parts of the body as I test something. This gets translated to JSON when the request is being sent.

So all in all, it’s a pretty decent client, and despite it being like many other clients out there, it’s pleasant to use. And I can’t really put my finger on why. Might be that it’s just simpler than all the other ones out there. It doesn’t have tabs, for example, which does making testing multiple endpoints a little annoying, and I’m wondering if it’s worth asking Claude to add them. Yet at the same time, the frustration I had with other clients came about with their implementation of tabs. Likewise for saved requests. This just has a single list of saved requests that are either REST or gRPC requests. No need to worry about folders, or request types I don’t use, like GraphQL.

I guess if you’re vibe coding something for your own use, less is indeed more.

Devlog: Weiro - Categories, Pages, and Upload Editing

Sun, 29 Mar 2026 08:53:00 +1000

Some more work on Weiro. Much of it is pretty mundane, mainly to get it to feature parity with other CMS’s out there. Yes, I know the existence of those other CMS’s make the entire project pointless. Doubly so when you consider that much of what I’m going to talk about was largely done by coding agents. It made me wonder whether it was worth writing this update at all. Well, it’s drafted up already so I may as well finish it off. At least one thing will get finished. Okay, enough wallowing in my self-doubt. What was added? Well, categories are now a thing. These can be defined in a new categories section and consist of a label, a slug, and a description. Going to /categories/<slug> will list all the posts with that category. Posts can be in zero or more categories, which can be selected from the edit post screen. Pretty simple stuff.

The Categories section, where the user can manage categories.

Editing a category.

Selecting categories from within the edit post screen.

The categories page as it appears on the published site.

The single category page.

Another small thing added was pages. This allows the user to define “slash” pages, with the option of appearing in the nav bar, and can also be used to replace the home page, by setting the slug to / (the posts are still available at /posts). I do need to spend some more time figuring out how to organise the nav bar as I would also like to include things like redirects. I thinking of making that an extension of the pages model, but I’m not sure.

List of pages of a site.

Editing a page. There’s only a single page type at the moment.

Those two features were mainly done with the help of Claude Code, but I did build some stuff manually. The largest addition was the ability to do some edits on uploaded images. I’ve never been a fan of how Apple produces the shadows around windows: the margins are just too large. So I added a way to do this within Weiro itself.

The edit upload feature.

It’s pretty simple. Just imagine the filter section from any image editor, then remove all the other features of that editor. Yeah, it’s that simple. There’s no cropping, rotating, or anything else of that nature: just a bunch of “processors” that you can add to the image, with the sole one being a drop shadow. This is actually done server side using a simple file-based approach. When opening an image upload to edit, you spawn a session. Each session has a JSON file maintaining the processors, plus a series of cached image files of all the intermediate steps along the processing chain. When a new processor is added, a hash is computed with the processor’s properties, and if they change, the cached image will be regenerated. The processor includes the hash of the previous step too, so that if processors further up the chain are modified or remove, they will force a recompute of subsequent images. The user is then served the last image in the chain. This differs from the image processor in Blogging Tools, which performed the transformations within the browser itself. The motivation there was to avoid the need of a slow upload of the image, but it came with the performance cost associated with doing the processing within the browser itself. WASM might be fast, but it’s not that fast[^fast]. Since the upload in Weiro is already uploaded, I figured it would be quicker to just do the image processing server side. My hope is that the computing power the server has access to would offset the time it takes to download the image. So far this seems to be the case. Finally, I did some minor work around the UI, adding a site chooser and a much needed way to open the published site from the admin section. So this project is still coming along, surprisingly. It’s probably the furthest I’ve got in a blogging CMS that I actually want to use. I do have a large list of things I want to add to it, and I certainly need to do something with the design of the actual site. It’s all a question of whether I’m interested in spending time on it. [^fast]: Although to be fair, I think the slow down comes from encoding the processed image as a data URI and setting it as the source of an img tag.

Devlog: Well Read - One Week Later

Tue, 17 Mar 2026 20:34:00 +1000

It’s been a week since I learnt about Inkwell’s API and got an agent to start work on an RSS reader. Since then, Well Read has been in a state of flux, as I ask for agents to make changes to the interaction and layout. I think I’ve got it in a pretty good state now, certainly in a state that works well for me.

The earliest screenshots of Well Read, taken on 11 March 2026

The Today and Recent tabs still retain their original behaviour, although the idea of using 6:00 pm of the previous evening as the cutoff for “todays’ posts” was a good one. Being where I’m located in the world, many people I follow publish posts while I’m asleep. I recently added a tab for bookmarks which uses Micro.blog’s bookmarking feature. Any post can be added as a bookmark, and since there’s no real way to get to a post once it drops out of the Recent tab (well, apart from opening it up in Inkwell proper), I have in mind this feature to act as a place to stash posts for later. So I tried making bookmarks as easy to add as possible. Sliding aside the item in the feed list will reveal a bookmark action, as will the overflow menu in the feed viewer.

The app now has an icon too, which was made with the help of ChatGPT. I haven’t consulted the agents about a better name though.

App logo of Well Read

I did get my hands a little dirty to try and speed up the Webviewer. I think I made some success with removing the max-resizing from the viewport meta tag aded to the HTML the Webviewer opens. It could be psychosomatic, though. It feels a little different after I added it, but only if I’m using the actual phone. I see no performance issues in the simulator. I even tried comparing two screen recordings I made before an after the change, but I couldn’t see a different. Since then, the scrolling has become choppy again. Sigh. That’s one downside with choosing Flutter for this: you’re bound by the speed of the language runtime. Maybe if I got the coding agent to port it to plain Android… 🤔

Screenshots of Well Read taken this evening

But the proof of any app is in the pudding: am I actually using it to read my RSS feeds? And the answer is yes, I am. It doesn’t do everything a full-featured RSS app does, but what it does it does reasonably well — jankiness Webviewer scrolling aside — and I find myself turning to it at lunch or during my commutes. And even though there aren’t any unread counts anywhere, it’s slightly ironic to find that this river-based pattern has turned me into a bit of a completeness, wanting to get through today’s feed before they drop over to recents. Oh I still save the odd post by keeping it marked as unread (the “Mark as Unread” action is optimised for this, in that it actually kicks you back to the post list when you use it), but not a frequently as I have been.

Anyway, my need for changes for this app might settle a little as I just go about simply using it for a while. But with the use of these agents, it’s pretty amazing what they can be accomplish in a week.

Devlog: Well Read - An Inkwell Client for Android

Thu, 12 Mar 2026 06:22:00 +1000

When Manton mentioned that Inkwell has an API, I… um… may have vibe-coded an Android client.

It’s called “Well Read”, which is not a great name but better than “Inkwell Client” which was the working title. Much like Inkwell, it follows the river approach to RSS. There’s a Today tab and Recent tab, each showing a portion of the entries in reverse chronological order. Today shows all the posts from today, plus the last 6 hours of yesterday. The motivation here is that this will be the tab you’ll be spending your time, with posts aging out to recent over time. There’s no Fading tab: all posts older than about half a week fall out of the app. And I’m not sure if I’m going to add one. There’s only a few feeds that I want to catch up on if I miss their post, and since I’m using NetNewsWire and Inkwell, I’m pretty sure I’ll catch them later.

At the moment this is pretty bare bones. There’s no bookmarks (yet), you can’t add feeds, there’s no search, and only a few actions you can perform on a post: copy the link, mark it as unread, and open it in the external browser. The webview itself is a little janky too, which is not great, and I need to do something about that as it does impact the reading experience.

But so far it’s been pretty good. Feels the time when all I have is a phone to read stuff, like during lunch or while commuting.

Devlog: Weiro - Update 6th March 2026

Mon, 09 Mar 2026 09:48:00 +1000

A small update on Weiro. I’ve been working on it over the past week, trying to get it in a state that is pleasant to use. I’m been trying to get something halfway usable before doubt scuppers my motivation and this project appears on the growing list of aborted attempts at making a CMS. There’ve been one or two close calls, but it hasn’t caused me to stop yet.

A large part of that was a feature I knew I wanted but was daunting to implement: uploads. The thought of writing the logic to manage large files, make sure EXIF data is stripped, and serve and manage them always seems like a pain. It’s the reason why I’ve abandoned CMS projects in the past. And I want something that support uploads: I’ve tried CMSes that didn’t have them and I never stayed long.

But this time, I rolled up my sleeves, cracked open Claude, and told it to… no, I’m kidding. Much of the feature was hand rolled, although Claude helped with some of the plumbing of getting uploaded files. But it’s done: uploads are now supported. Which means screenshots:

Main post screen, with this post shown as draft. The 'rebuild site' is temporary at the moment.

The post editor. Pretty basic at the moment.

The uploads list.

A single upload, allowing the user to copy the HTML to paste in the post.

Along with this are some updates to the published site. This I’m hand-rolling too, rather than relying on an existing SSG. Probably a mistake in the long run, but it does keep things flexible. There’s nothing flashy about the HTML version of the site right now. I’m using Kev Quirk’s awesome Simple.css, that I use as a base style for most things nowadays. But I have added an RSS and JSON feed. Two, in fact. One is a standard RSS feed for the site, and another is designed from crossposting to Micro.blog. The main difference between the two is that the crossposted version will have “Devlog” prepended to each of the post titles, as they’ll be imported as full posts in a blog that has other content.

This is all hard-coded at the moment but I’m hoping to extend this to support other sorts of feeds, allowing the author to modify, include, and exclude posts based on their desires. The motivation here is to make it easier to integrate with services that cross-post, while recognising that not all of these “views” of a post would be the same. Cross-post to a service that supports at-mentions and maybe you’d like to ensure their handle is properly written out. But display it on the site, and maybe you’d prefer that the handle is a link to their homepage. These should all derived from the post source, and expanded in the various feeds based on who’s consuming it.

Anyway, that’s the idea. We’ll see if any of this comes about.

Devlog: CSVTool - A Vibe-coded CSV Editor

Sun, 08 Mar 2026 06:11:00 +1000

One of the fun aspects of these new code agents is seeing what they’re capable of producing just form the prompt, so called “vibe-coding.” There are some that are definitely all in on the concept: I’m thinking of Steve Yeggie and his Gas Town work. As for myself, I still prefer to be a bit more hands on. But it’s still amusing to see what these agents are capable of just from the prompt.

I got inspired by Dave Winder’s post about how he asked Claude Code to make a spreadsheet app for him. I use a TUI app I made for myself called Ted that I use to edit CSV files, but seeing how good Claude was in making the spreadsheet, I thought I’d ask Claude to make me a GUI version. This is what it came up with:

The results were pretty decent, at least on the surface. I haven’t put it quite through it’s paces for editing large CSV files, but what it managed to do out of the box was pretty impressive. Not that it’s groundbreaking in any technical sense: think spreadsheet without the ability to define expressions. Most of the supportable commands are available via a command palette, invokable using Cmd+P. There are a few copy-and-paste options for getting table data out into usable formats, something I find quite frustrating with the spreadsheets I use. You can paste table data as CSV, a Markdown table, and Jira markup. There’s the usual load, save, and commands to insert rows; all pretty standard.

It isn’t perfect though. Despite two attempts to instruct it to make the header row fixed such that it won’t scroll off the top, Claude was unable to achieve this. The techniques of making such a header are pretty hacky: the one I’m aware of is making a second table with just the header, placing it above the first, setting the position to absolute, then adding some JavaScript to keep the column widths in sync. Granted, this worked a good 18 years ago, and for tables that didn’t scroll horizontally. Maybe there’s a better way of doing this nowadays? Oh and speaking of the columns, despite providing affordances for resizing the columns, Claude didn’t implement the code to actually resize them. This makes me wonder if HTML tables are probably not the best approach for such a control. I guess I probably should’ve told Claude that.

In any case, I’m not sure I’ll put this to any real use. One thing I like about Ted is that it’s modal, like Vim. Being able to move around using the keyboard (I, J, K, and L are mapped to up, left, down, and right, respectively) is something I’d miss. So I think Ted will still be useful for me. I may revisit this in the future though, and see if Claude could make a GUI version of Ted. That could be interesting.

I’ll finish up by talking about some of the other features I asked to be added. Some of them were pretty ill conceived, as they came while I had a flurry of needs for CSV data. I ended up using to tool more frequently than I expected.

Sorting: Sort the table (the actual model, not just the view) either alphanumeric ascending, or alphanumeric descending based on the values of the current cell. There was also a “Sort Advanced” that prompted the user to enter the columns to sort in priority order (although not the direction, it’s always alphanumeric ascending).

Match Cell: Select cells that had a value matching the current cell. This was an attempt of selecting rows to delete, but it never really worked well.

Set Where: Prompts the user for a source column, a set of match values, and a target value. Whenever the source column has one of the match values, it will set the current column to the target value. This acts like a poor man’s inner join, and was made as I needed to interleave values I had from a shell command in the table I was working on.

Devlog: 3rd March 2026

Tue, 03 Mar 2026 06:15:00 +1000

Oof! Everyone’s building blogging CMS’s now, apparently.

Since starting work on this project, I saw one other announce their own CMS that was vibe-coded with Claude. No shame in that: making something that works for you is part of the joy of participating in the Indie-web. I did take a brief look at it, and dismissed it because it was written in PHP. Yes, I am a snooty developer that looks down on those using PHP (it’s just so annoying to deploy; although credit to this person, they did prepare a Docker container).

Weiro is not vibe coded. Much of it is written by hand. Not all of it, mind you: I am using agents to help with the more mundane stuff. But the models, DB schema, and much of the UI is hand-rolled. And I’m conflicted as to whether that’s the right balance. Progress is slower: these vibe-coders are whipping up CMS’s in the same time it takes me to add a single feature to it. And there are probably better things I could be doing other than adding one more CMS into the world (although when my mind whispers that doubt, it usually suggests watching TV or doomscrolling as an alternative, so there are certainly worse things I could be doing).

I haven’t stopped working on it yet, so here’s a brief update. I’ve got a version of it up and running in Coolify. It currently supports posts at the moment: both draft and published via Netlify. Much of the work was just making the writing experience feel natural, given that working on posts is probably the core feature of any CMS. So there’s a very large window for the post body (maybe a little too large), and there’s a Cmd+S keyboard shortcut for saving updates. I would like to add an auto-save feature, but I’m not entirely sure how best to do that server side, so I may settle for something that’s browser only, just to save work for when the browser crashes or has no network connection. I’m also working on uploads, so there shouldn’t be too much time before I can start sharing screenshots.

Devlog: Hello

Sat, 28 Feb 2026 10:28:24 +1000

This is the inaugural post of Devlog, where I'm planning to write on what I'm working on. This post was created using Weiro, a new blogging CMS I've been working on. This post is little more than a test to see if the deployed version of Weiro is working. I'll post more about Weiro in later posts. For now, I just want to make sure this is being published correctly.

Devlog: Micro apps - Some Score Cards

Sun, 01 Feb 2026 09:05:02 +1000

In the spirit of maintaining a document of what I’ve been working on, and being somewhat inspired by Matt Birchler’s posts about his micro apps, I’d thought I’d document on some of the small apps I’ve worked on recently.

The fact of the matter is that I’ve been building quite a few of these apps over the course of the summer, primarily in response to a specific need I have at the time. I haven’t written about them before, mainly because there didn’t feel like there was much to say. Some were vibe coded, and saying that I “made” them didn’t feel correct. Others were made by hand, but they were super simple and there was no real challenge in making them at all. In either case, being able to say “I made it” is difficult as the amount of effort spent in making it is quite low, and that doesn’t make for interesting blog posts. Would a post saying that “I went to work today” be worth reading if that’s what I do every weekday?

But regardless of what my feelings are, the scales were tipped ever so slightly into wanting to write about them. So that’s what I’ll do. And there is some leeway in writing about the small things if I were to put them in the Devlog category, which could be seen as my public workbook.

So, let’s set expectations by talking about a series of score-keeping apps. These are among the simplest micro-apps I have, yet they do have their uses. You’re always left high and dry with a need to keep score of some ad-hoc game made in the moment (or, at least, I am). Notebooks and spreadsheets could work here, but I don’t carry a paper notebook, and use a digital notebook for this is always a little cumbersome. And a spreadsheet just feel a little overkill. Hence, the want for having a simple web-app for keeping scores:

Using it couldn’t be simpler: enter the score under the specific player, and it will be recorded in a table, alongside a running total. If you make a mistake, tap “Undo”. If you want to clear the scorecard, tap “Reset”. The scores are stored in local-storage so there’s nothing server side that’s running (although it does mean they’re tied to your browser). Focus automatically moves to the player to the right, but any entry could be entered at any time.

There’s a two player and four player variant. I was considering having a “player entry” screen to select the number of players, but that felt like adding too much complexity for something that’s designed to be quick to implement.

In fact, much of this was already implemented in the form of a Finska scorecard.

This is not a new project, I just moved it from a dedicated domain name and changed the styling. It’s been a while since I needed it for a game of Finska: I was more often than not using it for generic scorekeeping. But there were some aspects of scoring Finska which made it useful for that, and less useful for anything generic. For those who never played Finska, the goal is to get to exactly 50 points. If you go over 50, you go back to 25 points. There’s also a mode where all the zero point entries are highlighted to indicate a foul (3 fouls means you loose the game).

And, that’s it! Really, there’s nothing more to say about these apps. I had a need and some free time, so I put these together. There is one other scoring app that I’ve made, but that’s a little more complicated and warrants it’s own post.

Sat, 31 Jan 2026 09:38:13 +1000

More work on assets for my niece’s game. This time I tried something organic: a tree. Not the most difficult organic thing, I agree. Need to build up to the hard things.

Mon, 26 Jan 2026 08:36:01 +1000

Was not expecting to spend yesterday morning working on Dequoter, my Boop clone. Opened it up to do some light work and when I looked up, a couple of hours have passed. Added a few more processors to deal with lines, such as splitting and joining on commas. Added a status bar for processes that return information rather than filter text, such as returning a line count.

Also integrated UCL, because of course I did. Added two commands: a UCL: Evaluate which executes the input as a UCL script and displays the response in the status bar, and a UCL: Replace which replaces the UCL script with it’s output. This makes it possible to generate text from scripts, like a bunch of lines to test out the line count processor:

map (seq 20) { |n| "Line $n" } | strs:join "\n"

It’ll also make for a useful scratchpad for testing out some UCL commands.

Devlog: Game For Niece — More Art and Cropping Ebitengine Images

Tue, 20 Jan 2026 22:22:00 +1000

Drew some more artwork and started integrating it into the game. Replaced the previous bus image I got from an image search with one I created myself using Affinity Designer:

It’s orange, just like the busses round here, although this one has a plainer livery.

I’ve not integrated the car just yet: I’m hoping to finish off the animations for the bus first. But I’m so glad I embraced Affinity Designer for this. I’m surprise I haven’t considered it sooner. The whole image is designed as a whole, then I export each component — body, door, wheels — as a separate PNG and recompose it in the code, giving me the means of animating each component. I’d imagine this is a pretty typical workflow: and I’m surprise I haven’t considered this sooner too.

And yes, I will admit I did look at AI image generation for this. But I wasn’t happy with what it produced, and I wanted each image to have a consistent style. Turns out an “elementary vector artist” style works perfectly here.

Anyway, took a bit of time working out how to draw a cropped image in Ebitengine. Spent a little bit of time seeing what was possible with masks. But after looking at the examples, it looks like using SubImage for this is viable:

var busDoor *ebiten.Image = getSpriteImage("bus_door.png")

func render(screen *ebiten.Image) {
  doorDX := bp.sprBusDoor.Bounds().Dx()
  doorDY := bp.sprBusDoor.Bounds().Dy()

  pos := ebiten.GeoM{}
  pos.Translate(float64(doorDX/2), 0)

  screen.DrawImage(
    busDoor.SubImage(image.Rect(0, 0, doorDX/2, doorDY)).(*ebiten.Image), 
    &ebiten.DrawImageOptions{GeoM: pos})
  )
}

As for the other scenes I need to add, I plan to work through vehicles and objects first as they seem easier to draw. Hopefully by doing so I gain some practice before I tackle more organic objects like trees and animals. Tackling those feel a little daunting to me: more curves involved.

Mon, 05 Jan 2026 21:36:02 +1000

Was hoping to prepare an account and make a video on using Alto for that interested party, but ended up fixing all the dodgy UI elements that were either dangerous or just plain embarrasing. The evening’s over now, and it’s still pretty dodgy, but it’s probably good enough to share with this person.

Sun, 04 Jan 2026 07:09:05 +1000

Dusted off Alto to give access to someone I know. Need to finish off the multi-tenant functionality, along with removing the hardcoded tokens from the mobile app. Manage to start work after 5 or so minutes of waiting for dependencies/builds/whatever, which is a nice change.

Devlog: An API For a Keyframe Animation Package

Tue, 30 Dec 2025 16:42:35 +1000

Spent the day building a key-frame animator for an Ebitengine project. All coded in Go, which should make it usable for some other projects in theory. I don’t have a lot of experience with animation frameworks — other than as an end-user of apps that consist of a timeline where I place keyframes — so I’m not aware of what constitutes best practice for a programatic API. So I set out to build one with an API that made sense to me. After one major iteration, this is the approach I came up with.

The core model type is the Var, which represents an animatable value. These are essentially handles describing the type of value the (package) user wants to animate. At the moment there is only a single type, which is a float:

xPos := animator.NewFloatVar()
yPos := animator.NewFloatVar()

To animate the value, the user will need to create an Animation and place any created Vars on a track. Tracks consist of key-frames, which is essentially a time offset, and one or more values a Var should have at that time. These values can be created from the Var itself, by calling Set(). This returns a type holding what that Var should be set to at that key-frame (“set” is probably not the best term for this):

moveDown := animator.NewAnimation()
posTrack := moveDown.NewTrack()

posTrack.KeyFrame(0.0, []animator.FloatSet{
  xPos.Set(0.0), yPos.Set(0.0),
})
posTrack.KeyFrame(3.0, []animator.FloatSet{
  xPos.Set(100.0), yPos.Set(100.0),
})

Not all Vars need to be included on a particular keyframe, but prior to animating, each of the Vars known to the track will need to be tweened if there isn’t a “set” for them on the key-frame, so it’s generally good practice to set all the Vars for each track. Different tracks can have a different of Vars, so if different keyframes are needed, it probably should exist on a separate track.

Animations are designed to be reusable and simply encode what the Animation is. The motivation here is that all the expensive work should be done once, when the Animation is created. To actually animate the Vars, the user will need to create a timeline and set it to the Animation.

tl := animator.NewTimeline()
tl.SetAnimation(moveDown)

This will initialise all the Var values to what they would be at frame 0 (at the moment, frame 0 must be a time 0, but down the line I’d like to remove this to allow for blending between Animations). The value of each Var can now be retrieved, as Vars only have a value within the scope of a Timeline.

x := xPos.Value(tl)
y := yPos.Value(tl)

// x and y == 0, as that is the value they were set to in the first frame
drawRect(x, y, 100, 100)

The Timeline clock is advanced by calling the Advance() method, which takes a delta in fractions of a second. This will tween all the Vars based on the current action, and the new value can be retrieved by calling Value() again. Thus the core use of this package is advancing the clock and querying the values for each tweened frame.

Within an Ebitengine project, this is done by calling Advance within Update function and sampling the values within Draw:

var (
  tl *animator.Timeline
  xPos *animator.FloatVar
  yPos *animator.FloatVar
)

func Update() {
  dt := 1.0 / float64(ebiten.TPS())  
  tl.Advance(dt)
}

func Draw(screen *ebiten.Image) {  
    vector.FillRect(screen,  
       float32(xPos.Value(tl)), float32(yPos.Value(tl)),  
       100, 100,  
       col, true,  
    )  
}

Here’s an test animation:

I’m quite happy with how this turns out. I like how little “magic” is involved in the API: values and clocks are not changed from underneath you: you need to explicitly advance the clock yourself, and read the values when you need them.

Obviously there’s room for improvement. Tracks and keyframes cannot be changed; only linear interpolation of float values are supported; and there’s no looping, bouncing, or reversing of animations. I’d hope to add all this in due time, probably based on the needs of the project I’m working on. There are also a few things I’m a little wary of, such as a global atomic int responsible for allocating an ID for each newly created Var. And the Timeline holds Var values in a map which I hope is fast enough for rendering on a screen.

But it’s nice to have something like this in my toolbox. It’s been something I’ve been wishing for a while.

Sat, 27 Dec 2025 09:15:38 +1000

Just placed the exit on Level 3-2 on my Godot game. The level still needs prettying up but the critical path is now done. I’ll think this will be the last level I build. There’s still so much work to do: polishing, backdrops, menus, etc. But no more content. Kind of want to put a pin in this one.

Devlog: Godot Project - Level 3-2 And Bobbing Water

Mon, 08 Dec 2025 21:32:32 +1000

I spent a lot of time on distractions recently, so I was a little surprised to find myself wanting to get back to working on that Godot game. The level under design is going okay: it took a few attempts to find the right way to start. Turns out writing down how you want the level to progress helps, like having three “acts” where the first act consists of some jumping puzzles, then introducing one of the gimmick, than a variant of said gimmick, and so on.

Anyway, I’m working on the second “act” which will feature a low-power mechanic: there’s only so many units of power to energise the various zones, and the player needs to juggle it all to get through the act. Because power is involved, I wanted to have a generator sprite to provide some decoration. This is what I came up with:

It’s… fine. I did try to get some inspiration by uploaded the tile-set to Google Gemini and asked it to produce something that resembled a generator:

Using the tile set images uploaded, please provide some suggested designs for a tile-set representing an electric generator. The suggestion must match the style and colour scheme of the supplied image.

This helped, although what I have here is a fair bit different to what was generated. I may adjust the colours a bit (the simple colours are by design as I want to maintain the palette used by the existing tile-set) and I may need to add some wire decorations and maybe a sign that says “Generator” to make it clear that’s what it is, and that it’s not a cannon, as the alt-text generation suggests. We’ll see how it goes, I guess.

One other thing I found myself wanting to add is water that can be raised and lowered when activated. I came up with something from first principals that looks a little like this (the tile layer is hidden to show the Area2D shapes):

It consists of is a top-level Node2D which will raise and lower when triggered. This is done in code, just to allow me to configure the speed and displacement on a case-by-base basis. A child of that is a TileMapLayer containing the water tiles. It’s at Z-index 6, which is just in front of the player so that it looks like it engulfs them if they fall in. Four units below the top is the kill-plane — water is a hazard — and four units below that is the “bobbing” plane. This is a Node2D that simply oscillates up and down by 2 units every second. A child of that is an Area2D solid with allows the player to move through, but will collide with a new crate sprite, which also collides with everything else.

This has the effect of a water plane in which creates would appear buoyant yet slightly submerged, giving the player somewhere to stand. Activating the water layer will allow it to “drain,” bringing the crates down to rest on any solids. Activate it again and the water level will rise, catching any crates which will begin floating:

There’s probably better ways to do this, and there are some drawbacks. It can’t be bundled into a dedicated scene, so each instance will need to be built manually. And if more than one crate is in the scene, they will bob in unison (although that could probably be alleviated with splitting the bobbing plane into areas with distinct offsets). But for something devised from first principals, I’m quite happy with it.

Devlog: Blogging Tools - Image Collections Triage App

Sat, 29 Nov 2025 09:54:02 +1000

Up until now, I’ve been using Blogging Tools’ in-app notification system to sort images to collections. There were quite a few limitations in doing so: only notifications for the first for images were raised, galleries weren’t handled, and you could only select one category per image. I also wanted a way to automatically change the header image in this blog. This was done using the image category notification, but it was pretty basic in that the image was chosen as the header if any category was selected.

I had ideas of improving all of this but I realised quickly that I needed something more sophisticated to do so. So I added another app to Blogging Tools to do this. I also thought to try out using Antigravity to do this. I was curious to know how well Google Gemini would work with a project that already had an established architecture and coding patterns. I was also curious to know how Antigravity felt when I held a tighter grip of the reigns: I wanted to drive the development work and only turning to the agent when I need something done.

I started simply enough: I built a basic HTML template for this new app and asked Gemini to prepare a stub handler:

Please create a new handler that services the template “imageposts/show.html”. Implement it much like the other handlers found in the “handlers” directory.

I forgot to mention to the agent that this was to be a stub, and it ended up using the existing provider for the gallery app. But it was pretty close to what I wanted. I then asked it to suggest a good HTML entity for opening a page in an external window.

What’s a good HTML entity for indicating that a link will open up into an external window/tab?

It suggested ↗, also a good answer. I then went on to build the Go models and wanted to work on the CSS styling next, so I asked Gemini to produce some example data using these models:

Please generate 3 stubbed values for this item slice. Use real example image URLs for the source.

It struggled a little on this one, suggesting a URL that didn’t exist. But after pointing this out, it got there in the end. It was at this point that I needed to do some refactoring to some of the existing services, needing to extract out a type that used only the values from another type. Gemini did quite well here too:

Note the select call. At the moment this is returning a micropub.Item model. I want to make a new model called “ImageCollection” that uses only the values of this micropub.Item that the code is currently using now. I don’t want the micropub.Item changed, but any code that is using this for the purpose of the selected call should include a map to this new model.

I guess this is not a novel experience to anyone using Cursor, but what I really liked about this interaction is that I was free to work on other things or move away from the keyboard entirely while the agent was working. This is one annoyance that I have with Claude Code: the exchange pauses and waits for me to approve every change before it proceeds. Having Gemini plan the changes first, then have me approve them once I’m back is a much nicer way of working.

I then turned to the database schema, defining the table schema and SQL queries myself, and asking Gemini to produce the Go code that marshals the data: the so called “database providers”.

Implement the Go DB provider for DB tables for the queries in “imageposts.sql”. Do it in a similar style to the other providers found in “providers/db”.

This is where I get the most use out of using coding agents. This code is not hard, but it’s super annoying to write. I think it also helps that there’s an established code base for the agent to work on. I was a little unhappy with what the agent produced last week, probably because it was required to produce it from scratch and in a way that wasn’t how I would write it. But here, probably because the agent had existing patterns to work on, it’s producing something that looks a lot better, to my eyes at least.

It did struggle with the providers a little, choosing to merge an update to two models in a single method. I did try to correct it, but I just had to go in and fix it manually.

We can probably separate SaveImagePost into one that saves the ImagePost, and another that saves ImagePostItems.

One other thing I discovered is that getting Gemini to write unit tests is important, as it provides a way for the agent to verify it’s code. I approved a change Gemini produced without looking too closely, and if I were to test it myself, it would’ve failed. But only after I asked it to produce a unit test did I see it fix the errors introduced in the last interaction. Something to think about in the future.

Might be a good idea to implement a unit test to verify that the DB methods in imageposts.go work. Do this by creating a new db.Provider instance with a temporary file, and verifying that the provider methods of imageposts.go work as expected. Please use testify.Assert to do the asserts.

It was now time to build the service layer. This is something I wanted to drive myself, occasionally relying on auto-suggestions. Antigravity fell down (ha ha) here a little, producing suggestions that were pretty far from what I wanted. I guess it was lacking the context to know what I actually wanted, which is somewhat understandable as I wasn’t giving that context. It also seems to fight with IntelliSence from VS Code sometimes. I occasionally press tab in response to a type suggestion from the index, only to have it produce a completion for some unknown type. I do wish the suggestions use the code index more in general. There’ve been more than a few times where the agent suggested types and methods that just didn’t exist. Might be something that can be tuned.

One quality of life improvement I did make was increase the timeout for when suggestions would appear. The default was something like 50 milliseconds, which is pretty short. Bumping it to 1 second helps, but even so they still get in the way. I may bump it a little higher.

First cut of the collection triage screen at the end of day one.

I worked on this feature over the next couple of evenings, occasionally changing tack which required a change in the database schema. One thing I did like while working on this was getting to the point where I could simply ask the agent to update the provider code whenever I did so, and it knew what I meant:

The ImagePost model and table schema now has a PostDate field/column. Please update the providers.

I do get the feeling that the user interaction in Antigravity needs to be refined a little. For someone who is willing to do most of the driving, it would be nice if the editor was a little less “needy”. This is probably more to do with VS Code being the underlying editor: lots of annoying popups and mouse overs, all shouting for my attention: LOOK AT THIS, LOOK AT THIS. But the AI suggestions don’t help. Instead of displaying the generated code, disrupting my sense of where everything is, maybe a more subtler approach is warranted: a small indicator beside the cursor or in the margin indicating that a suggestion is present. I may miss it, but who cares? Remember, I’m the one driving here.

Today, I got the feature finished. The way it works is that it will poll the RSS feed of this blog, and for every post that has an image, it will add an entry for me to triage. Each entry consists of a brief summary, plus the images found within the post:

A new 'Sort Image Posts' app, with a badge indicating the number of images to triage.

List of posts with images that need triaging.

Clicking through to the post would list the collections, and I would select the ones I’d like the image to be filed in. And this would list all the images of the post, not just the first few. It is all or nothing, unfortunately. There are ways to get the images of the collection, but I couldn’t see a way for me to get the collections of an image. I don’t think this would be a problem, as Blogging Tools is the only way I actually set the collections of an image.

Selecting a post will list the images found within that post, with the collections they should be added to.

Saving the categories will queue the changes, which will be applied every 5 minutes. A first cut of this feature applied the changes immediately, but the way I imagine using this is that I’d go in, triage the images in one go, then leave. Applying all the changes then felt inefficient.

When it comes to setting the header image, I figured I’d add a dedicated collection for that. Every hour a cronjob will query the local database for images that is within this category, and write the URLs to a manifest file in the assets Git repository. A Forgejo Action on that repository runs every night, and will check the date-stamps for the next image to display on the header. If one is found, it will fetch the image, save it as a JPEG with reduced quality and upload to Netlify. Since Netlify flushes the CDN whenever a deployment happens, nothing more needs to be done: no MD5 sum hashes, no adjustment to CSS files. Just update header.jpg and go:

// Example of the manifest file.
{
  "images": [
    {
      "date": "2025-11-22T13:00:00Z",
      "url": "https://lmika.org/uploads/2025/pxl-20251120-083552448.jpg",
      "crop": "bottom"
    },
    {
      "date": "2025-11-26T13:00:00Z",
      "url": "https://lmika.org/uploads/2025/a3c74bfebd.jpg"
    }
  ]
}

Deployed all these changes to prod and naturally some rework that’s needed. First, I completely forgot about galleries. They do come through as image tags in the generated HTML, but I’m operating on the source post, and they appear as Hugo short-codes. Fortunately the way I write galleries is pretty standardised now, so I think a regular expression would work here.

var (
  galleryRegexp = regexp.MustCompile(`[{]{2}[<][ ]?glightbox.*src="([^"]+)".*[>][}]{2}`)
)

Oh, I also want to fix the layout of the collection checkboxes and hide some options, such as those that for past years. And finally, will turn off the existing image category notification logic.

Deployed again, and now I just need to wait for the next time I post a photo. I suspect some bug fixes or usability adjustments will come from this, but more on that when that happens.

Devlog: Blogging Tools - Chunking File Uploader

Sun, 16 Nov 2025 19:40:43 +1000

I’m hoping to use Micro.blog more for video hosting, meaning that I would like a fast way to upload them. Because my internet connection is not fast, I’m looking at adding some fancy uploading techniques to Blogging Tools. Blogging Tools does have a simple file repository, and I’ve already built a reusable upload component, which uploads the contents of a file with a single HTTP request. My goal is to improve the performance of this by using a few of the techniques found on this dev.to page, such as chunking and concurrency.

But first, I start with some baseline tests. I will simulate a slow upload by throttling the browser speeds at “Fast 4G”, and test the time it takes to upload a 45.4 MB file.

Test done. It took 269.174 seconds, about 4.49 minutes.

So my first goal is add some chunking and concurrency uploads, similar to what was suggested in the linked article. The way I’m thinking of doing this is as follows:

Modify the “new upload” request handler to accept a maximum file size, a chunk size, and the expected number of chunks. The server will respond by creating a empty file of that size, ready to be written to.
I’ll then make multiple fetches requests, each uploading a single chunk, identified by the chunk’s starting offset.
Then, once all chunks are uploaded, I’ll send a finalise request with a file hash. The server will also calculate the hash and say whether or not the upload was successful.

I’ll ignore the progress indicator for now, but that will need to be updated to reflect the actual upload progress.

First pass is to simply make sure the chunking works and doesn’t corrupt the uploaded file. I’ll begin on the server side by creating an empty file of size N. It was difficult to find how to do this in Go, but it turns out the os.Truncate method can be used to set the file size, once the file has been touched:

// This creates the file and sets it's length  
fileName := cfs.fsProvider.FilePath(file.SymID)  
if f, err := os.Create(fileName); err != nil {  
    return err  
} else {  
    f.Close()  
}  
  
if err := os.Truncate(fileName, file.Size); err != nil {  
    return fault.Wrap(err)  
}  
  
f, err := cfs.fsProvider.OpenFileFlags(ctx, file.SymID, os.O_RDWR)  
if err != nil {  
    return fault.Wrap(err)  
}  
  
cfs.fileHandles[file.ID] = uploadSessions{  
    mutex:      sync.Mutex{},  
    fileHandle: f,  
    size:       file.Size,  
    startTime:  time.Now(),  
}  
return nil

The rest of the Go code is pretty straightforward: requests to upload a file chunk will simply be passed down to the file handle and written at the requested position using WriteAt. The finalise call will seek to the start of the file, compute the SHA-1, and compare it with the SHA-1 computed by the browser. MD5 would probably have been my preferred hash because of it’s speed, but SHA-1 seems to be the one supported natively in browsers. The open files are kept in an in-memory “upload session” data structure. Finalising will also close the file handle and remove the session, but I do need to add a “session reaper” to deal with any sessions that died or terminated early.

Here’s the first cut of the JavaScript which does the chunking upload:

let fileMetadata = await (await fetch(`${window.bgtData.resPrefix}/files/new`, { 
    method: "POST",  
    headers: {  
        "Content-type": "application/json"  
    },  
    body: JSON.stringify({  
        "name": file.name,  
        "mime_type": file.mimeType,  
        "size": file.size,  
    })  
})).json();  
  
let chunkSize = 1024 * 1024 * 10;  
let chunks = Math.ceil(file.size / chunkSize);  
  
for (let i = 0; i < chunks; i++) {  
    // TODO: retries  
    let didUpload = await this._uploadFile(file, i, chunkSize, fileMetadata);  
}  
  
// Finalise the upload  
await this._finalizeUpload(file, fileMetadata);

Doing a quick test to see how this performs, although I’m not expecting any improvements. And indeed, there weren’t. In fact, it’s slower, at 270.301 seconds (4.51 minutes).

But this is where the performance boost comes in, in theory: replacing the loop with a call to Promise.all:

let chunkSize = 1024 * 1024 * 10;  
let chunks = Math.ceil(file.size / chunkSize);  

// Prepare promises for each upload chunk, then dispatch them all at once  
let uploadPromises = new Array(chunks);  
for (let i = 0; i < chunks; i++) {  
    uploadPromises[i] = this._uploadFile(file, i, chunkSize, fileMetadata);  
}  
await Promise.all(uploadPromises);  
  
// Finalise the upload  
await this._finalizeUpload(file, fileMetadata);

Testing this approach. As expected, the progress bar is jumping around as each upload fights to update it. But we can deal with that later. We’re only interested in the upload time.

And the results are in. And sadly, they’re not much better: 269.597 seconds, about 4.49 minutes.

I am curious if this has something to do with the browser limiting the number of parallel uploads per domain. I hacked a version of the JavaScript which would send each chunk to a separate subdomain, so chunk 0 will go to u0.localhost:3000, chunk 1 will go to u1.localhost:3000 and so on. This does mean adding CORS middleware to Blogging Tools, as this would be considered cross-origin requests. But alas, this didn’t help matter: the upload time was still around 269 seconds.

So the bottleneck must be the connection itself. I was afraid of that, especially when you consider that the browser and OS would be using all the available network bandwidth to do the upload. I can’t think of any reason why it would be throttled. Another issue might simply be distance. Blogging Tools is currently hosed in Germany, since Hetzner doesn’t offer any hosting locations in Australia. But there’s also Singapore, which is quite close. Maybe it’s worth moving Blogging Tools there.

Anyway, it may still be useful to keep the chunking uploader around, if for no other reason just to avoid timeouts due to long running connections. So I undid all that multi-domain work and finishing the feature off with a stalled upload reaper which will close file handles that haven’t been written to in the last 5 minutes. I also found that I wasn’t doing anything to throttle the fetch requests: a hundred or so were dispatched at once, and I think the stalled upload reaper was having an effect on all the requests fighting for bandwidth: I found uploads failing because no single request managed to finish after 5 minutes, and the reaper was killing the upload.

So made a few small changes to only dispatch batches of 5 at a time, each with a 2 MB chunk, and bumped the reaper to wait for 30 minutes. Gave it a test to see how long it took to upload a 469.4 MB file. It was not quick: 7,112 seconds, or around 118.5 minutes, or 1.98 hours. It did work, and I managed to get the file to Blogging Tools. Forwarding it on to Micro.blog failed though: turns out the file was too big anyway. So yeah, may need to do something about that.

Now, this may have been wasted effort, but I think it’s worth keeping. There are some benefits from uploading files this way, such as not having connections die due to timeouts. You also have the opportunity to retry particular chunks that individually fail, without causing the entire upload to be wasted. And I do think they are being uploaded in parallel: seeing the 5 requests in the console end around the same time suggests that they’re not entirely sequential. But I think the limiting factor here is that my upload speeds are terrible. It’d be probably easier upgrading the connection first before embarking on any more fancy upload techniques.

Devlog: Laying The Groundwork For Dynamic Header Images

Sat, 08 Nov 2025 11:11:26 +1000

Making some changes to the Card theme I’m using for this blog. First think I’m considering is a banner image, similar to the one in Scripting News. And like Scripting News, I’m hoping for the image to change occasionally. I’d like the change to happen when the blog is being built, and in order to do this, I need a way to configure this value. I’m hoping to use Blogging Tools to do this, but to actually make use of these values, I’m hoping to use Hugo’s resource data methods.

The idea is to setup an assets project which will be hosted on Netlify. When I want to change anything in it, Blogging Tools will push new changes via Git, which will run a pipeline that will deploy the updates to Netlify. Then, next time I add a blog post, Hugo will pick up these data fields and use it to build the site, probably by setting an attribute that will itself be picked by CSS’s newly improved attr() function to set the banner background.

This will start off as simple data values, stored in JSON, which will include things like the current banner header. One of the reasons why I’m keeping the actual data separate is that I don’t want any issues with Blogging Tools from impacting my ability to publish blog posts, and I have more confidence in Netlify’s ability to keep a service up than I do, especially as a company that is getting paid to do so.

Being a “backend guy”, I’ll start on the Netlify and Blogging Tools side of things first. Actually, I’ll start on the asset repo first, just so I can verify that Micro.blog’s Hugo instance can pull data like this.

I’ve added the new Git repo and got it publishing to Netlify on commit to main. At the moment there’s a single JSON file, /site/data.json that has a test message. The goal is now to get that onto my test blog and included in the Hugo template on build. Naturally, because Hugo loves changing their public API so often, I need to see how this is done in Hugo 0.117 as it seems to be completely different in Hugo 0.141, the current version as of this writing.

So, checking out the Hugo source code and doing a find/grep search to find the relevant technique. Turn’s out the way to do this in Hugo 0.117.0 is to use the getJSON function. Here’s a link to the docs for anyone else who’s interested.

<div>
  {{ $siteData := getJSON "https://assets.lmika.org/data/site.json" }}
  <div>My data is {{$siteData.data}}</div>
</div>

Okay, that seams to work:

So, the next thing to do is to add a banner image. I’m thinking of a slightly washed out version of the image, with a subtle gradient to the body housing the cards. I had to adjust the header template a little to move the site-header to a wrapping div. This allowed me to add new CSS to allow the header to span the entire width of the page. It’s really fortunate that the Card theme styles the CSS classes, and not the HTML elements themselves. It also allowed me to add a new HTML element with a header-end class to act as the gradient blend from the backing image to the regular background colour:

<!-- Template: layouts/partials/header.html -->
<header>
  <div class="site-header">
    <!-- The original content of 'header' -->
  </div>
  <div class="header-end"></div>
</header>

With that I could restyle the header:

/** Header */
body {
  /* The body had a 10px padding which I had to disable to allow
     the header to blend all the way to the client sides. */
  padding: 0;
}

div.page-content,
footer.site-footer {
  /* Disabling the 10px padding on the body meant I had to move
     the padding here.*/
  padding: 10px;
}

header {
  /* Using 'background-blend-mode' to mix the header image with the background
     colour. We're using 'color-mix' to make the background color slighty
     transparent to do this. */
  background-color: color-mix(
    in srgb, 
    var(--body-background-color),
    rgba(255,255,255,0.75)
  );
  background-image: url(https://lmika.org/uploads/2025/c1745b14d3.jpg);
  background-blend-mode: screen;
  background-size: cover;
  background-position: 50% 50%;
  margin: 0;
}

div.header-end {
  /* A small gradient from transparent to a fully opaque background colour
     allows a smooth transition from the header to the body. */
  height: 20px;
  background: linear-gradient(180deg,
    rgba(0, 0, 0, 0) 0%, 
    var(--body-background-color) 100%
  )
}

Various links to MDN documentation covering all these changes:

background-blend-mode to blend the header image with the background colour. This is to wash it out some so that the menu items are not illegible.
color-mix to set the transparency on the background colour.
linear-gradient to generate a linear gradient.

Oh, and I naturally forgot dark mode, so added a new style to darken the image in that mode:

@media (prefers-color-scheme: dark) {
  header {
    background-color: color-mix(
      in srgb, 
      var(--body-background-color),
      rgba(0,0,0,0.5)
    );
    background-blend-mode: darken;
  }
}

Okay, the next thing is to make the image dynamic. I want to try and use CSS attr() to do this. Basically the idea is that when the template is built, the image URL is pulled from the data and sent to the CSS via a HTML attribute. Here’s the new header template

<!-- Template: layouts/partials/header.html -->
{{ $siteData := getJSON "https://assets.lmika.org/data/site.json" }}
<header data-background-url="{{ $siteData.header.imageUrl }}">

And the updated CSS:

header {
  background-image: url(attr(data-background-url raw-string));
}

Ah, that didn’t work. Apparently there are some restrictions on attr which could potentially result in security issues. So, plan B is to simply set the background-image in a style a style attribute:

<!-- Template: layouts/partials/header.html -->
{{ $siteData := getJSON "https://assets.lmika.org/data/site.json" }}
<header style="background-image: url({{ $siteData.header.imageUrl }})">

Okay, that’s working. Now the final test: can I make it dynamic? I’ll change the URL to another image:

{"header":{"imageUrl":"https://lmika.org/uploads/2025/pxl-20251105-200617867.jpg"}}

And publish a new post to rebuild the template. Let’s have a look:

Awesome, that’s working. Now to apply it to the actual blog. I did make one last change: the header attribute is used for the header of posts too, so I had to add a class — site-header-wrapper — to only select the site header. But yeah, it looks good. I like it. For reference, here’s the final template changes I’ve made:

<!-- Template: layouts/partials/header.html -->
{{ $siteData := getJSON "https://assets.lmika.org/data/site.json" }}
<header class="site-header-wrapper" style="background-image: url({{ $siteData.header.imageUrl }})">
  <div class="site-header">
    <!-- The original content of 'header' -->
  </div>
  <div class="header-end"></div>
</header>

And the final CSS changes:

/** Header */
body {
  padding: 0;
}

div.page-content,
footer.site-footer {
  padding: 10px;
}

header.site-header-wrapper {
  background-color: color-mix(
    in srgb, 
    var(--body-background-color),
    rgba(255,255,255,0.75)
  );
  background-blend-mode: screen;
  background-size: cover;
  background-position: 50% 50%;
  margin: 0;
}

@media (prefers-color-scheme: dark) {
  header.site-header-wrapper {
    background-color: color-mix(
      in srgb, 
      var(--body-background-color),
      rgba(0,0,0,0.5)
    );
    background-blend-mode: darken;
  }
}

div.header-end {
  /* A small gradient from transparent to a fully opaque background colour
     allows a smooth transition from the header to the body. */
  height: 20px;
  background: linear-gradient(180deg,
    rgba(0, 0, 0, 0) 0%, 
    var(--body-background-color) 100%
  )
}

I didn’t have time to integrate this with Blogging Tools, but the groundwork has now been laid for that now.

Devlog: Dynamo Browse - Item View Annotations and Asynchronous Tasks

Tue, 04 Nov 2025 20:42:40 +1000

The laundry list of things to do in Dynamo Browse has grown over the last week, as I find myself wanting using it and wanting more from it. I’ve knocked off many of the small ones: fixing bugs, making it easier to get the first item from a result set. The time has come to tackle some of the larger ones.

The first is the ability to annotate fields in the item view, as in add additional information to the right of the value. I would personally find that useful for describing the value of an other table using it’s ID. But the point is that it can be anything. I’d like users to add these annotations via extensions, using UCL. The way I’m thinking of doing this is by “installing” a field annotator, something along the lines as follows:

ui:add-item-annotator { |rs attr_path| 
    return ""
}

This will add an annotator which will take a result set, and a path to an attribute. It will return the annotated value. We’ll see how well this will work, but first I’ll need to build the Go types. Rendering the item view is done using an ItemRenderer which basically walks through the attributes of an item and renders it as a table. I think I’ll change this to take an item annotation. I also realised I need a type to represent the attribute path. I think a simple linked list would work here. Here’s what I got:

type AttrPathNode struct {  
    Name    string  
    Index   int  
    IsIndex bool  
    Parent  *AttrPathNode  
}

The trouble is representing both string attribute names and numerical indicies in the same type. Go doesn’t have a great way of doing this, so I’ve gone with something simple and added boolean which would be true if the node is actually an integer index to a list or set.

Actually, that may not be necessary. Poking around the renderer code, I found this type used to represent the sub items of an item:

type SubItem struct {  
    Key   string  
    Value Renderer  
}

So I can simply use strings for the key. Good to know. Okay, added a test annotator and gave it a quick test: Not bad, although found some glaring issues. It would be nice if the annotation was closer to the value. And laying out the annotations in a dedicated column will cause the entire column to shift as I move through the items. So instead of making it a separate column, I’ll try simply concatenating it to the end of the value. I’ll also apply the meta-info styling to dim the text a little: That’s much better. I’m banking on only a few attributes having annotations, so I’m hoping it won’t be as busy as it looks here.

Now to consider the UCL integration. Hmm, how am I going to connect the two? As expected, injecting the annotation directly into ItemRenderer would introduce a dependency loop. What I could do instead is add a setter allowing the caller of the ItemRenderer to change the annotation on the fly, and inject the ItemRenderer as a dependency of the command controller.

Implemented this, and it was actually easier than I thought. Now to test this. I settled on the following UCL command:

ui:set-item-annotator { |rs item path| 
    "annotation"
}

where:

rs is the result set
item is the item being rendered in the view
path is the attribute path, represented as a list.

The attribute path is a new proxy, just to reduce the amount of memory copying between Go and UCL. I kept the same list index semantics, where > 0 starts from the left, and < 0 starts from the right, but it was a little mind bending to reverse this for a linked list.

A simple annotator which returns the currently displayed item value as an annotation can be implemented as follows:

ui:set-item-annotator { |rs item path|
    $item.($path.(-1))
}

Here it is in action: Brilliant, it’s working! Naturally as commands are simple UCL statements, this could be entered on the command line.

One thing to consider is that unlike normal commands, the UCL block rendering the annotation is running on the UI thread. Not sure I like this, but the amount of effort required to change this would be significant. I guess I’ll just tell everyone to keep annotation rendering fast. And as it’s currently implement, it is reasonably fast. Or more accurately, it’s not noticeably slow, which is good enough.

The next feature to add to Dynamo Browse is a way to asynchronously schedule blocks. I think it may be worth tapping into the existing command looper in some way.

Commands in Dynamo Browse are invoked on a dedicated goroutine. Calling execute() will attempt to send a command via a channel. If that fails, execute() will return an error indicating that a command is currently running. This keeps running UCL code off the UI thread, and it also makes it possible to implement commands like ui:prompt synchronously from the UCL code’s perspective, when in reality it pauses this goroutine and sends a message to the UI with a callback to resume the thread:

func (m *uiModule) uiPrompt(
    ctx context.Context,
    args ucl.CallArgs,
) (any, error) {  
    var prompt string  
    if err := args.Bind(&prompt); err != nil {  
        return nil, err  
    }  
  
    resChan := make(chan string)  
    cancelChan := make(chan struct{})  
    go func() {  
        commandctrl.PostMsg(ctx, events.PromptForInputMsg{  
            Prompt: prompt,  
            OnDone: func(value string) tea.Msg {
                resChan <- value  
                return nil  
            },  
            OnCancel: func() tea.Msg {  
                cancelChan <- struct{}{}  
                return nil  
            },
        })  
    }()  
  
    select {  
    case value := <-resChan:  
        return value, nil  
    case <-cancelChan:  
        return nil, nil  
    case <-ctx.Done():  
        return nil, ctx.Err()  
    }  
}

So tapping into this event loop would be nice. But how does one do so?

I think, probably the simplest way to do so is with buffered channels. These are bounded, meaning that there will be an upper limit to the number of pending tasks, but maybe that’s not a bad thing. Having too many pending tasks crowding out the user’s ability to run commands will probably make for a poor user experience. So let’s set the limit to something quite generous, say 50, and integrate it into the event loop:

for {  
    select {  
    case cmdChan := <-c.cmdChan:  
        res, err := c.ExecuteAndWait(ctx, cmdChan.cmd)  
        if err != nil {  
            c.postMessage(events.Error(err))  
        } else if res != nil {  
            c.postMessage(events.StatusMsg(fmt.Sprint(res)))  
        }  
        if execCtx.requestRefresh {  
            c.postMessage(events.ResultSetUpdated{})  
        }  
        // New code here
    case task := <-c.pendingTaskChan:  
        if err := task.task(ctx); err != nil {  
            c.postMessage(events.Error(err))  
        }  
    }  
}

Now for the UCL API. I have an idea of adding a new async package for this, which will provide commands for running things asynchronously. It’s simplest command would be async:do, which will schedule a block when the command loop is free. So invoking the following:

ui:command testasync {  
    echo "This"
    async:do {
        echo "Other"
        async:do { echo "Sierra" }
    }
    async:do {
        echo "Romeo"
    }
    echo "That"
}

Should display:

This
That
Other
Romeo
Sierra

In the logs. Trying it out and here’s how it looks: Okay, that’s pretty good. Now to build on this. The next thing to add is a command that will schedule tasks in the future, similar to window.setTimeout(). For this, I plan to use the gocron package. Lots there, but for my purpose, I plan to use the OneTimeJob type. It does feel like bringing in a crane to lift an empty wood pallet, but the alternative is building my own scheduler (or getting AI to vibe-code one) using a heap. Maybe something for later, but I think this is fine for now.

To make use of this, I’ll add async:in, which takes a timeout in seconds, and a block to run. To test this, I’ll do the classic “count down” tests:

proc countdown { |from|
    if (le $from 0) {
        echo "Blast off!!"
        return
    }
    echo "$from"
    async:in 1 { countdown (sub $from 1) }
}

ui:testasync {
    countdown 10
}

Here it is in action:

One last thing to add to async, the ability to run a query in the background and execute a block once the results are available, keeping long running queries off the UCL goroutine. I think for this I will add another thread-pool which will execute up to two queries in the background, then schedule a task to run once the results are ready.

The API for running queries is pretty well established, to the point that I’ve actually got a helper functions which deal with the UCL side of things, so it’s just a matter of implementing an asynchronous version of this. I’m thinking of an interface along the following lines:

async:query <query> { |rs|
    # do something with the result-set $rs
}

It should also support keyword arguments too. Unfortunately, UCL requires keyword arguments to be placed after the positional arguments, so they would need to be placed after the block, which is a little yucky.

Implemented the code, now for the test. This one’s a little contrived. What it does is asynchronously run a query on startup, counting the number of opened and closed offices in the business-addresses table. Once those results are available, it will install a new column annotation, which will display the count to the right of the officeOpened fields.

proc _prep_officeCount {
    openedOffices = 0
    closedOffices = 0

    async:query 'officeOpened=true' { |rs|
        openedOffices = len $rs
        async:query 'officeOpened=false' { |rs|
            closedOffices = len $rs

            ui:set-item-annotator { |rs item path|
                if (eq $path.(-1) "officeOpened") {
                    if $item.officeOpened {
                        "Count = ${openedOffices}"
                    } else {
                        "Count = ${closedOffices}"
                    }
                } else {
                    ""
                }
            }
        } -table business-addresses
    } -table business-addresses
}

_prep_officeCount

It is contrived, but I’m hoping to use this in a real setting where I annotate ID fields with a value I retrieve from another table, so a use case like this is pretty close to why I built this feature. Because the queries are running in the background, the goal is to avoid blocking the UI thread, but since this runs on startup before the user selects a table, the table needs to be specified as part of the call. I may have to have a think of how I am to fix that.

But here it is working. Note the Count = 244 next to the officeOpened field:

Excellent. The next thing to do is try this out in a real setting. I finished off making a proper goroutine pool for the task that actually runs the query — which I’m calling the “aux task pool” — and a few other changes, like making the meta information a little easier to see.

Devlog: Godot Game - Level 3-2 and a Rotating Platform

Tue, 28 Oct 2025 21:19:28 +1000

Okay, time to start level 3-2. This, like level 3-1, is in the mountainous regions. Except this time, the player will be a little higher. So that means jagged platforms, more verticality, lots of gaps: a real sense that care of where one steps must be taken.

Starting with some platforming to give this sense: a few narrow safe areas, separated by pits, leading into an area with platforms.

Now, I have two choices of mechanics here: I can have the rotating platform, or I can have an area of the tile map fall away when the player lands on it. I think I prefer the rotating platform for the sole reason that adding the falling away platform may be a little overwhelming for the player.

Actually, no. Danger and instability is the feeling I want to go for here. So I’ll add the falling platform, followed by a large safe area where the player can see the rotating player. Also, since this is a new mechanic, it’s only right that the player should play with it in a safe area first.

Okay, I’ve added the falling platform. Well to be honest, I designated an area of the field where the falling platform will go. This leads into a large safe area which would be an ideal place to introduce the rotating platforms I have in mind.

I’m thinking of a scene with an AnimationPlayer that will do the actual rotation animation: as in rotate the sprites from 0° to 90°. The question is, do I want the entire sequence timed using the AnimationPlayer? As in, have a single animation that will rotate the platform all 4 times, with gaps between each stage? Or should I stick with the AnimationPlayer simply animating a 90° rotation, and having another thing timing the wait time between stages?

I think I prefer the latter. That way, I have more control over the pause between stage rotations. I can do things like disable it, or making it configurable. All desirable features.

I guess the next question then is whether I want the AnimationPlayer at all. Wouldn’t it be more flexible to do the animation in code? Probably, but I’m not sure I want that flexibility, at least not yet. Maybe in time it would be needed, but I think for now, I’ll just stick with the AnimationPlayer.

So, let’s start building out the animation. I think I’ll start with four animations, each one rotate the platform from θ to θ+90°, just from different starting positions. I’ll use names that include the start and end angles, like rotate_0_to_90, to allow for bidirectional rotation, should I decided to add that. Angle 0° will be up, and rotation will be clockwise. I think each rotation will be 0.5 seconds at speed 1, just so I have a consistent speed control. I can adjust the speed of the overall playback on the AnimationPlayer itself.

Okay, animations made. Now to think of collision. I don’t like the idea of a collision box rotating along with the platforms. I’m not too certain about how well this plays with the physics engine, and even if there were no issues, non-orthogonal geometry is just not a thing in this game. So what I’m thinking of doing is adding two collision boxes, one horizontal and one vertical, and just toggling between the two as the platform moves into position. While the platform is rotating, both boxes will be turned off.

Ooh, I didn’t know the collision layer could be added as a animatable track. That’s pretty cool.

Now to add a Timer which will actually dictate the cadence. The included code is nice and neat: basically take the current rotation of the sprites in degrees, add 90 to it, and produce a string which will become the next animation to play:

func _on_timer_timeout() -> void:
	var next = _next_animation()
	animation_player.play(next)

func _next_animation() -> String:
	var rot = int(sprites.rotation * 180 / PI) % 360
	return "rotate_%s_to_%s" % [rot, rot + 90]

Added it to a test level, and I had to speed up the AnimationPlayer to 3x normal speed, just so that the movement is snappy enough to justify the loss of collision during the rotation. It may need further adjusting a little later, but so far works pretty well. Although I do think I need to telegraph when the platform is about to move.

Oh, I remember how I’m going to telegraph to the player when the platform is about to rotate! It’s to be an indicator panel mounted at the front that will rotate slowly on it’s own. When it reaches a new orientation it will pause, then the platform behind it will rotate.

Okay, new sprite. How the heck did I choose a canvas size with non-even dimensions and still end up with a sprite that doesn’t have a definitive centre line? 🤦

Turns out it was how I was removing the upper and lower circles. I was using the eraser for that, and I didn’t position it properly.

Okay, here’s the sprite:

Now to think about animating it. I’m wondering if it would be possible to simply add it to the same animation player as the platform. The thing is, I would like them to be somewhat independent. The indicator should move on it’s own accord, even when the platform sprite is rotating to a new orientation. I would also like the indicator to take on the platform rotation cadence, sending it a signal when it reaches a stop (this will remove the need for the Timer).

Can I do that all in a single scene though? Maybe with two animation players? I’ll try that and see how viable it is.

Okay, just tried that, and yes, using two AnimationPlayers is viable. Now to prepare the animation. The current rotation period is 3 seconds so I’ll start with a 6 second animation that would just rotate the indicator over 180° with a slight pause halfway to trigger the platform. One good thing about the indicator’s symmetry is that one animation can cover the full rotation. Although be sure to change the loop wrap mode to “Clamp”, otherwise you’ll see the sprite spin round to 0° when the animation repeats.

Here’s the full thing in action. And yeah, adding the indicator was pretty crucial in the end. The platform wouldn’t have been fair otherwise.

Devlog: UCL - Adding Some Missing Library Functions

Mon, 27 Oct 2025 21:26:14 +1000

Working on UCL, adding some missing builtins I’ve been finding myself wanting. Nothing too interesting. Just functions like strs:has-prefix, strs:trim-suffix, and other functions involving strings and lists that are missing. That sort of thing.

I am facing some decisions around strs:substr. See, the usual way these builtins are validating arguments is that if there are more arguments than are necessary, they are generally just ignored. One could describe this as the “JavaScript” approach to argument validation: open Node and evaluate "hello".substr(1,2,3,4,5,6,7,8,9) and one will get the same result as if they simply typed in "hello".substr(1,2). But now I’m wondering if it’s better to assert that the arguments are either one or two positions, along with the string. What if I wanted to add more positional arguments in the future?

Although in practice, why would I want to do that? Well, okay, I do have an idea of for taking multiple substrings in a single call. So could be useful? Ah, probably doesn’t matter: it’ll just be me using this for now.

Anyway, strs:substr. One deviation from JavaScript I think I would like to do is that if a single number is used, that would be treated as the end position of the substring; as oppose to JavaScript which treats it as the start position. So running strings:sub "hello, world" 5 will return "hello", rather than ", world". To start from the right, a negative position can be used: strings:sub "hello, world" -5 will return "world". Two numbers will be treated as a start (inclusive) and end (exclusive) positions, like most other languages; although both can be negative which will set the position from the right side.

strs:substr "hello, world" 5   
--> "hello"

strs:substr "hello, world" -5  
--> "world"

strs:substr "hello, world" 3 10
--> "lo, wor"

Okay, that’s working for strings. Need to do the same thing for lists. I could almost use the same function.

Oh, I keep forgetting that I need to worry about nils. How should sublists work if the passed in list is nil? Maybe it’s better to just return a nil. Less hassle than requiring the user to deal with nil values. I’ll start with that approach, but I may change that the future. Should I do the same for strs:substr? Hmm, no. I think I’ll keep the default binding rules for nil to strings, which is basically convert it into an empty string. That saves me from changing the binding logic. Maybe I should lock that decision in with a test.

Hmm, also, many of the other list builtins support iterators. If I were going to add iterator support to sublist , then how would I handle the negative numbers? Iterators, by definition, can be boundless. Maybe leave iterator support out for now.

lists:sublist [a b c] 1
--> [a]

lists:sublist [a b c] -1
--> [c]

It also looks like the argument binder has made my mind up about nils for me. I added the test which testing for nils, and I got an expected listable error.

lists:sublist () 1
--> error: expected listable

I think that’s fine. Might be better to fail on nils actually.

Getting back to the argument question raised earlier, I did run into an issue where I was using strs:split without a second argument. When one does that, UCL will do something similar to Go, and return a list of each individual rune. I think that’s a useful feature, but I think I’d prefer it if that option was explicit, and if a second argument were not set, raise an error.

Okay, a few more items on my todo list. I need a string replace function. I generally prefer such a function to replace all the instances of a string, rather than just the first; but that might be pushing my preference a little too far. So I’ll settle with replacing the first instance of a substring and add an -all flag to replace all instances.

Actually, no. Go has support for specifying a count, so I’ll use that by adding an -n COUNT option. If N is unset, then all the substrings are replaced.

strs:replace "hello hello hello" "hello" "world"
--> world world world

strs:replace "hello hello hello" "hello" "world" -n 1
--> world hello hello

Also, given that this is using Go’s builtin strings.Replace function, setting “match” to the empty string will cause replace to match between each rune, plus the start and end of the string.

strs:replace "each one" "" "|"
--> |e|a|c|h| |o|n|e|

Could be useful, so I’ll keep this feature in.

Nearing the end now: adding an -in switch to os:exec to set stdin for an command. The original idea was to name the switch -stdin, and I also considered -input. But when writing the tests, I naturally settled on -in.

os:exec tr '[a-z]' '[A-Z]' -in "hello"
--> HELLO

It would’ve been nice to accept stdin from a pipe. Sadly, that’s not possible with how pipes actually work.

And the last one: adding a nil? builtin, which will return true if the given item is nil. I guess in the grand scheme of things, it’s probably not strictly necessary, as anything that’s not zero, including nil, will be true. And one could always use eq $thing () to test if $thing is nil. So I may leave this one on the backlog.

Okay, finished. I wanted to add some stuff to Dynamo Browse, but I ran out of time. I will update the version of UCL used by Dynamo Browse, but all that other stuff will need to wait for tomorrow.

Devlog: Godot Game - Working On Backdrops

Mon, 20 Oct 2025 21:01:17 +1000

Time to focus on backdrops. Aaaaargh! What it’s time for is to face my inability to make artwork. I’ve been able to do small things: lifts, signs, even some pickups. But for the larger stuff, I don’t know where to begin. And regretfully it fills me with shame that I’m even looking at asset packs: despite most of the assets in this game already originating from an asset pack. But the alternative is drawing it myself. And I’m not confident that I’ll be able to do a good job here.

But let’s try. If it doesn’t work, I’ll use the asset pack.

The first idea is to get some inspiration. Well there are plenty of those in my photo collection. Here’s one that seems like a good start:

In short, foothills are slightly jagged yet mainly consist of sweeping curves at roughly the same vertical height. Colour and shadow provide the sense of distance. The hills in the photo are largely covered in vegetation, which could potentially be simulated by a vertical spray-brush: favouring light and dark green with some light brown on a largely mid-level green colour.

Also I need to remember a principal of illustrations: go from the foreground to the background. I guess one saving grace with modern art tools is that I’ve got layers to play around.

Okay, new sprite. Let’s start with the foothills. Taking inspiration from both the photo and the asset pack. Tracing around the foothills and adding some colour to give us depth yields the following:

Okay, not bad. A little plain but a good first start.

Trying out some ridge-lines.

Ugh, that looks awful! Not convincing at all. Okay, forget about the ridge-lines. I also tried out adding a spray to simulate vegetation but that didn’t look convincing either. I think given the distance, my skill level, and the fact that this is a backdrop, less might be more here.

Let’s try it out in the game. I’ll use the existing blue sky image I have for the desert world. Let’s also remove the existing backdrop.

Hmm, the issue is trying to get the hills to look convincing, and given that the player starts quite high, and given that the camera is position to track the player near the centre of the screen, they won’t see this background layer. But this might be okay as I also need to make a distant mountain layer too.

Okay, after a lot of trial and error, I’ve got the foothills positioned in a way that works well:

In order to do this I had to do the following:

Scale them down by 50% so that they don’t dominate the screen or have too large pixel edges.
Use a horizontal scroll offset of 0.7 and a vertical scroll offset of 0.2 so they look convincing in the background. I did try a vertical scroll offset of 0.0 to keep them anchored at the bottom of the screen. But because of how the camera follows the player vertically as they jump, that just looked disorientating.

A few things I’ve learnt about using Parallax2D: you want the “repeat size” to be the size of the sprite (after transforms) so that they’re laid out seamlessly across the screen. If the scaled size doesn’t make it all away across, bump the “repeat times” up a few times. You will get the automatic repeat, which you can verify during play through by overriding the camera, zooming all the way out, and turning on 2D controls:

Now it’s time for the mountains. For this I’m going to use this asset pack for inspiration. Obviously much taller than the foothills, with jagged peaks. Less sweeping curves, and more like upside-down Vs, yet with flat tops.

Here’s attempt one:

No, don’t like it at all. For one thing the canvas is too wide in the vertical direction and too short in the horizontal direction. For another, the mountains don’t look convincing. The peaks are too pointy. So what I’ll do is cut the canvas in half and try again with a steadier brush.

Here’s attempt two:

A little better. The general shapes look okay. Might look better after colouring it in.

Okay, here’s the mountain layer with the colouring:

It’s almost convincing. Let’s try it out in the scene.

Hmm, I’m not sure I like it. It looks a little cartoon-ie, what with it’s large, sweeping areas filled with a single colour. I also don’t like the outlines: they’re a little too bold. It may help if I made the mountain layer a little smaller and reduce the motion a little.

Okay, after doing that, I like it a little more. The outlines are less pronounced and you see the more of the mountain plate throughout the level. I added back the outline in the foothill plate. I think you can either go with outlines, or you go without: mixing the two looks bad.

I’ll try this for a while. I may play around with the scroll settings a bit but it is difficult coming up with a combination that will show the mountains for more of the level while also hiding the seams between the layers. I may need to adjust the contrast a little, just so that the background doesn’t clash with the foreground: even in the screenshot about it’s a little muddy. I also need to put in a cloud layer too. Not quite sure how I’ll make convincing looking clouds, but I’ll think of something.

Tue, 14 Oct 2025 20:51:47 +1000

Finally added some checkpoints to level 1-3. Also added some animated water as a new type of hazard. The asset pack I’m using came with a single “wave” tile. I used Aseprite to shift that across 4 cells and configured it as an animation in the tile set. I was planning to add some background waves too, but that made it a little too busy and hard to discern. Less is more here.

Mon, 13 Oct 2025 21:02:13 +1000

A small devlog entry today. Just painting the level, adding background tiles and kill-zones. I also added a frame around the lift door, just to define it a little. Discovered that it’s probably best not to have sprites with non-even dimensions. It makes positioning them a little awkward, resulting in fractional offsets. I elected to include a transparent row at the top of the frame sprite, just to avoid this.

Devlog: Godot Game - A Trigger That Reveals Secrets

Sun, 12 Oct 2025 10:09:40 +1000

Okay, I think I need something to hide secret areas from the player and reveal it to them when they approach it. I’ve lived without this mechanic for a while, placing secret areas a fair distance from the main game area, so that they’re not visible. But such distances make it difficult to place time-limited things within those secret areas, like the invulnerability power-up, since travelling back to the main area to use it would eat up into the player’s power-up time. I could add ways for the player to return to the main play area quickly, like doors, but that complicates the level geometry and I need to place them out of the way, like in the sky.

So, I’ll add a new element to do this. It will be an Area2D node with an associated tile layer as a child. When the player enters the Area2D, the tile layer is hidden away and the player can see the revealed secret. The player leaving the Area2D will make the tile layer visible again.

Okay, the new SecretReveal scene has been built. The root node is an Area2D which takes, as children when added to the main scene, a CollisionShape2D to act as the trigger space, and a tile layer to act as the tiles to hide/show. The tile layer needs to be in the foreground, which is Z Index 10 and with collision turned off, so that it’ll reveal the mid and background layers. The script simply tracks entering bodies (it monitors layer 2, which is dedicated to the player).

Now, the next question is how to fade the tile map in and out. I could just toggle the visibility but that’s way too jarring. What I hope to achieve is to add an animation player to SecretReveal which will play a fade transition as the player enters or exits the Area2D. But how to do that without using scripting? I rather use the primitives offered to me as I’m guessing they’ll be more efficient than throwing events around in GDSscript.

One possibility is the Modulate option in the CanvasItem properties. After quickly playing around with this in the editor, it looks like it’s possible to adjust the alpha channel on this property, and that would adjust the visibility of the tile layer:

So this looks viable. I’m actually glad that Area2D has this property, despite not actually rendering itself in any way (outside of the editor) Let’s give it a try.

Okay, that works. Hooking it up to an animation player playing an animation that fades out and fades in the tiles when the player enters and leaves the trigger area works. It needs some improvement though. If the player were to enter and leave the target area quickly, the visibility would quickly “jump” from partially visible to fully visible. I suspect the reason for this is that the animation is always starting from the start.

One technique I used in the past is to preserve the current timestamp of the animation when I change it. But the animations here are using non-linear easing as it makes for a better transition.

So what I’ll try is a single animation that will be played forwards and backwards. When there’s a need to play the animation, the logic will detect whether the animation player is currently playing, and will resume playback from it’s current position in the desired direction.

Okay, so this is what I settled with:

func _on_body_entered(body: Node2D) -> void:
	candidate_body = body
	if animation_player.is_playing():
		animation_player.play_section("reveal_secret", \
		    animation_player.current_animation_position)
	else:
		animation_player.play("reveal_secret")
	

func _on_body_exited(body: Node2D) -> void:
	if body == candidate_body:
		candidate_body = null
		if animation_player.is_playing():
			animation_player.play_section("reveal_secret", 0.0, \
			    animation_player.current_animation_position, -1, -1.0, true)
		else:
			animation_player.play_backwards("reveal_secret")

What I’m doing is when the player enters the trigger space, I’ll play the “reveal secret” animation, either from the start or from the current position, depending if the animation player is playing. If the player leaves the trigger space, and the animation player is not playing, I’ll play the “reveal secret” in reverse from the end. When it is playing, I do something similar but I need to set the end position to the current animation position, rather than the start position. I’m guessing the animation player will take the initial position as the end when it’s playing in reverse, which makes sense.

Here’s how the transition looks now:

Much smoother.

Now to try it out in level 3-1. And yeah, this is a much better way to do secrets. I can build them close to the play area and the player no longer needs to navigate them blindly, or walk a million miles just to get to the bonus. I won’t overdo it though. I’m going to keep some of the large secrets that are naturally out of the way as they were. The player will need to react fast to enter them so I want to make sure they have the opportunity to see them.

I am wondering whether to merge this with the “You found a secret” trigger, but I think I’ll keep them separate for now.

Devlog: Godot Game - More on Level 3-1

Sat, 11 Oct 2025 20:10:10 +1000

Starting on the Godot game again. Level 3-1 is nearing its finish. I’m building the corridor/cave that’ll lead to the lift that the player will travel up to the exit. To add some interest, I’m terminating the corridor at a spike section which should be impossible for the player to navigate. But now I’m wondering: what if the player managed to navigate this with invulnerability? Maybe a secret where the player picks up a vile given them this power that would allow them to travel through this section and pick up something nice?

I’ll come back to that. I’ll build the lift first and give it a try to see how it feels.

Need to remember how these lifts work. Okay, yeah, it’s all driven by the doors. Added the lift and gave the level a test from the midpoint. Feels good. I do like the lift. I jumped around the spike section a little and yeah I think something like a secret allowing the player to travel through that section for a bonus would be nice.

First implementation of the invulnerability power-up: simple enough. The current implementation taps into the existing death handler, in which an Area2D trigger will signal to the player to die when they enter a trigger area. Note that this happens when the player enters the area. If the player is already in the area, nothing happens. This means that if the player enters an area, and stays there when the invulnerability power-up times out, they’re still invulnerable until they exit that area. A way around this is to constantly signal to the player to die while they’re inside, but I think I’ll keep it this way for now. I like the idea of giving the player just a little more time than they have to be invulnerable (although when they leave and enter a new kill zone, they will die).

Okay, implemented the invulnerability power-up and gave it a quick test in the spiky area. Yeah, it works. I haven’t added the kill zones yet so the need for invulnerability is completely unnecessary, but the progress bar has been implemented, and based on how quickly that runs down, there’s ample time for the player to get the pickups and get back before the bar runs out.

Place the exit and now just playing through the level. It needs checkpoints, and there are some tricky jumps that I’m tweaking. But I think it plays rather well. I’ll start dressing up the level now.

Devlog: Godot Game - More On Level 3-1

Thu, 09 Oct 2025 21:19:54 +1000

Working on the Godot game again. Finishing off the top layer of zone 3 in 3-1. Made a floating balloon with a smaller drop-payload trigger area, just to avoid accidental triggers if the player gets too close.

One thing I didn’t appreciate is that if the player misses a jump, it takes a long time for them to fall and recover from that. I’m wondering if I need to add some kill zones between each horizontal level of the cross-back, along some checkpoints so that the player doesn’t have to spend so much time waiting to fall and redoing large chunks of the zone.

I’ll try this will some barbed-wire graphics. See how I feel about it. Here’s how they looks:

Doesn’t look inviting, although that’s kind of the point. But this may work as a kill-zone between hight levels. Although I’m uncertain as to how I’m going to introduce them to the player. Maybe they won’t need any introduction: some ugly, spiky coils might be deterrent enough.

Anyway, the elements of the actual jumping part has been filled in. I think adding this was a good idea: feels interesting again. Now it’s just a matter of adding the final lift, and then the exit.

Devlog: Godot Game - More On Level 3-1

Wed, 08 Oct 2025 21:07:17 +1000

Continuing the build out of the zone 3 in level 3-1. I’ll start with a platform that is self rising when the player jumps on it. Always a classic; and honestly, I think the start of the level can use that. Lift does feel a little slow. I boosted the speed from 70 to 90. Also started added floating balloons with mines around this part of the level, since those elements were introduced in zone 2.

Hmm, I wonder if I should make a variant of the red balloon with mine. The mines will kill you, but the player can safely land on the top of the red balloon. I’m wondering if a variant that is unsafe for the player could be useful. Maybe a “spiky balloon” type. It can be attached to the grey mine I made for level 2-2.

Alternatively, I can make it a different colour and just make it such that landing on it will pop the balloon. Took a brief look at a colour blindness test and it looks like red and green is the way to go for two distinguished colours.

Hit a roadblock: I need to import a type, the TextureVariant enum, from one script (the static mine) into another one (balloon_with_static_mine). Quick web search and it looks like the way to do this is by “preloading” a script. This gives you access to the script’s symbols:

const StaticMine = preload("res://scripts/static_mine.gd")

@export var mine_texture_variant: StaticMine.TextureVariant

To support popping when the player lands on it, the balloon is an AnimatableBody2D which can detect collisions as it bobs up and down. I wonder if I can actually test collision that way:

var coll = animatable_body_2d.move_and_collide(velocity)
if coll != null:
    if coll.get_collider().name == "Player":
        print("Bang!")

No, turns out I can’t. Tried this and even played around with layers a little, and it didn’t detect when the player landed on it. So moving to plan B, which is adding an Area2D set to layer 2, the same layer as the player. When the player lands in that, it will pop the balloon… eventually. At the moment it just makes the balloon disappear. I’ll need to add the pop animation and sound effects. It will also give the player a jump at at about the same distance they would travel in a normal jump.

Okay, new variant implemented. Here’s how it looks:

I was going for something a little deadlier, but I think having the extra jump could make for some interesting mechanics. I guess we’ll see. Anyway, I’ll leave it there this evening. I’ve managed to get 2/3rds of new section built out.

Devlog: Godot Game - The Back Third of Level 3-1

Tue, 07 Oct 2025 21:15:22 +1000

Working on Blogging Tool was a nice little break but I’m back on my Godot game. I’m trying to get the layout of level 3-1 finished. I’m actually not very happy with the back 60% of this. For you see, world 3 is meant to be a mountainous world, with more pits and more vertical levels. Much of this is to be seen in 3-2, with 3-1 being merely the foothills of the range the player is expected to climb. And part of this is that this level was originally the second one, but some players I shared this with said the difficulty curve ramped up a little too much. So I’ve moved it in world 3. making it the 5th level of the game.

So the first part of the level sort of double backs on itself as the player is to negotiate some pretty careful jumping with moving platforms and enemies:

This is largely untouched from when this was the second level. It’s a little clunky — mainly due to some of the platforms being mistimed — but it works. This then extends to the second part, which is more linear and traditional:

It’s this part I’m not happy with. For one thing, it’s not challenging enough. Not that it has to be super challenging, but the pacing just sort of peters out here. It’s also on the smaller size, especially compare to world 2 and their significantly longer levels.

But I think I have an idea. The relatively flat section, featuring a bridge and a bit of dodging of balloons, I think I’ll keep. But for the third part, I think I’ll bring back the double backing jumping challenge, and finish it off with a lift ride. My theory is that the flat section will act as a nice palate cleanser to break up the other two “acts” of the level, along with reintroducing the floating balloons with mines. Act three will mirror act one, in the precision jumping while giving the sense of ascending difficult terrain. The lift at the end will also add to this which also giving a sense of accomplishment.

So that’s a lot of words I’ve picked up from YouTube videos about game design. Let’s see if I can execute on this. I’ll start with a pit, designed to to be an allusion to the pit at the start. I wish I can draw convincing organic terrain: it never looks right. Although the pit at the start has straight walls, so maybe this one should too.

Okay, so tried to draw some floating land masses that the player will need to navigate. Gave it a quick test, and nothing disastrous so far. The player should be able to see where they need to go while they’re traversing the horizontal parts. Verticals are another story, but the path should be easy to follow once platforms are added.

Have jumped around this area a few times and I think this may work. It makes for a more interesting area, and I can see where hazards and platforms would go. Tried it from the start and yeah, I like it. I think I’ll go with this.

Devlog: Blogging Tools - Podcast Clip Favourites

Mon, 06 Oct 2025 21:29:36 +1000

Back on Blogging Tools. Gave up on trying to set a remote ref: all the ones I’ve tried were not working. Looking at the examples it looks like I can call Push with defaults (apart from authentication which looks like I’ll require with my repo). Gave it a try and that looks like it worked. I peeked in the source code and the default ref that work was the following:

DefaultPushRefSpec = "refs/heads/*:refs/heads/*"

Turns out by adding refs/remote/origin/… I was using the default pull ref. I suppose I should learn more about how Git works in this respect. But later, it’s working now. That’s all I need. Finishing off the Blogging Tool side of things by moving all the hard coded values to the config, and I think this is ready to go.

Turning to the Hugo site. I’ve started a new site from scratch, which does little than just push to Netlify. Same template and same changes I’ve made for the test site. I don’t love the template, but I think it’ll work for now.

Created a new site and installed the theme. Set the menu and cleared the footer as per this Hugo config:

baseURL = 'https://example.org/'
languageCode = 'en-us'
title = 'Podcast Favourites'

theme = 'hugo-flex'

[params]
footer = ""

[[menu.main]]
name = "Clips"
pageRef = "/about"
weight = 1

[markup.goldmark.renderer]
unsafe = true

Produces this nice clean home page:

Added CI/CD to build the site and deploy it to Netlify. All seems to work.

Merge changes in Blogging Tools, pushed to Forgjo, and redeployed in Coolify. Status is degraded. Reason is that the temp directory, which is where the workspace for the Hugo site’s Git repo, doesn’t exist:

I forget that this is deployed in a scratch container, which has no guest OS. So creating the /tmp directory is something I’ll need to do. Will add a quick bit of logic to Blogging Tools to do this prior to creating the Git provider.

Actually, no. Change of pace. I’ll do it in the Docker file. What I’ll do is touch a file in /tmp during the build phase and just copy it over to scratch (I can’t do this in scratch as it doesn’t come with executables). This should force the directory creation.

RUN touch /tmp/t
    
# ---------  
FROM scratch  
  
COPY --from=builder /tmp/t /tmp

Redeployed and… what?

Oh, that’s meant to be:

COPY --from=builder /tmp/t /tmp/t

Copying directly to /tmp makes a new file with the filename tmp in /.

Okay, Blogging Tools is working again. Let’s try it out.

Ah, error: author field is required. I need to set this for the commit call. That may be why the commit author was me and not the bloggingtools user when I was testing this on my own machine.

Trying again. Hey, success! Blogging Tools added the clip to the repository and pushed the changes as the bloggingtools user. This kicked off the CI/CD to rebuild the Hugo site and pushes it to Netlify. Did forget to fix the base URL add the Hugo template for making the audio playable, but fixed that and now the clip is playable.

Nice! Calling this feature done. I will need to style the Hugo site, but I’ll do that over time. It’s getting a bit late and I want to wrap this up.

Devlog: Blogging Tools - Podcast Clip Favourites

Sun, 05 Oct 2025 16:52:30 +1000

Inspired by the way Ron Jeffries writes about his work, as well as by Martin Hähnel attempt at this, I thought for this instalment of Devlog I’ll try a more “lab notes” approach. This means a potentially more mundane and less satisfactory description of project work today: less showing of what was accomplished, and more of a running commentary. I wanted to see if I liked this style of writing, and if it helped me or slowed me down. Preliminary results were mixed: it did slow me down, but I found it enjoyable. I’ll try this a few more times to see how I feel about it.

Today’s coding session is to continue the work of adding saved favourite podcast clips from Blogging Tools. Previously this was a standalone application connected to a database, but along with the tech stack used to build this falling out of active support, along with the feeds no longer working for reasons I rather not look into, I want to replace this with a static site that takes nothing to maintain. One other thing is that I want to move away from referencing links to the podcast clips and saving actual audio. Since I’m dealing with copyrighted material, some of it that is for pay, this site will be kept private.

I’m choosing to use Blogging Tools for this as it has a relatively decent podcast clipper that I use for clips I publish to my site. For the static site itself, I’m choosing to look at Hugo, given that I’m comfortable with this approach too. Although I do have some concerns about the long term viability of this for sites I don’t touch that often. An initial attempt at this used another site I use to track audio that used a template that fell out of maintenance and broke when I moved to Hugo 0.151. The Hugo maintainers really need to clean up their versioning act.

Anyway, the current status of this is a temporary Hugo site with a new template. At this stage, I’ve got Blogging Tools running a new job type that will take the clipped audio and image thumbnail, save them as static data to Hugo, and publish a new post.

At the moment, the template isn’t including the thumbnail or audio HTML. This is now a point of committing to this site and template: moving it out of the temp directory and into a proper workspace. So let’s start that now. I’m using the Hugo Flex theme as it provides a nice clean canvas to start with, and it seems to be in active maintenance (something to watch out for when dealing with Hugo themes). Added the git module, and setting Goldmark unsafe to true (once again) and now the new Hugo site is ready:

Now testing this with Blogging Tools. I’ve already got an ATP clip ready to go. And after remembering to change the target directory from the temp Hugo site to the real one, I am ready to test the extract. All that involves is clicking “Save”:

Okay, that didn’t work for a very stupid reason. I set the target directory to Blogging Tools repo, not the Hugo site. Made the change (Blogging Tools is being run using Air which gives me file watching capabilities one would see in the frontend world) and can now see that the new post was written to the new Hugo site.

Now to adjust the template a little, just to add the audio and thumbnail. The Markdown files produced by Blogging Tools have all the clip information in the front matter:

---

title: "449:  An Unclean Mouse"
episode:

  guid: some-random-id
  url: "https://atp.fm/449"
  title: "449:  An Unclean Mouse"
  date: 2021-09-23 12:27:49 +1000 AEST
  image_src: "https://cdn.atp.fm/artwork-bootleg"
  audio_src: "https://example.com/file.mp3"
show:
  title: "Accidental Tech Podcast: Unedited Live Stream"
clip:
  start: 0s
  duration: 60ms
date: 2021-09-23 12:27:49 +1000 AEST
thumbnail: images/DQkt_Lk5QRjXVHE0rdR09.jpg
audio: audio/5fFX8m0BqqFqTwlVAtb2Y.mp3
---
Test thing

Oh, before I do that. Let me fix the dates should that they’re in UTC and formatted as ISO 8601. This is a quick change to the template used by Blogging Tools:

date: {{.Clip.EpisodeDate.UTC.Format "2006-01-02T15:04:05Z07:00"}}

Another quick test… perfect. The dates are coming through again. Oh, and a quick aside: because the front matter is all included in one Go template, I had to make sure to properly quote the string values, lest they contain colons and other constructs YAML doesn’t like. Easy way to do that is pipe them through printf "%q", which quotes them as if they’re Go strings. May not be perfect but it’s good enough for now:

title: {{.Clip.EpisodeTitle | printf "%q"}}

So, lets fix the template. I’ll start by creating a new template in layouts/clips/singles.html which is a copy-and-paste of the existing theme post. I’ll then add details of the front matter, such as the thumbnail and audio player. This I can get using the Params template construct. I’ll start by just printing it:

{{ define "main" }}

  {{ $context := dict "page" . "level" 1 "isdateshown" true }}
  {{ partial "heading.html" $context }}
  
  Audio = {{.Params.audio}}, image = {{.Params.thumbnail}}
  
  {{ .Content }}
  {{ partial "tags.html" . }}
  {{ partial "comments.html" . }}

{{ end }}

I had to restart Hugo, but that’s coming through. I need to convert these into proper URLs though. I’ll start by trying the absurl template function:

Audio = {{ absURL .Params.audio}}, image = {{ absURL .Params.thumbnail}}

Okay, getting better. Let’s try and turn them into proper HTML elements:

  <img src="{{ absURL .Params.thumbnail}}">
  <audio src="{{ absURL .Params.audio}}" controls></audio>

Good, the image and audio is coming through. Image looks a little large but I’ll fix that at little later. Oh, and it looks like the clips are showing up in the home page already. That’s a nice touch. I do want to change the clip paths though. At the moment, they’re saved using the auto-incrementing clip ID, which is not ideal. Clip metadata is stored in an Sqlite database and the library I’m using — which is a Go port of Sqlite3 that doesn’t need CGO — has a nasty habit of reusing row IDs. I really should replace this with an auto-generated primary key. But that’s for later.

Right now, I think I’ll use a hash of the episode Guid for the path. ATP seems to have randomly generated hashes but I know that some RSS feeds choose to use URLs, which won’t do for a path. Why the Guid? Well, I want to be able to regenerate a clip if need be, replacing the existing one.

Actually, no, that won’t do. I want to be able to create multiple clips from the same episode. So maybe I’ll use a hash that includes the Guid and clip timestamp. So let’s try that.

clipPathRaw := fmt.Sprintf("%v:%v", clip.EpisodeGUID, clip.ClipStart)  
clipPathHash := md5.Sum([]byte(clipPathRaw))  
clipPath := hex.EncodeToString(clipPathHash[:])  
  
postFilename := filepath.Join(postDir, fmt.Sprintf("%v.md", clipPath))

MD5 should be fine: these don’t need to be cryptographically safe. Also, md5.Sum returns an [16]byte array, and I keep forgetting how to return this as a []byte slice. Apparently it’s array[:].

Another quick test. Yeah, it works. The clip paths now look like this: http://localhost:1313/clips/210da5b665c38c809eb1ea481b1b22de/ Not great, but I tend to browse this visually so I think I can live with this.

Okay, next thing is having Blogging Tools generate the clips and commit them to Git. The goal is to have the Hugo site stored in a Git repository and built to a private, undisclosed website. I did this for Nano Journal, which synchronised journal entries to a Git repo after saving it locally. Looking at the code, though, I have no idea how I setup the credentials. There’s nothing in the config hinting at a certificate so I may have done it manually.

I’m also wondering if I can use Go Git for this. I didn’t use this for Nano Journal, as I wanted to use LFS; which Go Git doesn’t support (instead I just shelled out to git). But I’m wondering if I can live without LFS for this.

Okay, let’s try this. I’ll push the current Hugo site to a test repo on my Forgejo instance and see whether I can check it out from Blogging Tools. This will be done as a user that I’ll configure Blogging Tools to act as.

Then I added Go Git to blogging tool and created a new provider for Git. Added a method to either pull or clone a remote repository, checkout a branch, and pull updates from the repo. I’ve chosen to use the file system for keeping the local workspace so that Blogging Tools doesn’t need to keep cloning the repo whenever there’s a need to add a clip. This I’m just going to keep in ephemeral storage so that when the Docker container gets cycled and the workspace lost, it’ll just recreate it.

func (p *Provider) CloneOrPull() error {  
    var (  
       repo *git.Repository  
       err  error  
    )  
  
    if !isDir(p.workspace) {  
       // Workspace doesn't exist. Pull from repo.  
       repo, err = git.PlainClone(p.workspace, &git.CloneOptions{  
          URL:               p.repoURL,  
          RecurseSubmodules: git.DefaultSubmoduleRecursionDepth,  
          Auth: &http.BasicAuth{  
             Username: p.username,  
             Password: p.password,  
          },  
       })  
    } else {  
       repo, err = git.PlainOpen(p.workspace)  
    }  
    if err != nil {  
       return fault.Wrap(err)  
    }  
  
    worktree, err := repo.Worktree()  
    if err != nil {  
       return fault.Wrap(err)  
    }  
  
    if err := worktree.Pull(&git.PullOptions{  
       RemoteURL:         p.repoURL,  
       RecurseSubmodules: git.DefaultSubmoduleRecursionDepth,  
       ReferenceName:     plumbing.ReferenceName(p.branch),  
       Auth: &http.BasicAuth{  
          Username: p.username,  
          Password: p.password,  
       },  
    }); err != nil {  
       return fault.Wrap(err)  
    }  
  
    if err := worktree.Checkout(&git.CheckoutOptions{  
       Branch: plumbing.ReferenceName(p.branch),  
    }); err != nil {  
       return fault.Wrap(err)  
    }  
  
    return nil  
}

Quick test of this, and I’m seeing a “repository doesn’t exist” error.

Actually, no. That was because of an earlier use of the repository URL when I actually wanted the workspace. But now I’m seeing a “reference not found” error when I try to pull from the origin:

Ah, okay. I should’ve used plumbing.NewBranchReferenceName here. That’s fix that. Now it’s an already up-to-date error. That’s fine, I’ll just add an ignore case here:

if err := worktree.Pull(&git.PullOptions{  
    RemoteURL:         p.repoURL,  
    RecurseSubmodules: git.DefaultSubmoduleRecursionDepth,  
    ReferenceName:     plumbing.NewBranchReferenceName(p.branch),  
    Auth: &http.BasicAuth{  
       Username: p.username,  
       Password: p.password,  
    },  
}); err != nil {  
    if !errors.Is(err, git.NoErrAlreadyUpToDate) {  
       return fault.Wrap(err)  
    }  
}

Another test:

And that worked. I managed to pull changes from remote.

Deleting the workspace and trying a clone. Hmm, that’s strange. I’m seeing an authentication required error, yet it looks like the repository was cloned successfully. Let me try and ignore that error.

if repo == nil && err != nil {  
    return fault.Wrap(err)  
} else if repo != nil && err != nil {  
    log.Printf("warn: error received alongside repo: %v", err)  
}

Okay, that worked. It’s not pretty, but I think it’s fine for now.

Now to commit the new files to the repository. Although partial failures from staging multiple files — one file is stage but the second one fails for some reason — may introduce some problems where the state is not completely settled, so I may also need a method of ensuring the workspace is pristine: no files should exist after cloning or pulling that is not staged. So need to add that first. I don’t see a method to do this on the Workspace type, so basically what I’ll do is reset hard, and then iterate over the untracked files and remove them:

if err := w.Reset(&git.ResetOptions{  
    Mode: git.HardReset,  
}); err != nil {  
    return fault.Wrap(err)  
}  
  
status, err := w.Status()  
if err != nil {  
    return fault.Wrap(err)  
}  
for f, s := range status {  
    if s.Worktree == git.Untracked {  
       if os.Remove(f) != nil {  
           return fault.Wrap(err)
		}
	}
}

Well, that’s the plan eventually. I’ll just log them for now.

Then it’s just a matter of adding files and commit the changes. First attempt at this failed with an entry not found error. That was just a bad path. Adding files now working, along with committing, and now we’re failing at the push. Reason is that unlike all the other Go Git methods, which were pretty high level, this one leaks a bit of the mechanics of Git, requiring the use of a Ref Spec. I’ve just hard coded it to a version which will push the local main branch to the origin main branch.

if err := repo.Push(&git.PushOptions{  
    RemoteURL: p.repoURL,  
    Auth: &http.BasicAuth{  
       Username: p.username,  
       Password: p.password,  
    },  
    RefSpecs: []config.RefSpec{  
       config.RefSpec(fmt.Sprintf("+refs/heads/%v:refs/remotes/origin/%v", p.branch, p.branch)),  
    },  
}); err != nil {  
    return fault.Wrap(err)  
}

Oh, before that, I finding a cannot create empty commit: clean working tree. I’m opening the work tree between staging the files and committing. Is that the problem?

Hmm, no. It might be that I’m adding the full path of the file to the index. Running git status in that repository yielded full path entries, which is very uncharacteristic. Furthermore, trying to unstage them did nothing. I’ll remove the repo directory and change the logic to get the relative path of the file before they’re added:

for _, f := range files {  
    relPath, err := filepath.Rel(p.workspace, f)  
    if err != nil {  
       return fault.Wrap(err)  
    }  
  
    if _, err := w.Add(relPath); err != nil {  
       return fault.Wrap(err)  
    }  
}  
  
if _, err := w.Commit(msg, &git.CommitOptions{}); err != nil {  
    return fault.Wrap(err)  
}

Oh, it was because the files I was trying to add were already added to the repository from earlier in the day, and they weren’t being changed by the job. Let me remove them from the repository and try again.

Okay, it was that. Commits are now working, although pushes don’t seem to be synchronising the commits. When I look at recent changes in Forgejo, it only shows my commits, not the ones created by the service user I made for Blogging Tools. Will try adjusting the refspec to push all heads:

RefSpecs: []config.RefSpec{  
    "+refs/heads/*:refs/remotes/origin/*",  
},

Nah, that didn’t help. The method is returning no error, yet it doesn’t seem to be synchronising the branches. Will need to learn more about how this all works. But I think that’s something for later.

Tue, 30 Sep 2025 22:14:06 +1000

Bit more on lifts in Godot today. Re-engineered how the whole thing works: now everything is driven by the stationary lift doors. Each door has a reference to a lift and a target door. This makes the lift carriage itself rather passive: it will continue to reparent the player as before, but it no longer needs to track activations or have animating doors of it’s own. The doors just tell it where and when to go.

This has got proper targeting working so that a lift can now move between a pair of doors. It also allows for the player to call for the lift when it’s not positioned at the doors already. There are a few more dependencies amongst the various entities, but I think it makes for a more robust system.

Sun, 28 Sep 2025 11:09:06 +1000

Small change to the thirst mechanic for my Godot project. Switched from discrete thirst levels to a single timer that will tick down if the player is thirsty. This allowed for a change to how I indicate this to the player, replacing text messages that’ll be displayed at each thirst level with a gauge that shows up on the HUD.

I thought that the messages would be enough, but after playing through with them, they turned out to be more of a hindrance. They didn’t communicate the player’s thirst level well enough: they show up for a few seconds, then disappear, leaving the player to wonder how much time they have. This information is now always present with the gauge. Plus, I think it gives more of a sense of urgency, in that a gauge that’s constantly ticking down will encourage the player to play a little faster in order to reach the next water bottle before they perish from thirst.

Sat, 27 Sep 2025 11:02:07 +1000

More on that Godot project today. Finally bit the bullet and started working on proper backdrops. I’ve leaving the artwork until later (i.e. I’m procrastinating), today was all about the mechanics, trying to get parallax scrolling working. Fortunately Godot has built-in nodes to make this easy, although I did have to upgrade Godot to 4.5.

This is my first cut, some simple colour banding to represent sky and sand in one of the desert levels:

These two layers were added to the scene as Parallax2D nodes, each with a single Sprite child. Positioning the sprites was a little tricky. Godot suggests placing them at the origin, but what I didn’t understand was that the actual position of the sprite depends a lot on the scroll scale of the Parallax2D node. I added the sky layer without changing the scroll position and it took me a while before I discovered that it was appearing below the camera. Only after reducing the scroll scale to about 0.1 did it start showing up in the viewport (one nice thing about Godot 4.5 is that the “play scene” window includes debug options, allowing you to hijack the camera and move it around the scene).

I will need to introduce some detail in the backdrop layers soon. The bands are too simple and may induce some vertigo when the player is ascending to high platforms. Plus, it just looks boring. Not that they’re meant to be eye-catching but something a little more interesting would be nice.

Tue, 23 Sep 2025 22:10:06 +1000

Small DevLog update today. Continuing to work on that Godot game. I got the bulk of the mechanics of level 2-1 built (it still needs dressing up), and I spent this evening mainly just play-testing it, and tweaking it. I think I may need to get someone else to play-test it though, just to make sure it’s not too hard. It’s significantly long, coming in at 10,560 units horizontally, and given how many new mechanics are involved, I’m a little worried about the difficulty curve. But I am happy about the mix of elements I do have. It doesn’t feel boring, which was the concern I had. There are some timing challenges, and I the pacing across the level feels fine. But I really should convince someone else to try it out.

Devlog: Trying OpenAI Codex to Produce Freelens Logo Creator

Sat, 13 Sep 2025 09:05:30 +1000

Tried a bit of vibe coding using OpenAI Codex yesterday.

I needed something to produce nice logos for Freelens so I can tell each cluster apart. The default avatar for Freelens is an abbreviation of the cluster name set atop a random background colour, which doesn’t do me a lot of good as the cluster names I work with are all similar, and would produce the same abbreviations. Furthermore, Freelens doesn’t have a way to change these default avatars. But it does have a way to upload new avatars as an image.

This was a perfect candidate for trying out OpenAI Codex. A use-case that I needed but was not important nor interesting enough for me to build manually. I mean, knowing me, I probably would’ve built it, but it wouldn’t have been good use of my time. So I created a new Github repo, connected it to Codex, and gave it this prompt:

I would like a static web-page, consisting of one HTML file, and one JavaScript file. This is to be vanilla JavaScript that does not require NPM or any external dependencies.

This page is to have a form which accepts the following input:

A text input with the name “Upper” which accepts a string of any kind.

A text input with the name “Lower” which accepts a string of any kind.

A selection for background colour. Instead of a colour picker, please allow the user to select one of 8 distinct colour choices. Each one should be distinct enough such that the user is able to distinguish them from each other. They should be of medium darkness, such that if white text were to be display atop of it, the text should be legible; yet not too dark as to be hard to distinguish.

The page is also to have a canvas element, of about 80 x 80 points, which will take input of the form above and do the following:

Fill the background with the chosen background colour.

Draw a thin dividing line — no greater than 6 px wide — in white across the width of the canvas.

Print the form value of “Upper” in uppercase, in a san serif font, centred and above the dividing line in white. The printed text should be large enough to fill most of

Print the form value of “Lower” in uppercase, in a san serif font, centred and below the dividing line in white.

Ensure that a suitable margin of 15 pixels padding around the border of the canvas content, and that the printed text and dividing line is not drawn beyond the margin.

The canvas element is to be updated when the form values are changed.

Below the canvas should be a button which should allow the user to download the canvas image as a PNG.

And here’s what it produced:

It pretty much did what it said on the tin: when I changed the value of upper or lower, or changed the background colour, the canvas would redraw showing a different preview. Clicking “Download PNG” would allow me to save it as an image.

On the whole, what I received was pretty good. It’s not perfect — there’s no styling; orange is a bit too close to amber, as is green and teal; and the alignment of the bottom line is slightly off. But it’s a pretty decent start.

There were some things I wanted to change, so I went in myself. I found a nice looking stylesheet called Pico CSS, which I’ve applied. Normally I would’ve turned to simple.css, but I’ve been using that a lot and I wanted to try a nice, fresh look. I also adjusted the vertical positioning of the lower row in the icon, added support for single line icons, and adjusted the colours a little.

The whole thing looks quite nice now, and I’m quite happy with the icons it produces:

I’m still working out how these AI coding agents will fit into my life. I’ve been using Claude Code occasionally, and that’s been a good fit for projects where I’d prefer to be the driver. But there are occasional things that I’d like to see exist but are a little too trivial to spend the time on. In those cases, I may not have built it at all; or I would’ve built it and then immediately regret spending the time doing so it. And that might be where Codex could be useful: it provides a good starting point, which I can then refine.

If you’re interested in trying this tool out, you can find a link to it here. The source you can find on GitHub.

Sun, 07 Sep 2025 11:14:00 +1000

Ah, MacOS’s locked-down nature strikes again! Was testing the CI/CD build for Dequoter and after downloading the artefact and attempting to open it, I got this warning message:

Turn’s out it was being quarantined by MacOS, and these instructions resolved the issue:

xattr -dr com.apple.quarantine '/Applications/Your Application.app'

The binaries not notarised so I wasn’t expecting it to work out of the box. I was hoping that it would do that thing where the app will be listed in settings and I can allow it to launch from there, but I guess there’s something about where this file came from that was too much from MacOS. Ah well, I can live with this for the short term.

Devlog: Dequoter — Something Different Today

Sat, 06 Sep 2025 16:10:04 +1000

When it comes to hobby projects, I’m trying to reduce the number of things I build for work. But I’ve decided to revisit one that I’ve abandoned, given the frequent need for this. The issue is simple: I have a JSON data structure that is quoted within a string, and I want to “unquote” it and filter the JSON so that I can read it.

So I thought I’d spend today restarting the Dequoter project. The best way to describe it is a Boop clone. I like Boop, I use Boop. And I know Boop is extendable, so I could implement any processor I need using JavaScript. But I’m not a huge fan of JavaScript, and Go has the function I need right there in the standard library. And with most things I encounter, I want to build my own. I know I shouldn’t, and I’ve been better at resisting. But I’d figured this could be a nice little distraction. And I thought I could get much of what I need done in a morning.

So I started a new Wales project and after about 3 hours, I had a scratchpad with an invokable command palette that I can use to dequote a string and format a JSON object:

Granted, it’s not much for 3 hours. While I did use Claude Code a little, much of this was hand rolled. And I spent a lot of time learning the ins and outs of Code Mirror. This is why I wanted to use Wales this time around. The previous run at this used Fyne, which was really nice to use, but the available widgets were quite limited. The nice thing about Wales is that I can build the UI in HTML and it uses the built-in HTML renderer from the OS: no need to bundle Chrome. The frontend is mainly vanilla HTML and JavaScript, although I am using Stimulus to help with some of the controls. The backend is, of course, Go.

Final build size is 8.4 MB.

By the end of this morning’s session, what I had was not pretty, but it’s functional and it already does what I need it to do. I do have some plans for integrating this with UCL, and using that to define processors. That should hopefully leave all the heavy core stuff to be built near the beginning, and that could sit pretty while the extensions are added in a nicer language. Anyway, more on this if this goes anywhere.

Wed, 03 Sep 2025 21:27:26 +1000

Starting to work on the background tiles. This is what I have so far. I hope it’s not too busy or distracting.

Devlog: Godot Project — Bricks in Level 2-3 Laid

Mon, 01 Sep 2025 21:31:56 +1000

Just a quick update today. I’ve finished all the brickwork in level 2-3. And it didn’t go too badly. Made one significant mistake which would’ve involved a lot of rework, that I patched up with some single tiles:

Top: the mistake. Bottom: the fix.

Doing the rest of it was pretty dreary work. Godot does have some tools to make this easier, but there was no getting around the level of care needed to place the bricks correctly. But it’s all pretty much done now. And just for comparison to the before screenshots I took when I started, here’s how how the level looks now:

There’s still plenty of work. The background is not yet done, nor are any of the pickups placed. The HUD needs updating to show the key gems the player has, and I also need to repair some dodgy mechanics around moving platforms. But I guess that’s just a matter of plowing ahead.

Devlog: Shutting Down Nano Journal

Sat, 30 Aug 2025 15:34:43 +1000

With the move to Obsidian for my journalling needs, I shut down my bespoke journalling web-app. I deployed it on 26th August 2024, which makes it just over a year old. I did start using Obsidian on the 20th though, so it didn’t quite make it the entire year. Even so, not bad for something hand made and somewhat neglected. Most things I eventually abandon last way less than that.

Anyway, here are some screenshots of the final version, just before I shut it down:

The landing page, with a full-width post text area.

A post with attachments.

Clicking 'Posts' will bring up a chronological list of posts from the last 6 months. Those with a paperclip are ones with attachments.

An example of a viewing a post. Attachments would appear below the body.

You can compare this with some earlier screenshots documented in these posts:

22 August 2024: Development work started
15 September 2024: Addition of attachments and titles
4 October 2024: Adding Git synchronisation of entries

I guess you can describe the ascetics of this project as the “minimum amount to get working.” That was true of the backend too, which was somewhat rushed and difficult to maintain. That might be why I never really gave this a lot of love as other projects like Blogging Tools. But I added some features I that I thought were neat: such as attachments, and keeping a copy of a draft in the browser’s local storage before it was sent to the server.

Anyway, I’m now in the middle of migrating the old posts over to Obsidian. One thing I’m glad I added was Git synchronisation. And given that entries were plain markdown files, migration was simply a matter of checking out the repository and moving the posts across. Easy enough work, although I am taking my time. There’s a bit of reliving involved with moving the posts over, and I’m approaching a period where some pretty sad things happened.

Anyway, that was Nano Journal. Good at what it needed to do.

Devlog: Godot Project — Level 2-3 Update

Tue, 26 Aug 2025 22:27:58 +1000

Been working on level 2-3 these past few weeks. And it’s a large one: 4,720 x 2,928 units. There are longer levels but those tend to be quite horizontal, unlike this one which feels like a mixture of the two dimensions:

Reached a milestone this evening: the critical path has been built. And yeah, it ended up pretty much being a standard key-hunt. I did have an idea for having more puzzle elements, like requiring the player to work out combinations based on key colour, but given the length one has to travel to get the necessary keys, it felt like a hit to the pacing. I figured it would be more interesting to vary the mechanics in each zone the keys are located. They’re not complicated mechanics, but they did reach the limit of what I had in my toolbox. It got to the point where I started adding the classics, like dart shooters which fired projectiles.

Pick up these gems…

…and place them on this holders to activate things. And yes, I stole this directly from Commander Keen.

Given that this is in the desert world, the sand spinners make an appearance.

The new dart shooters and the projectiles they fire.

There are a few mechanics I wanted the player to learn. Trying to telegraph that via gameplay is not easy, but I’m hoping it will comes through. Some approaches I’m trying are slight cues in colour and making sure interaction indicators are clear. And also level geometry, such as moving the player through one-way gates and making it obvious that all they have to proceed is right in front of them.

I’ve also decided to move away from paid checkpoints, where you need to pay 5 coins to save your progress. I’ve replaced these with hidden checkpoints, of which this level has a few. I think given the size, and that some backtracking is involved, it makes more sense being quite generous with checkpoints in this level. I think I’ll keep the 5 coin toll gates though for other things.

Work remaining on this level is prettying it up (it’s still rather barebones) plus adding pickups and maybe some bonuses. Painting in the brick tiles will take a while, but could be a nice exercise in calming work when one can switch off their brain somewhat. I also need to make the background tiles and (ugh) the actual level backdrop. Not quite sure what that will be. Something deserty no doubt.

Devlog: UCL — Comparing UCL To Some Early Ideas

Fri, 22 Aug 2025 11:14:02 +1000

I was browsing through some very old notes I had when I came across one that contain an idea for a hypothetical shell-like command language, sort of what UCL was designed for. It was designed for a project called Nuget, which was a CLI torrent downloader (now defunct). Much like UCL, it was REPL based with a simple token-based command language, and I was thinking of ways to extend this to including scripting. Here’s the note in full:

The nuget command language will be something very similar to shell. It will be used to configure nuget (in an RC file), and configure extensions.

Example - check that the VPN is running
let piaCtlState = !piactl get connectionstate

if eval 'trim(${piaCtlState}) ne "Connected"' {
    notice "Warn: VPN not connected.  Use 'piactl' to connect"
}

on newjob {
    on done { echo "Job done: ${job}" }
}

I never implemented this language, opting to simply use TCL. But it’s interesting to see the desire of having an embeddable language for both REPLs and scripting alive and well back in February 2021 (the note’s datestamp).

Assuming that UCL existed at the time, I image the script would’ve looked a little less TCL-ly and a little closer to how Go templates work. Maybe something likes the following, assuming that the built-ins were implemented:

piaCtlState = os:shell "piactl get connectionstate"

if (ne (strs:trim $piaCtlState) "Connected") {
    notice "Warn: VPN not connected.  Use 'piactl' to connect"
}

# I'm guessing 'on' in the original idea would have job as a global variable.
# UCL can simply pass it in as an argument.
on newjob { |job|
  on done { echo "Job done: $job" }
}

Devlog: Blogging Tools — All About Images

Sun, 10 Aug 2025 21:30:40 +1000

I’ve been working on Blogging Tools on-and-off the last few weeks, mainly around images and image management. They’re not super exciting in their own right so I’d figure I’ll briefly write about them here.

The first major addition is the ability to save images to the server from the Image Tool app. There are some composition processors available and if one wanted to pre-process the image in some way, they would have to download the pre-processed image and upload it again for composition. This change will hopefully reduce the need for this back-and-forth. The download button has been replaced within the Image Tool app with a Save button, which will upload the processed image to the server.

Saved images can be recalled from the processor by using a new picker. This replaced the Load image by URL option, which I never really used (although I may add it back within the Saved Image app).

There is also a new Saved Images app which allows one to browse the saved image, file them into categories, and download them to the local file system.

Now those with keen eyes may notice the quality of the saved images in the screenshots above. That’s because the second change I made was to integrate calls to an image generator. There was no reason for doing this other than to try out the API, and I haven’t really used it for anything other than make test images. The API I’m using is Flux 1 provided by together.ai, but there’s also a partially-functional version of Gemini’s API too (the API is region locked and is not available in Germany, which is where Blogging Tools is running).

Saved Images are build on the existing Files subsystem, which has been relegated to a new Settings section, the third change I’ve made to Blogging Tools. This allows one to manage files, see running jobs, and managed sessions from the system. That section may expand in time as I think of things I’d want to change without redeploying the app (most of the config is managed using environment variables).

The last change was the addition of a new notification type. I’ve added photo collections to my blog a few months ago, but I never remember to actually file photos into any of them. My workflow with making a photo post usually ends at the point I hit “publish”, and as far as I can tell, there’s no way to set the category of the image from the new post screen.

So I’ve added a new notification type to help me with this. When the RSS feed of my blog is polled, and a new post with the “Photos” category is found, the first 4 images are extracted and presented as a notification asking if I want to file the photos into a photo category.

This also worked for posts you add to the “Photos” category via the notification subsystem itself. Both act on the same RSS feed and setting the category of a post to “Photos” will clear the fetch history of that item so that it can run through the processors again.

I’ve decided to use notifications for this, rather than add something to do this automatically, as I do want to go through the candidate photos first. But there are some limitations to this approach. For one, the notification system only allows one user selection per notification, making it impossible to add a photo to more than one collection. And to keep the number of notifications low, only the first 4 images are presented, with the rest being ignored. I have some ideas of how this could be improved in the future — probably by adding a separate app — but for now, I think I could live with taking corrective action manually.

One last think about this new category processor: it’s activated by a change to the RSS feed, but it scans for images using the original post retrieved from Micro.blog’s micropub API. The reason for that I’ve enabled the CDN option, and the image URLs which use the CDN do not work with the collections API. It took me a while before I figured this out, mainly because I wasn’t using a CDN on my test blog and that worked perfectly. Something to keep in mind for anyone else using this API.

Devlog: Godot Project — Level 4-2 And Level 2-3

Sat, 26 Jul 2025 15:47:11 +1000

I can’t quite remember how far I got into building of level 4-2 when I wrote my last Dev Log entry. Based on how I described it, I think I was still working on the bones of it: a level with little more than the tile-map. I got a fair bit of the level built since then. It now features three major zones, each with a different camera lock. The central mechanic of a tile layer that can be raised by hitting switches has been built, along with it’s variations within each zone. Pickups and enemies have now been placed. And decoration. A lot of decoration.

I wish I had a screenshot of how it looked when I wrote that last entry, but here’s a shot of how it looks now:

I’m quite happy with how the mid-layer tiles add some interest to the aesthetics. But while thinking of the actual background to set the scene, I now feel that simply adding sky-coloured tiles is not enough. I may need to make a proper landscape backdrop, for no other reason than to give the level some place. Not entirely sure how I’m going to do that.

Anyway, that’s for later. For now, I had another go at working on world 2. Level 2-1 is still barely started: I haven’t returned to it yet, and probably won’t until I have an idea of how to put that level together. I do have some ideas for level 2-2 and 2-3 and to ease myself in working on them, I started the week by simply building game elements, like a thirst mechanic (world 2 is a desert, as pre-ordained by Yahtzee Croshaw). More on those in the future I’m sure, but it was enough to get me to start work on level 2-3, which I’m thinking might be set in an underground pyramid with some windy maze elements (original, I know). Not too many windy maze elements, mind you: I generally don’t like playing maze levels. But I think a little would add some variety to what I’d imagine would be simply jumping puzzles until then. So far this has meant reverting to “playing the hits” as it were, by starting off with adding a simple lock-and-key hunt, albeit with gems (shout-out to my fellow Keen 4-6 fans). I do have a twist in mind which I hope to slowly introduce to the play as they progress through the level.

I’ll just finish this update off with some screenshots of how level 2-3 looks now in it’s super early state.

So that’s where things currently stand.

Devlog: Blogging Tools — Category Fixer

Sun, 20 Jul 2025 23:09:07 +1000

I was going to say that this one was pretty quick to implement, but thinking of it now, it took several hours all up, so maybe it wasn’t that quick. It’s a feature I’ve creatively called the “category fixer”. Yeah, the name’s not great, but it’s been a feature I’ve been wishing for a while, and it’s nice to see it added to Blogging Tools.

I post blog posts of images which could either be a photo or a screenshot, along with the occasional meme. I have categories for all three, and when I create a new post with an image, I try to make sure the appropriate category is set. Only issue is that I usually forget, and most image posts remain without any category at all. Micro.blog does has a feature of automatically adding posts with image tags into a photos category, but I’ve turned that off as I don’t like the idea of screenshots going there.

So I needed a way to fix this, and that’s what this new feature does. I’ve added a RSS feed parser which will poll my blog’s RSS feed for any new or updated posts. If there are any, the HTML content will get parsed to find at least one img. If it does, an in-app notification will be added. I can then go in and triage these posts, selecting the category they should be sorted into.

The in-app notifications, accessible from a new top-level menu item.

Tapping the disclosure icon reveals the first image found, along with a link to view the entire post.

Selecting an option will update the notice then and there (using HTMX). The notice isn't cleared until the category change has been made, which does require a page refresh.

About half the work was building out these in-app notifications. They’re all stored in a sqlite3 table and mainly consist of a notification type, brief message, and an extendable set of properties which are stored as a JSON object.

For the feed parser, I’m using gofeed, the same library I’m using for reading podcast feeds. I can definitely recommend this library for parsing RSS feeds if you’re using Golang. There was a little bit I had to add around this library to detect when feed items have previously been read or updated. The way I’m detecting this is by recording the GUID, along with a SHA-1 hash of the title, description, and content in the database. Parsing the feed HTML is done using goquery, allowing me to easily select the first img tag of a HTML fragment. For fetching and setting the categories, I’m using a home grown Micropub client.

So that’s “category fixer”. A lot of infrastructure was added for this one which could be useful in the future. I do have some other ideas that would follow this sort of pattern: a new post is picked up by the RSS reader, and an in-app notification is raised for me to action. I suspect this may becoming a bit of a pattern.

Devlog: Godot Project — Some Feelings

Sat, 12 Jul 2025 15:43:59 +1000

Spent a little more time working on my Godot game, mainly around level 4-2. Got around two-thirds of it done, including one of the more difficult thirds to build. It’s nice seeing the ideas I had for this level come together. It’s not exactly how I envisioned it, but truth be told, I didn’t have many ideas solidified before I started building the level. Just some vague ideas of how I wanted the level to look, then I went where my virtual paintbrush took me.

Now, some feelings about this entire project. It’s hard spending time on this knowing that it’s not really going to amount to anything. I’ve been watching a lot of game reviews recently, particularly from Yahtzee Croshaw, who is also a game developer. Seeing his work, along with the work he critiques, is a little discouraging. Here are accomplished game developers doing amazing work, making games that move emotions and look and sound amazing. And here’s this project, using assets from a Godot tutorial and is little more than another Mario-esc clone. And I know going in to this that this will be little more than a bit of a time-waster and an excuse to try Godot out: something to work on evenings and weekends.

But there’s always this niggling feeling saying, “if you’re not expecting anything, why are you wasting your time on this? Why don’t you kill this now and focus on something bigger? ‘Make no small plans’ and all that.” And once you found the answer surrounding the underlying question motivating you to start the project in the first place, if you know it won’t amount to nothing, why continue?

And yeah, ultimately might be good advice. But I think I’ve abandoned too many projects in the past. I’d like to see something through, even if it’s something other than this. So maybe I will kill this. But not yet.

Devlog: UCL — More About The Set Operator

Sat, 14 Jun 2025 11:35:00 +1000

I made a decision around the set operator in UCL this morning.

When I added the set operator, I made it such that when setting variables, you had to include the leading dollar sign:

$a = 123

The reason for this was that the set operator was also to be used for setting pseudo-variables, which had a different prefix character.

@ans = "this"

I needed the user to include the @ prefix to distinguish the two, and since one variable type required a prefix, it made sense to require it for the other.

I’ve been trying this for a while, and I’ve deceided I didn’t like it. It felt strange to me. It shouldn’t, really, as it’s similar to how variable assignments work in Go’s templating language, which I consider an inspiration for UCL. On the other hand, TCL and Bash scripts, which are also inspirations, require the variable name to be written without the leading dollar sign in assignments. Heck, UCL itself still had constructs where referencing a name for a variable is done so without a leading dollar sign, such as block inputs. And I had no interest in changing that:

proc foo { |x|
    echo $x
}

for [1 2 3] { |v| foo $v }

So I made the decision to remove the need for the dollar sign prefix in the set operator. Now, when setting a variable, only the variable name can be used:

msg = "Hello"
echo $msg

In fact, if one were to use the leading dollar sign, the program will fail with an error.

This does have some tradeoffs. The first is that I still need to use the @ prefix for setting pseudo variables, and this change will violate the likeness of how the two look in assignments:

@ans = 123
bla = 234

The second is that this breaks the likeness of how a sub-index looks when reading it, verses how it looks when it’s being modified:

a = [1 2 3]
a.(1) = 4
$a
--> [1 4 3]
$a.(1)
--> 4

(One could argue that the dollar sign prefix makes sense here as the evaluator is dereferencing the list in order to modify the specific index. That’s a good argument, but it feels a little bit too esoteric to justify the confusion it would add).

This sucks, but I think they’re tradeoffs worth making. UCL is more of a command language than a templating language, so when asked to imagine similar languages, I like to think one will respond with TCL or shell-scripts, rather than Go templates.

And honestly, I think I just prefer it this way. I feel that I’m more likely to set regular variables rather than pseudo-variables and indicies. So why not go with the approach that seems nicer if you’re likely to encounter more often.

Finally, I did try support both prefixed and non-prefixed variables in the set operator, but this just felt like I was shying away from making a decision. So it wasn’t long before I scrapped that.

That Which Didn't Make the Cut: a Hugo CMS

Mon, 09 Jun 2025 16:32:49 +1000

You’ve probably noticed¹ that I’ve stopped posting links to Open Bookmarks, and have started posting them here again. The main reason for this is that I’ve abandoned work on the CMS I was working on that powered that bookmarking site. Yes, yes, I know: another one. Open Bookmarks was basically a static Hugo site, hosted on Netlify. But being someone that wanted to make it easy for me to post new links without having to do a Git checkout, or fiddle around YAML front-matter, I thought of building a simple web-service for this.

I don’t want to say too much about it, other than I managed to get the post functionality working. Creating a new link post would involve fetching the page, and pre-populating the link and optional via link with the fetched page title. I’d just finish the post with some quotes or quips, then click Post. That’ll save the post in a local database, write it to a staged Hugo site, run Hugo to generate the static site, and upload it to Netlify. There was nothing special about link posts per se: they were just a specialisation of the regular posting feature — a template if you will — which would work the same way, just without the pre-fills.

The other thing I added was support for adding arbitrary pages. This was a dramatic simplification to what is possible in Hugo, in that only one page bundle was supported. And it was pretty compromised: you had to set the page title to “Index” to modify the home page. But this was enough for that, plus some additional pages at the top level.

One other feature was that you can “preview” the site if you didn’t have a Netlify site setup and you wanted to see the Hugo site within the app itself. I had an idea of adding support for staging posts prior to publishing them to Netlify, so that one could look at them. But this never got built.

Finally there were some options for configuring the site properties. Uploads were never implemented.

Here are some screenshots, which isn’t much other than evidence that I was prioritising the backend over the user experience:

Once authenticated, you're presented with this: the list of your sites.

Clicking on a site opens up the list of posts. Clicking 'New Post' will take you directly to the editor.

Clicking 'New Link Post' will bring up this form, where you can specifying the post link and 'via' link.

Clicking through will preload the post title and body with the page title and URL. If a via is set, the page title and URL of the via link will also be included.

Also supported were the ability to add standalone pages, accessible by clicking the 'Pages' section in the side-bar. Most created sites come with an index page (which needed to have the title 'index').

The 'Settings' nav item bought up the site settings, including the site tile, theme, and Netlify site ID.

So, why was this killed? Well, apart from the fact that it seemed like remaking prior art — I’m sure there are plenty of Hugo CMSes out there — it seemed strange having two systems that relate to blogging. Recognising this, I decided to add this functionality to Blogging Tools:

The 'Posts' app in Blogging Tools. Unlike Hugo CMS, this is just a databased-backed list of posts.

The goal with this app is to easily create posts from templates, so there's no way to create a blank post. Clicking New will require one to select a template to use. There are currently only two: creating a post with a Mastodon embed, and creating a link post.

Once the template is selected, the steps are the same: specify the post link, optional via link, and link emoji.

These get pre-filled as before. It's just a matter of adding any commentary after that. Clicking 'Save' will only save the post locally: one needs to click 'Publish' in the posts list to send it to the Micropub service.

This can’t really be described as CMS. It’s more of a place to create posts from a template which can then be published via a Micropub endpoint. But it does the job I need, which is creating link posts with the links pre-filled.

Or not. I mean, I’m not expecting you to notice. You’ve got lives of your own, after all. ↩︎

Thu, 05 Jun 2025 21:35:04 +1000

Had a reason to write a journal entry today, which meant I had a reason to work on the journaling app. Biggest change was moving the entry list to a separate page and supersizing the text-area to allow for larger entries. Good thing too: today’s was going to need all the space it could get.

Before

After

Wed, 04 Jun 2025 22:26:41 +1000

Working on that Godot game again, mainly coming up with mechanics for a new level 2. This is what I’ve got so far: a mine tethered to a balloon. Their idle state is just bobbing up and down, but I am planning a variant which will drop their payload and fly away when the player is nearby.

Devlog: Dynamo-Browse Now Scanning For UCL Extensions

Sun, 25 May 2025 13:55:49 +1000

Almost finished integrating UCL with Dynamo-Browse. As stated earlier, the goal is to use UCL for both the command and scripting language. Today, the goal was closer, with a development version of Dynamo-Browse no longer using old extension scripting language, and now scanning and evaluating UCL-based extensions on launch time.

This required some changes in how UCL evaluated script files. There’s a new ucl.WithSubEnv() option that can be passed into inst.Eval() to instruct the evaluator to fork the environment prior to evaluating the script:

f, _ := os.ReadFile("my-file.ucl")

inst := ucl.New()
inst.Eval(ctx, bytes.NewReader(r), ucl.WithSubEnv())

Under the hood, this forks the top-level environment frame and marks it as the frame to store all procs and global variables for the evaluated file. This effectively makes it a separate namespace, separating it from all other scripts and preventing cross-contamination when scripts share the same symbol.

I’m hoping that this will eventually be the basis for UCL modules, but for now, this is use within Dynamo-Browse to isolate each UCL extension. The downside of this is that it’s no longer possible to add new commands simply using the proc keyword. I think this may end up being a good thing though, as extensions will probably not be interested in exposing internal functions defined just for their own use.

To compensate for this, there is now a new command — ui:command — which allows the script author to expose a function as a command. In fact, what it does is save the command in the root environment frame, effectively making it work the old way proc did, and acting like a poor-mans extension export.

Now with all the extension loading in place, it’s time for the inaugural UCL extension. Here it is:

# Gets the partition key of the currently selected row, and adds it
# to the pasteboard. Bound to the 'U' key.

ui:command copy-pk {
    # Get the partition key value from the currently selected item
    $pkKey = @table.Keys.PK
    $pkValue = @item.($pkKey)

    # Place it in the pasteboard
    pb:put $pkValue

    echo "$pkKey copied to clipboard"
}

ui:bind "U" { copy-pk }

All the features are not quite implemented yet, but I think it’s good enough to start using it as part of my normal workflow. That’s usually the best way to find the rough edges.

Sun, 18 May 2025 14:07:39 +1000

All the recent changes to UCL is in service of unifying the scripting within Dynamo Browse. Right now there are two scripting languages: one for the commands entered after pressing :, and one for extensions. I want to replace both of them with UCL, which will power both interactive commands, and extensions.

Most of the commands used within the in-app REPL loop has been implemented in UCL. I’m now in the process of building out the UCL extension support, start with functions for working with result sets, and pseudo-variables for modifying elements of the UI.

Here’s a demo of what I’ve got so far. This shows the user’s ability to control the current result-set, and the selected item programatically. Even after these early changes, I’m already seeing much better support for doing such things than what was there before.

Devlog: UCL — Assignment

Sun, 18 May 2025 09:35:57 +1000

Toying with changing how assignment works in UCL. Up to now, assigning a variable to a value involved calling the set command, which took a variable name, and the value to assign:

set a "hello"
$a
--> "hello"

This has been fine, but I’m now running into few limitations with this approach. The first is that it doesn’t support setting subscript values within lists or values. Because set is just a regular command, the parser will evaluate any dot or sub-pipe expressions prior to invoking set. One way around this is to quoted the variable to assign as a string:

set list [1 2 3]
set "$list.(1)" 4

But this looks ugly, and will involve another pass of the parser every time set is called.

Another approach is converting set to a macro, which receives the arguments as the parse tree. This gives set more control over how to interpret the arguments. But it doesn’t resolve the second issue, which is an inconsistency introduced with a new feature I’m planning.

I want to add the notion of pseudo variables, which act like regular variables except that getting or setting them will invoke some hander logic from the embedding system, rather than set a value in memory. Think of them similar to how document and window work in a browser’s JavaScript runtime. My plan for them is to embed this Dynamo Browse to allow the user to access or modify the result-set or selected item, which will result in the UI changing. I’m sure I’ll have more to say about that in the future.

To distinguish them from normal variables, they’ll have an at-sign instead of a dollar sign:

@value
--> something

But as far as the user is concerned, they should act just like normal variables. And just like normal variables, they should be modifiable using the set command. I don’t want the variable and pseudo-variable to share the same namespace (since they look different, there’s no reason for them to share one), so I needed a way to distinguish between setting a pseudo-variable from a regular variable.

This first means they needed to be quoted but I hacked a quick version of set on this feature branch to support setting them:

set "@value" "new value"

But compare this with the first example. Notice that when we were setting a, we didn’t need to include the dollar sign. Here we have a case where one sort of variable requires the prefix symbol, and the other doesn’t. This form of inconsistency was not appealing to me.

Since the various modes of assignments has outgrown the ability of one command to do it all, I’ve decided to remove set and add basic assignments to the language. These look like typical assignments you see in most other languages:

$a = "hello"
$a
--> "hello"

But they’ll be able to fix the issues from using set, such as assign the value of subsets and showing consistent representation of what you’re actually trying to modify:

@value = "new value"

$list = [1 2 3]
$list.(1) = 4
$list
--> [1 4 3]

It also leads to some improvements of the common case. One small issue with using set is that it always required one to wrap the result of a sub-expression in parenthesis.

set h (strs:to-upper "Hello")

But since this is a grammar change, these parenthesis are no longer necessary.

$h = strs:to-upper "Hello"

$h = "Hello" | strs:to-upper

There may also be room for different forms of assignment, such as ensuring the value you’re trying to set is not nil. There was a variant of set, called set!, which threw an error when attempting to assign a variable to nil. This could be expressed as a different assignment form:

$h = "this is fine"

$h =! ()
--> error: trying to set $h to a nil value

I will acknowledge that doing this will mean loosing out on some theoretical benefits that came from using set. For one thing, it will no longer be possible to indirectly set a value. You can’t, for example, do this:

set b "a"
set $b "hello"
$a
--> "hello"

I’ve never really needed to do this, but I could see this being potentially useful. One way to support this might be to do something similar to what Lua does, and expose the environment as a hash. Pseudo-variables could be useful here:

$b = "a"
@_G.($b) = "hello"
$a
--> "hello"

I’ll hold off from adding something like this until I absolutely need to.

So that’s the current idea. I spent around an hour on this so far, just trying it out and seeing how it feels, and I think it’s got promise. I’ll keep it on the feature branch for now, but I suspect this will eventually become the new way to do assignment in UCL going forward.

Devlog: Blogging Tools — Finished Podcast Clips

Thu, 15 May 2025 23:18:00 +1000

Well, it’s done. I’ve finally finished adding the podcast clip to Blogging Tools. And I won’t lie to you, it took longer than expected, even after enabling some of the AI features my IDE came with. Along with the complexity that came from implementing this feature, that touched on most of the key subsystems of Blogging Tools, the biggest complexity came from designing how the clip creation flow should work. Blogging Tools is at a disadvantage over clipping features in podcast players in that it:

Doesn’t know what feeds you’ve subscribed to,
Doesn’t know what episode you’re listening to, and
Doesn’t know where in the episode you are.

Blogging Tools needs to know this stuff for creating a clip, so there was no alternative to having the user input this when they’re creating the clip. I tried to streamline this in a few ways:

Feeds had to be predefined: While it’s possible to create a clip from an arbitrary feed, it’s a bit involved, and the path of least resistence is to set up the feeds you want to clip ahead of time. This works for me as I only have a handful of feeds I tend to make clips from.
Prioritise recent episodes: The clips I tend to make come from podcasts that touch on current events, so any episode listings should prioritise the more recent ones. The episode list is in the same order as the feed, which is not strictly the same, but fortunately the shows I subscribe to list episodes in reverse chronological order.
Easy course and fine positioning of clips: This means going straight to a particular point in the episode by entering the timestamp. This is mainly to keep the implementation simple, but I’ve always found trying to position the clip range on a visual representation of a waveform frustrating. It was always such a pain trying to make fine adjustments to where the clip should end. So I just made this simple and allow you to advance the start time and duration by single second increments by tapping a button.

Rather than describe the whole flow in length, or prepare a set of screenshots, I’ve decided to record a video of how this all fits together.

The rest was pretty straightforward: the videos are made using ffmpeg and publishing it on Micro.blog involved the Micropub API. There were some small frills added to the UI using both HTMX and Stimulus.JS so that job status updates could be pushed via web-sockets. They weren’t necessary, as it’s just me using this, but this project is becoming a bit of a testbed for stretching my skills a little, so I think small frills like this helped a bit.

I haven’t made a clip for this yet or tested out how this will feel on a phone, but I’m guessing both will come in time. I also learnt some interesting tidbits, such that the source audio of an <audio> tag requires a HTTP response that supports range requests. Seeking won’t work otherwise: trying to change the time position will just seek the audio back to the start.

Anyway, good to see this in prod and moving onto something else. I’ve getting excited thinking about the next thing I want to work on. No spoilers now, but it features both Dynamo Browse and UCL.

Finally, I just want to make the point that this would not be possible without the open RSS podcasting ecosystem. If I was listening to podcasts in YouTube, forget it: I wouldn’t have been able to build something like this. I know for myself that I’ll continue to listen to RSS podcasts for as long as podcasters continue to publish them. Long may it be so.

Devlog: Blogging Tools — Ideas For Stills For A Podcast Clips Feature

Wed, 07 May 2025 23:18:48 +1000

I recently discovered that Pocketcasts for Android have changed their clip feature. It still exists, but instead of producing a video which you could share on the socials, it produces a link to play the clip from the Pocketcasts web player. Understandable to some degree: it always took a little bit of time to make these videos. But hardly a suitable solution for sharing clips of private podcasts: one could just listen to the entire episode from the site. Not to mention relying on a dependent service for as long as those links (or the original podcast) is around.

So… um, yeah, I’m wondering if I could building something for myself that could replicate this.

I’m thinking of another module for Blogging Tools. I was already using this tool to crop the clip videos that came from Pocketcasts so it was already in my workflow. It also has ffmpeg bundled in the deployable artefact, meaning that I could use to produce video. Nothing fancy: I’m thinking of a still showing the show title, episode title, and artwork, with the audio track playing. I pretty confident that ffmpeg can handle such tasks.

I decided to start with the fun part: making the stills. I started with using Draw2D to provide a very simple frame where I could place the artwork and render the text. I just started with primary colours so I could get the layout looking good:

I’m using Roboto Semi-bold for the title font, and Oswald Regular for the date. I do like the look of Oswald, the narrower style contrasts nicely with the neutral Roboto. Draw2D provides methods for measuring text sizes, which I’m using to power the text wrapping layout algorithm (it’s pretty dumb. It basically adds words to a line until it can’t fit the available space)

The layout I got nailed down yesterday evening. This evening I focused on colour.

I want the frame to be interesting and close to the prominent colours that come from the artwork. I found this library which returns the dominant colours of an image using K-means clustering. I’ll be honest: I haven’t looked at how this actually works. But I tried the library out with some random artwork from Lorem Picsum, and I was quite happy with the colours it was returning. After adding this library¹ to calculate the contract for the text colour, plus a slight shadow, and the stills started looking pretty good:

I then tried some real podcast artwork, starting with ATP. And that’s where things started going off the rails a little:

The library returns the colours in order of frequency, and I was using the first colour as the border and the second as the card background. But I’m guessing since the ATP logo has so few actual colour, the K-means algorithm was finding those of equal prominence and returning them in a random order. Since the first and second are of equal prominence, the results were a little garish and completely random.

To reduce the effects of this, I finished the evening by trying a variation where the card background was simply a shade of the border. That still produced random results, but at least the colour choices were a little more harmonious:

I’m not sure what I want to do here. I’ll need to explore the library a little, just to see whether it’s possible to reduce the amount of randomness. Might be that I go with the shaded approach and just keep it random: having some variety could make things interesting.

Of course, I’m still doing the easy and fun part. How the UI for making the clip will look is going to be a challenge. More on that in the future if I decide to keep working on this. And if not, at least I’ve got these nice looking stills.

The annoying thing about this library is that it doesn’t use Go’s standard Color type, nor does it describe the limits of each component. So for anyone using this library: the range for R, G, and B go from 0 to 255, and A goes from 0 to 1. ↩︎

Devlog: Dialogues

Sun, 04 May 2025 10:45:36 +1000

Added something fun, and potentially useless, to this blog: styling for dialogues:

L: Hello?

🦆: Oh, hello. What's up?

L: Just showing how dialogues look on this site.

🦆: Oh, wow. Super nice.

I don’t rubber-duck often, but I have a few times in the past, and I’ve published those as blog posts on this site. And it’s been fun. Not only has it been helpful to work through the problem this way, there’s an element of creativity involved in the whole process: imagining two characters working through a problem like this. The exchange is purely fictional, but the benefits are very real.

And part of me is wondering whether to do more of this style of writing. Not only when I’m facing a programming dilemma, but when I’m feeling… I don’t know, “whimsical.” I’ve seen a few other sites having this style of dialogues, both real and imagined; and I’ve recently read posts from others wishing more sites had them.

So all these justifications — along with boredom, a desire to work on something novel, and ship something small — accumulated into this silly little change.

The dialogues themselves are all HTML. If you were to view source, you’ll see that they’re essentially styled blockquotes. It was important to me to make sure that came through correctly on RSS feeds, with all the styling stripped away. In a browser, they’re meant to resemble an imaginary chat experience. I’m hoping this is versatile enough: the one concern I have is that this style of interaction seems to encourage short exchanges between the participants, so I may adjust the styling a little in the future. But I’ll start with this and see how it goes. If you’re curious as to how the CSS looks, you can find it here.

Of course, most people would stop here; but because I’m me, and will take any opportunity to spend several hours on something if I think it would save me a few minutes, I added a module to Blogging Tools to render dialogs from a mini-language. This uses a small parser written in participal that will produce the HTML for a dialog that I can copy and paste directly into the Micro.blog editor. There’s also a tiny bit of AI that would automatically generate a name for the interaction if I were to leave it blank. For that, I’m using “Meta Llama 3.1.8B Instruct Turbo” made available via together.ai (the name is not published, it’s just used for listing the dialogue).

The new Dialogues module.

The dialogue editor. This is essentially a plain-text post editor.

An example of the mini-language used to styling the dialogue. The real avatar properties are slightly different to what appears here.

Click Render to generate the HTML. Copy and paste at will.

🦆: Great work.

🦆: So are you going to use this often?

L: Well, probably not. I think it's one of those features where most of the fun is building it. Then once you have it, you're left wondering what to do with it.

🦆: Oh, okay.

L: Yeah, it's not the first time I did this. I did it before with marginalias. Built the Hugo short-code, added it to a few posts, then never looked at it again.

🦆: Oh, yeah. I vaguely remember you doing that before.

L: Yeah, I may actually revisit that some time. Might be that doing something similar to what I've done here with dialogues, i.e. adding a module in this Blogging Tool, will make it that I'm more likely to use this.

🦆: Well, let's see. The feature is here now, so there's no reason why you wouldn't use this in the future. Maybe next time in one of our rubber-ducking sessions.

L: Yeah, we'll see.

L: Anyway, I'll catch you then.

🦆: Yeah, see ya later.

So anyway, that’s what I’ve been doing the last week.

Devlog: Godot Game Update

Sun, 27 Apr 2025 11:29:47 +1000

A brief status update on that Godot game. I think we’re pretty close to a finished 4-1 level. The underground section has been built, and the level has been decorated. I’ve also added a couple of secrets, which needed a few new mechanics — like doorways, which are used to transport the player around the level — plus some refinement to existing ones. I am a little concerned about the amount of waiting involved near the end of the first half, where the player will need to make their way across a large gap by jumping on the slow cycling “layer 2” tile layer. I’ll see what feedback I get from play-testers about this.

I’ve also filled in the backdrop of all the levels to date. The grey default background colour has been replaced with a sky made of colour bands that came with the asset set I’m using. It’s basic, maybe too basic, but as long as it doesn’t clash with the interactive elements, and in the interest of “merely shipping something,” I think it’ll do for now.

Finally, I made some changes from feedback I got from play-testers. Movement can now be made using WASD, along with the inverted-T arrow keys. I’ve also bound jump to the Enter key for those that want to use WASD. I made some changes in how secrets are telegraphed with the goal of making them more consistent. This is always a bit of a balancing act, as I don’t want to make them too obvious: they’re secrets after all. I added a tiny bit of “slip” to the player’s movement, so they don’t stop dead whenever the key is released. I didn’t want to go too far here: adding too much inertia introduced the risk of the player just falling off a platform after landing from a jump, which I don’t think is a great experience. And I fixed a window resizing bug (Godot enables window resizing by default, for anyone else who needs to know this).

I think the next thing to do is to organise another version for play-testers, just to gauge what they think of level 4-1. Then it’s either on to level 4-2, or even introduce a new level between 1-1 and 1-2 to reduce the ramp up in difficulty between the two levels. Fortunately I’ve got some ideas on what I want to do for both of theses, so they’ll be easy starters. I’ve still got nothing for level 2-1, but I have a few ideas on what I want to do for 2-2, so maybe I’ll start that too.

Fri, 18 Apr 2025 14:30:18 +1000

Made some more progress on that Godot game. I haven’t gotten any further with the first level of world 2, so I’ve been spending much of my time making mechanics. One of them was the slow moving “level 2” mechanic that I stole wholesale from Super Mario World. That mechanic, despite it being frustrating to speed-runners, was always slightly interesting to me. To have areas of a level become accessible or hazardous just due to a layer of it oscillate up and down, it promised to make for some interesting timing challenges. At least in theory.

Portion of the new level showing all three new mechanics.

I decided to put that theory to the test, and start work on one the later levels. And despite being a little skeptical about whether the mechanic could carry through a level on it’s own, I came up with one that I’m reasonably happy with. The mechanic is introduce slowly, and in a rather non-threatening way, proving the player the means to get to higher ground. This leads into the second half, which will be a long underground section which will ramp up the difficulty by introducing the risk of getting crushed or missing platforms.

To compliment this is a new enemy that rushes the player. The player cannot do anything to defeat this enemy: combat is not really a thing in this game. All they could do is evade it before the enemy gives up. I am reusing the same “green slime” sprite for this but I’m hoping that the differing animations provide some hints of how this enemy’s behaviour differs from that of the simpler one.

Finally, it was time to consider checkpoints. While the first few levels were too short to justify adding them in, this one is just that bit too long without one. And given the difficulty ramp-up in the second half, having the player go through the slower first half every time they died would probably lead to frustration. So checkpoints are now a thing. They’re not free — costing 5 coins to activate — and they are sometimes mandatory, blocking the player from progressing until they pay the toll. But I think their presence helps with eliminating the areas of the level that would just be boring to play through again and again.

So yeah, I’m quite happy with this level. And I’m also happy in realising that I’m not bound to building this game in the same progression that the player will experience it. It’s better sometimes to just work on the areas that you’re ready to. I mean, it’s sounds obvious to say that now. Not sure why it took me this long to actually do so.

Tue, 08 Apr 2025 22:35:06 +1000

Spent some time over the last few days working on that Godot game, mainly building new mechanics. This evening I started working on an interceptor, something that would jump out of the quicksand in order to disrupt the player’s jump. Here’s an example of how they look in the test bed:

And yeah, they’re pretty much a carbon-copy of the Podoboos from Mario. But I think there’s a reason they’re still making an appearance in games, years after their debut in Super Mario Brothers. They’re quite a versatile enemy, making jumping challenges a bit more interesting than just seeing whether the player the clear a gap. Plus they’re reasonably easy to make.

Another mechanic taken from Mario was a switch that revealed coins and tiles for a limited time. Hit it once and the child nodes of this “timed_limited_visible” scene are displayed and activated for 10 seconds, before they disappear again:

Much like the blue P switch this mechanic takes inspiration from, the switch can only be activated once. So it may be only useful for bonuses and areas the player can afford to miss.

I had to do some special handling for nested TileMap nodes, since the player could still collide with them even when they’re hidden. How I solved this was nothing too spectacular: basically I just walk the child tree looking for TileMap instances, and when encountering one, just enabling or disabling the first layer:

func _show_and_activate_children():
    visible = true
    process_mode = Node.PROCESS_MODE_INHERIT
    for tm in find_children("*", "TileMap", false):
        tm.set_layer_enabled(0, true)

func _hide_and_deactivate_children():
    visible = false
    process_mode = Node.PROCESS_MODE_DISABLED
    for tm in find_children("*", "TileMap", false):
        tm.set_layer_enabled(0, false)

Building these elements was fun, but the main problem is that I’m struggling to come up with a centrepiece mechanic for level 2-1, something that defines the level in some way. I have an idea for level 2-2 — this world is set in a desert so I’m hoping to introduce a thirst mechanic — but level 2-1 I’m hoping to keep relatively plain so as to avoid overwhelming the player with too many new things. The fear is to avoid making it little more than what the player encountered in world 1: a series of jumping puzzles over pits. Sure, that’s pretty much the entire game in a way, but some variety would be nice.

I’m hoping one of these mechanics could help here. I guess I’ll find one once I’ve start seriously building the level.

Tue, 01 Apr 2025 20:52:50 +1000

Started working on world 2, and one of the main mechanics of this world: quicksand. It won’t kill the player directly, but it will make it difficult for them to manoeuvre, and getting too low could cause death. Might be one of the more annoying mechanics in the game, but that’s kind of the point.

Tue, 01 Apr 2025 20:17:48 +1000

The results of my first play-test are in. And overall, they were pretty positive: movement was good, hit-boxes were fair, and it was described as “quite fun,” which was better than I was hoping for.

One thing I’ll need to look out for is telegraphing secrets. The number of secrets is indicated at the end of the level, and based on the play-tester’s feedback, they seemed to have spent a lot of time running against walls trying to find them. There is one secret in the level 1-1 that I thought was telegraphed well, and I can confirm that the player found them all. But I will concede the others required the player to make a leap of faith, and fall into an area that the player will usually want to avoid, which is pretty unfair. So I’ll need to fix that.

Sat, 29 Mar 2025 10:36:52 +1000

A bit more on the Godot game this morning, this time working on background tiles artwork. Made some grey masonry tiles for the end castle sequences. Also tried some background rocks for underground areas. I’m pretty rubbish at anything organic, but they didn’t turn out too bad.

Right side has the background tiles surrounded with their complementary foreground tiles on the left.

Thu, 27 Mar 2025 21:12:26 +1000

Added a few final things to my Godot game, such as a really boring title and end-title screen, before preparing a release for play testers (or play tester, I’ve got exactly one lined up). I think we’re ready.

Mon, 24 Mar 2025 21:39:10 +1000

A bit more on Godot this evening, mainly working on pausing the game, and the end-of-level sequence. Have got something pretty close to what I was looking for: a very Mario-esc sequence where the player enters a castle, it start auto-walking the character, and the level stats show up and “spin” for a bit. Not too bad, although I may need to adjust the timing and camera a little to keep the stats from being unreadable.

Mon, 17 Mar 2025 21:14:01 +1000

A bit more Godot work this evening. I wanted to add a foreground layer of tiles that obscured in the player. This is for making false walls to hide secrets around the level. It took me a while to work out how to turn off collision of this foreground layer: there wasn’t really any way to do so within the designer.

Fortunately, this Github comment showing how to do so using a script worked like a charm:

extends TileMap

const foreground_layer = 1

func _use_tile_data_runtime_update(layer: int, coords: Vector2i) -> bool:
	return layer == foreground_layer
	
func _tile_data_runtime_update(layer: int, coords: Vector2i, tile_data: TileData) -> void:
	tile_data.set_collision_polygons_count(0, 0)

Only limitation seems to be that it will disable collision for the whole layer, but that’s perfectly fine with me.

Mon, 17 Mar 2025 06:42:31 +1000

Spent more time on my Godot platformer yesterday, mainly rebuilding the first level from scratch. The previous version was rush and was just not fun (it didn’t look good either, but I didn’t dress it up yet). This new one is much nicer, and allowed me to use a few new mechanics I’ve built.

I still need to build out the level ending sequence. I’ve got less than the basics at the moment: a drawbridge descends and that’s pretty much it. I need to add the conditions for when the bridge descends (specifically, a minimum number of coins), stopping the player movement for a brief scripted sequence, then transition to the next level. I think I know how I want to do this, it’s just a matter of implementing it. Once level 1 is dressed up and working, I think that’ll be the next thing I do.

I’m enjoying working on this project so far, although part of me is a little afraid that I’m spending time on something that just isn’t good, like there’s some sunk cost with the time I’m spending on this that’s better put to use on something else. Of course, when I give in to these feelings, I usually just spend the time watching TV. So which activity is the real waste of time? Is producing something that may be crap (or worse, just boring) better than not producing anything at all?

Anyway, not should how this became a feeling post. This game might be rubbish, but I still enjoy working on it: Godot is actually quite fun to use. Surely that is reason enough to continue.

Mon, 10 Mar 2025 20:29:31 +1000

Was looking at how I could add hazards to my Godot project, such as spikes. My first idea was to find a way to detect collisions with tiles in a TileMap in Godot. But there was no real obvious way to do so, suggesting to me that this was not the way I should be going about this. Many suggested simply using an Area2D node to detect when a play touches a hazard.

I was hesitant to copy and paste the scene node I had which handled the collision signal and kill the player — the so-called “kill zone” scene —but today I learnt that it’s possible to add multiple CollisionShape2D nodes to an Area2D node. This meant I needed only a single “kill zone” scene node, and just draw out the kill zones over the spikes as children. The TileMap simply provides the graphics.

This discovery may seem a little trivial, but I’d prefer to duplicate as few nodes as a can, just so I’ve got less to touch when I want to change something.

Mon, 10 Mar 2025 08:49:38 +1000

Tried opening my Godot project this morning and was greeted with the following error:

scene/resources/resource_format_text.cpp:284 - res://scenes/falling_platform.tscn:14 - Parse Error: 
Failed loading resource: res://scenes/falling_platform.tscn. Make sure resources have been imported by opening the project in the editor at least once.
Failed to instantiate scene state of "res://scenes/falling_platform.tscn", node count is 0. Make sure the PackedScene resource is valid.
Failed to load scene dependency: "res://scenes/falling_platform.tscn". Make sure the required scene is valid.

Traced it back to the technique I was using to respawn the falling platform. Looks like Godot didn’t like the two preloads I included:

# file: scripts/falling_platform.gd

@onready var FallingPlatform = preload("res://scenes/falling_platform.tscn")
@onready var FallingPlatformScript = preload("res://scripts/falling_platform.gd")

This resulted in a parse error and Godot thinking the level scene was corrupted. In retrospect, this kinda makes sense. What I doing was technically a circular dependency, where the scene and script was trying to preload itself. I was hoping Godot was smart enough to recognise this, but I guess not.

So I had to change the respawn code. I modified it to make use the duplicate method. Here’s the revised version:

func respawn():	
    var dup = self.duplicate()
    dup.global_position = init_global_position	

    # Duplicate also copies the velocity so zero it out here
    dup.velocity = Vector2(0, 0)
    get_tree().current_scene.add_child(dup)

After some limited testing, this seems to work. One good thing about this approach is that it looks like duplicate copies the script, so I no longer need to do anything special here.

So I guess the lesson here is don’t try to preload the script within itself.

Adventures In Godot: Respawning A Falling Platform

Sat, 08 Mar 2025 11:35:07 +1000

My taste of going through a Godot tutorial last week has got me wanting more, so I’ve set about building a game with it. Thanks to my limited art skills, I’m using the same asset pack that was used in the video, although I am planning to add a bit of my own here and there.

But it’s the new mechanics I enjoy working on, such as adding falling platforms. If you’ve played any platformer, you know what these look like: platforms that are suspended in air, until the player lands on them, at which point gravity takes a hold and they start falling, usually into a pit killing the player in the process:

The platforms are built as a CharacterBody2D with an Area2D that will detect when a player enters the collision shape. When they do, a script will run which will have the platform “slipping” for a second, before the gravity is turned on and the platform falls under it’s own weight. The whole thing is bundled as a reusable scene which I could drag into the level I’m working on.

I got the basics of this working reasonably quickly, yet today, I had a devil of a time going beyond that. The issue was that I wanted the platform to respawn after it fell off the map, so that the player wouldn’t get soft-locked at at an area where the platform was needed to escape. After a false-start trying to reposition the platform at it’s starting point after it fell, I figured it was just easier to respawn the platform when the old one was removed. To do this I had to solve two problems:

How do I get the platform starting point?
How can I actually respawn the platform?

Getting The Starting Point

A Node2D object has the property global_position, which returns the position of the object based on the world coordinates. However, it seems like this position is not correct when the _init function of the attached script is called. I suspect this is because this function is called before the platform is added to the scene tree, when the final world coordinates are known.

Fortunately, there exists the _ready notification, which is invoked when the node is added to the scene tree. After some experimentation, I managed to confirm that global_position properly was correct. So tracking the starting point is a simple as storing that value in a variable:

var init_global_position = null

func _ready():
	init_global_position = global_position

Another option is to use the _enter_tree() notification. From the documentation, it looks like either would probably work here, with the only difference being the order in which this notification is invoked on parents and children (_enter_tree is called by the parent first, whereas _ready is called by the children first).

Respawning The Platform

The next trick was finding out how to respawn the platform. The usual technique for doing so, based on the results of my web searching, is to load the platform scene, instantiate a new instance of it, and added it to the scene tree.

@onready var FallingPlatform = preload("res://scenes/falling_platform.tscn")

func respawn():	
    var dup = FallingPlatform.instantiate()
    add_child(dup)

Many of the examples I’ve seen online added the new scene node as a child of the current node. This wouldn’t work for me as I wanted to free the current node at the same time, and doing so would free the newly instantiated child. The fix for this was easy enough: I just added the new node as a child of the current scene.

@onready var FallingPlatform = preload("res://scenes/falling_platform.tscn")


func respawn():	
    var dup = FallingPlatform.instantiate()
    get_tree().current_scene.add_child(dup)
    queue_free()

I still had to reposition the new node to the spawn point. Fortunately the global_position property is also settable, so it was simply a matter of setting that property before adding it to the tree (this is so that it’s correct when the newly instantiated node receives the _ready notification).

@onready var FallingPlatform = preload("res://scenes/falling_platform.tscn")

func respawn():	
    var dup = FallingPlatform.instantiate()
    dup.global_position = init_global_position
    get_tree().current_scene.add_child(dup)
    queue_free()

This spawned the platform at the desired positioned, but there was a huge problem: when the player jumped on the newly spawn platform, it wouldn’t fall. The Area2D connection was not invoking the script to turn on the gravity:

It took me a while to figured out what was going on, but I came to the conclusion that the packed scene was loading properly, but without the script attached. Turns out a Script is a resource separate from the scene, and can be loaded and attached to an object via the set_script method:

@onready var FallingPlatform = preload("res://scenes/falling_platform.tscn")
@onready var FallingPlatformScript = preload("res://scripts/falling_platform.gd")

func respawn():	
    var dup = FallingPlatform.instantiate()
    dup.set_script(FallingPlatformScript)

    dup.global_position = init_global_position
    get_tree().current_scene.add_child(dup)
    queue_free()

Finally, after figuring all this out, I was able to spawn a new falling platform, have it positioned at the starting position of the old platform, and react to the player standing on it.

The time it took to work this out is actually a little surprising. I was expecting others to run into the same problem I was facing, where they were trying to instantiate a scene only to have the scripts not do anything. Yet it took me 45 minutes of web searching through Stack Overflow and forum posts that didn’t solve my problem. It was only after a bit of experimentation and print-debugging on my own that I realised that I actually had to attached the script after instantiating the node.

To be fair, I will attribute some of this to not understanding the problem at first: I actually thought the Area2D wasn’t actually being instantiated at all. Yet not one of the Stack Overflow answers or forum post floated the possibility that the script wasn’t being loaded alongside the scene. This does suggest to me that my approach may not be optimal. There does exist a “Local to Scene” switch in the script inspector that could help, although turning it on doesn’t seem to do much. But surely there must be some way to instantiate the script alongside the scene.

Anyway, that’s for later. For now, I’m happy that I’ve got something that works.

Mon, 24 Feb 2025 21:02:53 +1000

Prototyped a game I had in mind, sort of a 2D Sokoban-like thing, where you control a robot with a retractable pushing arm that is to push gems to a “receiver” tile. Not entirely sure if it’s fun enough to actually build.

Used PixiJS to build it. Not a bad framework.

Making A Small Two-Letter Country Code Lookup Page

Thu, 20 Feb 2025 07:57:14 +1000

Made a small thing yesterday.

I’ve been seeing links to the uchū color pallette and Atkinson Hyperlegible font on Mastodon this past week and I wanted to give them a try on something. With little else to work on, I thought I’d spend yesterday evening building a small thing I’ve been wishing I had at work for a while.

My tasks at work has me looking up the two-letter country codes a lot recently. My goto is the ISO-3166-1 alpha 2 page on Wikipedia, but getting there involves a few clicks, a web-search, and a “find on page.” I wanted an easier way to get to this list, and a much easier way to filter it. I also wanted something that could work as a Vivaldi side panel, so that I can call it up while I’m looking at something else.

What I ended up building is this, which I’ve called 2LCC, which stands for Two Letter Country Codes.

Start entering a country name or country code to filter the table.

Because two-letter codes will appear as letters within country names, any exact match in either the code or the country name will highlight the row in yellow.

How it looks in Vivaldi's sidebar.

It was built using Hugo, which some might think may be a bit overkill, but Hugo does has some nice utilites for reading JSON data and running it through a Go template to produce static HTML. And that’s all this is: a static HTML page with a big table of country codes with a smattering of CSS and JavaScript. The JavaScript is just vanilla, and is simply used to power the filter, which basically walks the table, hiding rows which don’t contain the filter string. I was originally going to have the filter logic test the inner text elements of the table cells, but I decided to have the template spit out the country code as data- attributes instead. It’s easier to fetch in JavaScript using the dataset property list and since the values are spat out by the template in lowercase, it makes case insensitive slightly easier.

We’ll see how often I end up using this: there’ve been instances when I’ve over-index needing something for my job, only to build it and find myself using it less than I imagined. It probably wasn’t a great project to choose for trying out uchū colour scheme, seeing that I only ended up using two colours. But I’m happy with how the two colours I used turned out, and the the font looks great. And I do like that I took on more of the layout styling myself, rather than simply defer to simple.css.

Here’s a link to it should this be useful to you: https://2lcc.lmika.app/.

Sat, 15 Feb 2025 10:10:08 +1000

Dusted off Podcast Favourites (last commit 25 April 2022) and fixed a longstanding issue of thumbnails being lost when they’re changed in the feed. Editing the feed properties will now force a refresh of the thumbnail URLs. Didn’t need to change anything else, which was a nice change.

UCL: Some Updates

Sat, 08 Feb 2025 09:32:00 +1000

Made a few minor changes to UCL. Well, actually, I made one large change. I’ve renamed the foreach builtin to for.

I was originally planning to have a for loop that worked much like other languages: you have a variable, a start value, and an end value, and you’d just iterate over the loop until you reach the end. I don’t know how this would’ve looked, but I imagined something like this:

for x 0 10 {
    echo $x
}
# numbers 0..9 would be printed.

But this became redundant after adding the seq builtin:

foreach (seq 10) { |x|
    echo $x
}

This was in addition to all the other useful things you could do with the foreach loop¹, such as loop over lists and hashes, and consume values from iterators. It’s already a pretty versatile loop. So I elected to go the Python way and just made it so that the for loop is the loop to use to iterate over collections.

This left an opening for a loop that dealt with guards, so I also added the while loop. Again, much like most languages, this loop would iterate over a block until the guard becomes false:

set x 0
while (lt $x 5) {
    echo $x
    set x (add $x 1)
}
echo "done"

Unlike the for loop, this is unusable in a pipeline (well, unless it’s the first component). I was considering having the loop return the result of the guard when it terminates, but I realised that would be either false, nil, or anything else that was “falsy.” So I just have the loop return nil. That said, you can break from this loop, and if the break call had a value, that would be used as the result of the loop:

set x 0
while (lt $x 5) {
    set x (add $x 1)
    if (ge $x 3) {
        break "Ahh"
    }
} | echo " was the break"

The guard is optional, and if left out, the while loop will iterate for ever.

The Set! Builtin

Many of these changes come from using of UCL for my job, and one thing I found myself doing recently is writing a bunch of migration scripts. This needed to get data from a database, which may or may not be present. If it’s not, I want the script to fail immediately so I can check my assumptions. This usually results in constructs like the following:

set planID (ls-plans | first { |p| eq $p "Plan Name" } | index ID)
if (not $planID) {
    error "cannot find plan"
}

And yeah, adding the if block is fine — I do it all the time when writing Go — but it would be nice to assert this when you’re trying to set the variable, for no reason other than the fact that you’re thinking about nullability while writing the expression to fetch the data.

So one other change I made was add the set! builtin. This will basically set the variable only if the expression is not nil. Otherwise, it will raise an error.

set! planID (ls-plans | first { |p| eq $p "Missing Plan" } | index ID)
# refusing to set! `planID` to nil value

This does mean that ! and ? are now valid characters to appear in identifiers, just like Ruby. I haven’t decided whether I want to start following the Ruby convention of question marks indicating a predicate or bangs indicating a mutation. Not sure that’s going to work out now, given that the bang is being used here to assert non-nullability. In either case, could be useful in the future.

And the seq builtin ↩︎

Idea for UCL: Methods

Sun, 02 Feb 2025 14:46:00 +1000

I’m toying with the idea of adding methods to UCL. This will be similar to the methods that exist in Lua, in that they’re essentially functions that pass in the receiver as the first argument, although methods would only be definable by the native layer for the first version.

Much like Lua though, methods would be invokable using the : “pair” operator.

strs:to-upper "Hello"
--> HELLO

The idea is to make some of these methods on the types themselves, allowing their use on literals and the result of pipelines, as well as variables:

set h "Hello"
$hello:to-upper
--> HELLO

"Hello":to-upper
--> HELLO

(cat "Hello " "world"):to-upper
--> HELLO WORLD

The use of methods would be purely conveience. One could conceive of a type, like a CSV table, where there’s a need to perform a series of operations over it.

The potential downside of using : is that it may make defining dictionaries more ambiguous. When the parser sees something that could either be a list or a dictionary, it does a scan to search for any pairs that may exist. If so, it treats the literal as a dictionary. But would this work with using : as the method separator?

["Hello":to-upper]
--> Is this a dictionary {"Hello":"to-upper"}, or the list ["HELLO"]?

So that would require something. Might be that I need to require method invocations to occur within parenthesis for list literals.

Ambiguities like this aside, I’m planning to keep it simple for now. Methods will not be user definable within UCL out of the gate, but would be effectively another interface available to native types. For the builtin ones that exist now, it’ll most likely be little more than syntactic sugar over the standard library functions:

$hello:to-upper

# equivalent to

strs:to-upper $hello

Speaking of the standard library, the use of : as a module-function separator may need some changes. A the moment it’s a bit of a hack: when a module is loaded, any procs defined within that module is stored in the environment with the operator: strs:to-upper for example. The parser is sophisticated enough to recognised : as a token, but when it parses two or more identifiers separated with :, it just joins them up into a single identifier.

What’s needed is something else, maybe using something based on methods. The current idea is to define modules as being some special form of object, where the “methods” are simply the names of the exported symbol.

I was curious to know whether the language was capable of doing something similar now. It’s conceivable that a similar concept could be drafted with procs returning dictionaries of procs, effectively acting as a namespace for a collection of functions. So a bit of time in the playground resulted in this experiment:

proc fns {
  return [
    upper: (proc { |x| strs:to-upper $x })
    lower: (proc { |x| strs:to-lower $x })
  ]
}

(fns).upper "Hello"
--> HELLO

(fns).lower "Hello"
--> hello

A good first start. This is theoretically possible in the language that exists at the moment. It’s not perfect thought. For one thing, the call to fns needs to be enclosed in parenthesis in order to invoke it. Leaving them out results in an error:

fns.upper "Hello"
--> error: command is not invokable

The same is true when using variables instead of procs. I tried this experiment again using variables and it came to the same limitations:

set fns [
  upper: (proc { |x| strs:to-upper $x })
  lower: (proc { |x| strs:to-lower $x })
]

($fns).upper "Hello"
--> HELLO

$fns.upper "Hello"
--> error: command is not invokable

Obviously the parser needs to be changed to add additional support for the : operator, but I also need to fix how strong . binds to values too. But I think this may have legs.

UCL: Iterators

Fri, 31 Jan 2025 08:41:44 +1000

Still working on UCL in my spare time, mainly filling out the standard library a little, like adding utility functions for lists and CSV files. Largest change made recently was the adding iterators to the mix of core types. These worked a lot like the streams of old, where you had a potentially unbounded source of values that could only be consumed one at a time. The difference with streams is that there is not magic to this: iterators work like any other type, so they could be stored in variables, passed around methods, etc (streams could only be consumed via pipes).

I augmented the existing high-level functions like map and filter to consume and produce iterators, but it was fun discovering other functions which also became useful. For example, there exists a head function which returned the first value of a list. But I discovered that the semantics also worked as a way to consume the next element from an iterator. So that’s what this function now does. This, mixed with the fact that iterators are truthy if they’re got at least one pending value, means that some of the builtins could now be implemented in UCL itself. Like the example below, which could potentially be used to reimplement itrs:to-list (this is a contrived example, as foreach would probably work better here).

proc to-list { |itr lst|
   if $itr {
      lists:add $lst (head $itr)
      return (to-list $itr $lst)
   }
   return $lst
}

to-list (itrs:from [1 2 3 4 5]) []

But the biggest advantage that comes from iterators is querying large data-stores with millions of rows. Being able to write a UCL script which sets up a pipeline of maps and filters and just let it churn through all the data in it’s own time is the dream.

list-customers | filter { |x| $x.HasPlan } | map { |x| $x.PlanID } | foreach echo

I’ve got a need for this in the internal backend tool that spurred the development of UCL, and I’m looking forward to using iterators to help here.

Sat, 25 Jan 2025 15:03:08 +1000

This week’s distraction: building a Wordle clone. No particular reason for doing this other than I felt like building one, although I did miss the small time waster of the original Wordle, and watching a game show with my parents that had a similar concept just made those feelings stronger. Main difference between this and Wordle classic: board randomly selects between 4-letter, 5-letter, and 6-letter words; no daily limit or social-media sharing when you guessed the word correctly; and the biggest one: UK English spelling.

Some remarks on how this was built: I used 11ty to build the static site. It originally started as just a HTML page with some JavaScript, but I wanted to leave the option open for bundling and minifying the JS with Stimulus. The dictionary I got from Hunspell, which is apparently the spell checker Apple has based their work on. There is a little bit of Go to filter and sort the dictionary of words. The words are in sorted order for the binary search algorithm to check if a word exists or not. The puzzle order is predetermined and was done by “shuffling” the indices in a separate array. Base styles are, of course, from simple.css.

If you’re interested in checking it out, you can find it here. Just be aware that it may not be as polished as much of the other stuff you find out there. Turns out that I can tolerate a fair few shortcomings in things that I build for my own amusement.

Mon, 20 Jan 2025 21:18:23 +1000

Learnt a very import thing about Stimulus outlets this evening: the outlet name must match the controller name of the outlet target. If this is not the case, the outlet will not bind and you’d be beside yourself struggling to find out why the outlet target cannot be found.

From the docs:

The outlet identifier in the host controller must be the same as the target controller’s identifier.

Took me 30 minutes and stepping through with code with the debugger to find this out.

Sun, 19 Jan 2025 09:24:18 +1000

Started filling out the UCL website, mainly by documenting the core modules. It might be a little unnecessary to have a full website for this, given that the only person who’ll get any use from it right now will be myself. But who knows how useful it could be in the future? If nothing else, it’s a showcase on what I’ve been working on for this project.

Wed, 15 Jan 2025 21:40:42 +1000

I’ve been using UCL a lot recently, which is driving additional development on it. Spent a fair bit of time this evening fixing bugs and adding small features like string interpolation. Fix a number of grammar bugs too, that only popped up when I started writing multi-line scripts with it.

Project Update: DSL Formats For Interactive Fiction

Sun, 12 Jan 2025 10:39:47 +1000

Still bouncing around things to work on at the moment. Most of the little features have been addressed, and I have little need to add anything pressing for the things I’ve been working on recently. As for the large features, well apathy’s taking care of those. But there is one project that is tugging at my attention. And it’s a bit of a strange one, as part of me just wants to kill it. Yet it seems to be resisting.

About 6 months ago, I started working on some interactive fiction using Evergreen. I got most of the story elements squared away but much of the interactive elements were still left to be done. And as good as Evergreen is for crafting the story, I found it a little difficult tracking down all the scripts I needed to write, debug and test.

So in early November, I had a look at porting this story over to a tool of my own, called Pine Needle (yes, the name is a bit of a rip-off). Much like Evergreen, the artefact of this is a choose-your-own-adventure story implemented as a static webpage. Yet the means of building the story couldn’t be more different. Seeing that I’m more comfortable working with code and text files, I eschewed building any UI in favour of a tool that simply ran from the command line.

But this meant that I needed someway to represent the story in text. Early versions simply had the story hard coded in Go, but it wasn’t long before I started looking a using a DSL. My first attempt was a hand-built one based on Markdown with some additional meta-elements. The goal was to keep boilerplate to a minimum, with the meta-elements getting out of the way of the prose. Here’s a sample of what I had working so far:

// Three dashes separate pages, with the page ID following on.
// Also these are comments, as the hash is reserved for titles.
--- tulips-laundry-cupboard

You open the cupboard door and look at the shelf
above the brooms. There are a couple of aerosol cans up there,
including a red one that says "Begone Insecticide".
You bring it out and scan the active ingredients. There are a
bunch of letters and numbers back there, and none of them have
the word "organic."

\choice 'Take Insecticide' to=tulips-take-insecticide
\choice 'Leave Insecticide' to=tulips-leave-insecticide

--- tulips-take-insecticide

You return to the tulips with the insecticide, and start 
spraying them. The pungent odour of the spray fills the air, 
but you get the sense that it's helping a little.

\choice 'Continue' to=tulips-end

--- tulips-leave-insecticide

You decide against using the can of insecticide. You put the
can back on the shelf and close the cupboard door.

\choice 'Look Under The Trough' to=tulips-laundry-trough
\choice 'Exit Laundry' to=tulips-exit-laundry

The goal was to have the meta-elements look like LaTeX macros — for example, \option{Label}{target-screen} — but I didn’t get far in finishing the parser for this. And I wasn’t convinced it had the flexibility I wanted. LaTeX macros relies pretty much on positional arguments, but I knew I wanted key-value pairs to make it easier to rely on defaults, plus easier to extend later.

I did imagine a fully LaTeX inspired DSL for this, but I quickly dismissed it for how “macro-heavy” it would be. For reference, here’s how I imagined it:

\screen{tulips-laundry-cupboard}{
  You open the cupboard door and lo ok at the shelf
  above the brooms. There are a couple of aerosol cans up there,
  including a red one that says "Begone Insecticide".
  You bring it out and scan the active ingredients. There are a
  bunch of letters and numbers back there, and none of them have
  the word "organic."

  \choice{Take Insecticide}{to=tulips-take-insecticide}
  \choice{Leave Insecticide}{to=tulips-leave-insecticide}
}
\screen{tulips-take-insecticide}{
  You return to the tulips with the insecticide, and start 
  spraying them. The pungent odour of the spray fills the air, 
  but you get the sense that it's helping a little.

  \choice{Continue}{to=tulips-end}
}
\screen{tulips-leave-insecticide}{
  You decide against using the can of insecticide. You put the
  can back on the shelf and close the cupboard door.

  \choice{Look Under The Trough}{to=tulips-laundry-trough}
  \choice{Exit Laundry}{to=tulips-exit-laundry}
}

I wasn’t happy with the direction of the DSL, so I looked for something else. I briefly had a thought about using JSON. I didn’t go so far as to try it, but the way this could work is something like this:

{"screens": {
  "id": "tulips-laundry-cupboard",
  "body": "
    You open the cupboard door and look at the shelf
    above the brooms. There are a couple of aerosol cans up 
    there, including a red one that says \"Begone Insecticide\".
    You bring it out and scan the active ingredients. There
    are a bunch of letters and numbers back there, and none of 
    them have the word \"organic.\"
  ",
  "options": [
    {
      "screen":"tulips-take-insecticide",
      "label":"Take Insecticide",
    },
    {
      "screen":"tulips-leave-insecticide",
      "label":"Leave Insecticide",
    }
  ]
}, {
  "id":"tulips-take-insecticide",
  "body":"
    You return to the tulips with the insecticide, and start 
    spraying them. The pungent odour of the spray fills the air, 
    but you get the sense that it's helping a little.
  ",
  "options": [
    {
      "screen":"tulips-end",
      "label":"Continue"
    }
  ]
}}

I generally like JSON as a transport format, but it didn’t strike me as a format that suited the type of data I wanted to encode. Most of what this format would contain would be prose, which I’d prefer to keep as Markdown. But this would clash with JSON’s need for explicit structure. Setting aside the additional boilerplate this structure would require, all the prose would have to be encoded as one big string, which didn’t appeal to me. Also no comments, especially within string literals, which is a major deal breaker.

So, the current idea is to use something based on XML. This has some pretty significant benefits: editors have good support for XML, and Go has an unmarshaller which can read an XML directly into Go structures. JSON has this too, but I think it’s also a pretty decent format for at editing documents by hand, so long as you keep your XML elements to a minimum.

I think one aspect that turned people off XML back in the day was format designer’s embrace of XML’s ability to represent hierarchical data without leaning into it’s use as a language for documents. The clunky XML documents I had to deal with were purely used to encode structure, usually in a way that mapped directly to an domain’s class model. You had formats where you need 10 nested elements to encode a single bit of information that were a pain to read or edit by hand. These were usually dismissed by the designers with promises like, “Oh, you won’t be editing this by hand most of the time. You’ll have GUI design tools to help you.” But these were awful to use too, and that’s if they were available, which they usually were not (did you know building GUIs are hard?)

If you have an XML format that skews closer to HTML rather than something that’s representable in JSON, I think it could be made to work. So yesterday I had a go at seeing whether it this could work for Pine Needle. Here’s what I’ve got so far:

<?xml version="1.0">
<story>
<screen id="tulips-laundry-cupboard">
  You open the cupboard door and look at the shelf
  above the brooms. There are a couple of aerosol cans up 
  there, including a red one that says "Begone Insecticide".
  You bring it out and scan the active ingredients. There
  are a bunch of letters and numbers back there, and none of 
  them have the word "organic."

  <option screen="tulips-take-insecticide">Take Insecticide</option>
  <option screen="tulips-leave-insecticide">Leave Insecticide</option>
</screen>
<screen id="tulips-take-insecticide">
  You return to the tulips with the insecticide, and start 
  spraying them. The pungent odour of the spray fills the air, 
  but you get the sense that it's helping a little.

  <option screen="tulips-end">Continue</option>
</screen>
<screen id="tulips-leave-insecticide">
  You decide against using the can of insecticide. You put the
  can back on the shelf and close the cupboard door.

  <option screen="tulips-laundry-trough">Look Under The Trough</option>
  <option screen="tulips-exit-laundry">Exit Laundry</option>  
</screen>
</story>

The idea is that the prose will still be Markdown, so things like blank lines will still be respected (the parser strips all the leading whitespace, allowing one to easily indent the prose). Attributes satisfy the key/value requirement for the elements, and I get the features that make this easy to modify by hand, such as comments and good editor support.

I think it’s going to work. It would require some custom code, as Go’s unmarshaller doesn’t quite like the mix of prose and declared <option> elements, but I think it’s got the bones of a decent format for this interactive fiction. Already I’m coming up with ideas of how to add script elements and decompose fragments into sub-files to make it easier to test.

I’ll talk more about this project in the future if I’m still working on it. I don’t know if the story that started all this will see the light of day. I’ve gone through it a few times, and it’s not great. But shipping stuff you’re proud of comes from shipping stuff you’re not proud of, and given how far along it is, it probably deserved to be release in one form or another. That’s probably why it’s been saved from the chopping block so far¹.

Yes, this is probably just a rationalisation for trying to minimise sunk-costs, but I’ve got nothing else to work on, so why not this? ↩︎

Sun, 05 Jan 2025 16:10:47 +1000

I recently got a new phone, a Pixel 9 Pro, which meant I needed to bring Alto Player up to date. I probably could’ve gotten away using the version I was using on my Pixel 6. But I didn’t have a binary build, and I needed to upgrade Gradle anyway, so I decided to spend a bit of time bringing it up to date to API version 35, the version used in Android 15.0. Fortunately it was only a few hours in total, and once I got it running in the simulator, I side-loaded it onto my phone and started using it.

It worked, but there were some significant UI issues. The title-bar bled into the status bar, the album image in the Now Playing widget was cropped by the curved corners of the phone, and the media notification didn’t display playback controls.

Evolution of the window insets, from left-to-right: before any changes, version with the album cover and margin, final version with no album art.

I set about fixing these issues today, starting with the title-bar and Now Playing widget. These was an issue with the views not respecting the window insets, and after a quick Google search, I found this article showing how one could resolve this by adding a ViewCompat.setOnApplyWindowInsetsListener and reacting to it by adjusting the margins of the view.

val topLevelLayout = findViewById(R.id.top_level_layout) as CoordinatorLayout
ViewCompat.setOnApplyWindowInsetsListener(topLevelLayout) { v, windowInsets ->
    val insets = windowInsets.getInsets(WindowInsetsCompat.Type.systemBars())
    v.updateLayoutParams<ViewGroup.MarginLayoutParams> {
        leftMargin = insets.left
        bottomMargin = insets.bottom
        rightMargin = insets.right

        // applying a top margin here would work, but will not have the toolbar
        // background colour "bleed" into the status bar, which is what I want.
    }

    val t = v.findViewById<Toolbar>(R.id.toolbar)
    t.updateLayoutParams<ViewGroup.MarginLayoutParams> {
        // make the toolbar a little "narrower" than full height
        topMargin = insets.top * 3 / 4
    }

    WindowInsetsCompat.CONSUMED
}

It took a few attempts, but I managed to get this working. Just using the top inset for the toolbar margin made it a little larger than I liked, so I adjusted the height to be 75% of the inset. This means the toolbar will actually encroach into the area reserved for cut-outs like the front-facing camera. This is arguably not something a “real” Android apps should do, but this is just for me and my phone so it’s fine.

I went through a few iterations of the album artwork cutoff on the bottom right corner trying to find something I liked. I tried bringing in the horizontal margins a little, but I didn’t like the alignment of the album art in the player, particularly compared to the covers that appear in the track list screen. One thing I didn’t try was raising the bottom margin so that it would fit “above” the curve. But those corners curve in quite a bit, and doing this would sacrifice a lot of vertical space. So I settled on hiding the album art altogether. It’s a bit of a shame to loose it, but at least it looks neater now.

The next thing I looked at was fixing the playback controls in the media notification. After some investigating, I found that this was because I was not setting the available actions in the PlaybackStateCompat builder. This, if I understand correctly, is used to communicate to various systems the current state of the playing media: what the track name is, whether it’s playing, whether one can skip forward or back. I have my own types for tracking this information — which is probably not correct but I wasn’t completely sure as to what I was doing at the time with Android’s media stack¹ — and when I needed to convert this to a type understood by the framework, I made instances of this builder without setting the available actions. Earlier versions of Android seemed not to care, and the controls always appeared on the notification. But I guess they changed that.

Evolution of the playback notification, from left-to-right: before any changes, the request to show notifications upon first launch (this is defined by the system), the playback notifications with controls again.

One other thing I needed to do was to explicitly ask the user permission to show a notification before I could publish one. This is also relatively new: my experience with Android goes back to the early days where these permissions were disclosed up front when the app was installed. But I can completely understand why they changed it, as it was easy to simply tap through those screens with reading them. I am wondering whether media playback notifications are in some way exempt from these permission checks, as I was actually getting one to appear before I made this changes. But I figured it was probably worth doing anyway, so I added this permission request on first launch. Arguably I should be asking for this permission when playback starts, but again, this is just for me.

One final thing I needed to address were long album titles. The text view displaying the the album title had a width that autosized to the title itself, and the start and end constraints were set such that it appears centred in the view. This worked for “normal” length titles but when the length became excessive, the text view would fill the entire width of the screen and the title will appear left justified.

Before (left) and after (right) shot of the fixed album title.

The fix for this was to set the text width to be calculated by the start and end constraints (setting layout_width to 0dp), bringing in the margins a little, and making the label text centre justified. I did this already for the track title, so it was easy to do this here too. Not sure why I didn’t do it earlier, or why I haven’t done it for the artist’s name yet.

This was all rushed, and I’ll admit I wasn’t 100% sure what I was doing. I was going down the route of trial-and-error to get this working, mixed in with web searches and a trip to ChatGPT. And yeah, the methods I used won’t make this a portable Android app that would work on every phone out there. But I’m reasonably happy with how it turned out.

This is still true to this day. ↩︎

Sat, 04 Jan 2025 14:25:43 +1000

Thanks for my new found fondness of buying mainstream music instead of streaming it, I needed a way to get these albums into Alto Catalogue. There exists a feature for fetching and importing tracks from a Zip referenced by a URL. This works great for albums bought in Bandcamp, but less so for any tracks I may have on my local machine.

I’ve managed to get Alto Catalogue building again after updating Webpack and a few NPM packages, so in theory, I could add an Upload Zip file action. But there’s more to this than simply accepting and unpacking a Zip file. I have to read the metadata, maybe even preview the tracks that will be imported, just in case I’m importing something I rather not (I did see this once, where zipping a bunch of tracks in the Finder introduced duplicates). This already exists for Zip files that are downloadable online.

I had a though about what my options are, until I remembered that I had a Gokapi instance running in Pikapods. So I tried using that to temporarily host the Zip file with a publicly available URL that could be read by Alto Catalouge.

The only problem is my internet upload speed is sooooo sloooooow. The Gokapi instance is hosted in Europe, and I suspect the instance itself is a little underpowered. So uploading 100 MB Zip files would take a fair bit of time: maybe 15-30 minutes. When I tried doing this via the web frontend, the connection timed out.

Fortunately, Gokapi has an API and one of the methods allows you to upload a file in “chunks,” which Gokapi will assemble back into the original file. Even better is that this chunking can be uploaded in parallel.

So I built a CLI tool which made of this chunking API to upload the Zip files. Once the upload is complete, the tool will display the hot-link URL, which I can copy-and-paste into Alto Catalogue.

The whole process isn’t fast (again, slow upload speeds). But it works, and I can use this tool to queue a bunch of uploads and let it do its thing while I’m doing something else. I really like tools that do this, where you’re not forced to babysitting them through the process.

There are a few limitations with it. It doesn’t allow for an awful lot of customisations on the lifecycle of the uploaded file. And the tool stalled out once when my computer went to sleep, and I had to start the upload from scratch. I could probably add something to track the chunks that were successful, allowing one to continue a stalled upload. If this happens frequently, I may look more into adding this.

But even so, this could be a useful addition to my use of Gokapi for transferring temporary files. If you think this might be useful to you, you can find the tool here.

Tue, 31 Dec 2024 15:28:19 +1000

Fixed the UI of Alto Player, plus addressed some long standing issues I’ve been having.

One was displaying the album covers for playlists instead of the generic “missing album” image. It’s technically possible to set an album cover on a playlist, but I never built the UI to do this in the web-app. So the app now uses the album cover of the first track in the playlist if one isn’t specified. Another was getting automated release builds working in GitHub, as per these instructions

But the biggest improvement was finally getting around to storing the position of the album list, so that going back up the navigation stack wouldn’t reposition the list to the top. I tried this a way back, but couldn’t get it working, probably because I was testing RecyclerView.scrollToPositionWithOffset by passing last constant numbers, like 100, only to find the list not actually scrolling. It turns out that this method actually takes the index of the item to position at the top, not a pixel offsets. So the view wouldn’t scroll if you happen to have a list with less than 100 items. It only started working after I tried smaller numbers, like 5.

So all in all, a good day.

Tue, 31 Dec 2024 10:51:11 +1000

Only took two hours to uplift Alto Player from Android SDK version 30 to 35. Fought an upgrade to Gradle (because of-course), skirted around a migration from ExoPlayer to Media 3, and battled a NullPointerException due to my inability to properly my own navigation args. All in all, not bad.

Sat, 28 Dec 2024 10:30:36 +1000

More fun today working on Blogging Tools. Finished a feature for uploading larger videos to object storage so they can be added to a post using the standard video tag, as opposed to an embedded video player. If you see the screencast below, that means it’s working.

Tue, 17 Dec 2024 21:00:25 +1000

Exploring Godot to see if I could use it to make a card game. I got this far:

Yep, I’m on a roll. 😄

Might need to work through a couple Godot tutorials first, just so that I understand the basics.

Thu, 12 Dec 2024 07:48:00 +1000

I plan to integrate UCL into another tool at work, so I spent last night improving it’s use as a REPL. Added support for onboard help and setting up custom type printing, which is useful for displaying tables of data. I started working on the tool today and it’s already feeling great.

Sat, 07 Dec 2024 13:19:05 +1000

In other building-small-things-for-myself news, I spent a bit of time this morning on the image processor for Blogging Tools. The big new change was adding support for working with multiple source images, instead of just one. This made way for a new “Phone Shot” processor, which arranges multiple screenshots of phone apps in a row, while also downscaling them and proving some quick crops and distribution options.

This should reduce the vertical size of mobile app screenshots I post here, something that’s been bothering me a little.

Sat, 07 Dec 2024 10:05:54 +1000

Some more work on that feed reader app. You know how I said that I wanted to hold off looking at appearances? Well, I abandoned that approach when I installed the app on my phone and tried viewing a few articles. And oof! There’s work to be done there.

First, it’s slow. Very slow. It’s taking a good second or two to pull feeds and entries from Feedbin. I was expecting this, and I’ve got a few plans on how to speed this up. But the biggest concern is the janky list scrolling. I mean, I wasn’t expecting the buttery smoothness of iPhone list scrolling, but I expected Flutter to be better than what I was experiencing. I’m hoping that it’s just because I was running a debug build, but part of me fears that Flutter is just not optimised for smooth list scrolling, favouring ease of development and a common runtime. I rather not change frameworks now, especially after spending an evening dealing with all the build issues, but I don’t want to live with this for ever.

But speed is not the biggest issue. The biggest offender was the feed reader view. The embedded web-view was only lightly styled, and it felt like it. The margins were all off, and I didn’t like the default font or colours. It made reading the article a bad experience to a surprising degree. I’ve dealt with rushed or poorly designed UIs in the past, but I didn’t have much tolerance for this. Not sure why this is, but I suspect it’s because I’ve been using feed readers that put some effort into the design of their reader screen.

In any case, a couple of evenings ago, I decided to put some effort into the styling. I replace the body text font with Noto Sans and the fixed-font with Ubuntu Mono. I dropped the font-size a little to 1.05 em (it was previously 1.1 em, with felt a little big, and 1.0 em felt a little small). I bought the margins in a little. And I styled the block-quote, figure, and pre elements to an appearance that, despite being a little overused, felt quite modern.

The results look much better, at least to my eye (and my emulator). Here are some side-to-side comparison shots of the changes (left side is the unstyled version, while the right side has the new styling changes):

Demonstration of the new font and link colouring choices.

Demonstration of the changes to block-quotes. Having a line down the left-sided is a pretty common style, but it's one I like.

Demonstration of code blocks and figures with captions.

I still need to actually install this on my phone and try it out. I’m wondering whether I should do so after a bit more work syncing the read status with Feedbin. That’s a feature that’s keeping me on Feedbin’s PWA for now.

Tue, 03 Dec 2024 21:45:18 +1000

Spent the last few evenings continuing work on a Flutter-based RSS feed reader. This project is now further along then my previous attempts at doing this. I’m at the point where feeds and feeds items are being fetch from Feedbin and displayed in a list view:

The beginning of the feed list

The feed item list. Title-less posts are to be fully supported, with a bulk of the summary shown in the list. Titled posts should have a smaller summary.

The aesthetics are taking a bit of a back seat in favour of functionality for now; I haven’t even changed the default accent colour. But the infrastructure is there: tapping a feed will bring up the entries for that feed. This is using named routes for navigation, and cubits for state management. It’s a bit more work upfront, but it does make for a neater codebase.

The biggest challenge so far was getting the actual reader view working. I hoping to use the webview plugin, but when I tried adding it, I ran into a bunch of Gradle errors. These were either class version errors or dependency errors, depending on what I tried to fix it (I didn’t get screenshots, sorry). I eventually stumbled upon this Flutter Gradle plugin migration guide, and following this, along with upgrading Java to OpenJDK 25, got Gradle working again.

But I was still getting build errors. The first couple were Gradle plugins com.android.application and org.jetbrains.kotlin.android that needed to be updated. Easy stuff. And then I got this error message:

Execution failed for task ':webview_flutter_android:compileDebugJavaWithJavac'.
> Could not resolve all files for configuration ':webview_flutter_android:androidJdkImage'.
   > Failed to transform core-for-system-modules.jar to match attributes {artifactType=_internal_android_jdk_image, org.gradle.libraryelements=jar, org.gradle.usage=java-runtime}.
      > Execution failed for JdkImageTransform: /Users/leonmika/Library/Android/sdk/platforms/android-34/core-for-system-modules.jar.
         > Error while executing process /Users/leonmika/Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/jlink with arguments {--module-path /Users/leonmika/.gradle/caches/8.10.2/transforms/575ccd1a7426c0be21d9fe3a81898be3-05a021da-a1a7-409f-a30a-bba769b57371/transformed/output/temp/jmod --add-modules java.base --output /Users/leonmika/.gradle/caches/8.10.2/transforms/575ccd1a7426c0be21d9fe3a81898be3-05a021da-a1a7-409f-a30a-bba769b57371/transformed/output/jdkImage --disable-plugin system-modules}

Running a web search on the error revealed this Stack Overflow answer, which resolve it. There were still a few complaints about the the NDK version but after all that, the app finally launched with the web-viewer.

The web-viewer, with a "hello world"-ish test HTML document

I still need to actually render the entry, plus style the HTML a bit. The immediate goal after this, once the reader view, is getting this on my phone to start playing with it. It’s just barebones for now, but I find that the sooner I can start using a project myself, the more likely I am to keep at it.

Mon, 25 Nov 2024 05:53:42 +1000

Looking for my next project to work on. I have a few ideas but my mind keeps wandering back to an RSS reader for Android. I read RSS feeds quite frequently on my phone and Feedbin’s web app is great, but I think I prefer a native app.

I just need to get over the hump of setting up my Android Studios. There’s something about starting a new project there that just sucks the life out of you.

Sun, 24 Nov 2024 05:41:00 +1000

Playing around with some possible UI design choices for that Android RSS Feed Reader. I think I will go with Flutter for this, seeing that I generally like the framework and it has decent (although not perfect) support for native Material styling.

Started looking at the feed item view. This is what I have so far:

Note that this is little more than a static list view. The items comes from nowhere and tapping an item doesn’t actually do anything yet. I wanted to get the appearance right first, as how it feels is downstream from how it works.

The current plan is to show most of the body for items without titles, similar to what other social media apps would show. It occurred to me that in doing so, people wouldn’t see links or formatting in the original post, since they’ll be less likely to click through. So it might be necessary to bring this formatting to the front. Not all possible formatting, mind you: probably just strong, emphasis, and links. Everything else should result with an ellipsis, encouraging the user to open the actual item.

Anyway, still playing at the moment.

Sat, 16 Nov 2024 13:43:04 +1000

It’s done! Cyber Burger, the Pico-8 arcade game I’ve been working on for the last few months, is finished and can now be played online in a (desktop) browser. Check it out here.

Sat, 09 Nov 2024 12:44:59 +1000

I also decided to put the documentation “on-board”, as opposed to putting it on the web. Yes, it breaks from what was typical during the 8-bit gaming period, but I’ve got the space, and it makes adding illustrations easier.

Also forces me to keep it brief, which is no bad thing.

Sat, 09 Nov 2024 12:37:44 +1000

Building out the meta elements of Cyber Burger, including the “menu du jour” a.k.a. the main menu. I’ve used food-service terms for the menu items to maintain the theme, but there is a button to switch them over to more conventional names should it be too unclear.

Weekly Update - 3 Nov 2024

Sun, 03 Nov 2024 20:39:41 +1000

I probably should stop calling these “weekly updates,” seeing that they come up a lot less frequently than once a week. Maybe I should switch to something like “Sunday updates,” or maybe something closer to what this is, which is an excuse to procrastinate by writing about what I’ve been working on, rather than just working on it.

But I’m sure you’re not interested in my willowing about the frequency of these updates, so let’s just get straight to the meat of it.

Cyber Burger

All the logic, graphics, and sound-effects for power-up/power-downs are now finished. They now spawn in randomly, with a frequency and specific types dictated by the current stage. I also added the notion of a “milkshake bonus” which awards the player a bonus multiplier for a short amount of time.

I’ve also made a few balancing changes around demerits. Based on my own testing, I was pretty blasé about loosing demerits, as you could recover a demerit every time you finish a burger. I wanted to discourage that, so I changed things around a little. You still loose demerits if you screw up the burger your trying to build — such as making it too high or not catching the correct item — but you no longer recover demerits for every burger you complete. Instead, you recover one demerit for every $50 you’re awarded. This is now every three to four burgers, depending on how sophisticated they are, which I hope would make loosing demerits something the player would want to avoid.

There’s still the difficulty curve stuff left to do, but I think I’ll start working on the meta elements, like the title screen, main menu and high score tables. I can probably leave out the addition of stages and a difficulty curve if I’m honest, but I would like to have a decent title and menu screen.

The other thing to do is write the manual. I made a start the other day, but there’s much left to do on this front. Part of me wonders whether it make sense adding “on-board documentation.” But part of the fun of using Pico-8 on this project is to imagine a time where this game came out during the late 70’s and early 80’s and the 8-bit era of home consoles. And those games didn’t have on-board documentation. That said, I might add a quick start guide for those that didn’t RTFM.

UCL

I’ve been using that tool I’ve written for work quite often so there was a need to add some additional features to UCL. The biggest one was adding exceptions, but there’ve been a lot of little things like rounding out the standard library.

All the code written in a rush is starting to weigh this project down though, and I do think I’ll need to do some refactoring to tidy things up a little. I may need to work on documentation too, just for my own sake more than anything else. I doubt this would be anything more than the toy language it currently is, but it does have it’s uses, and whenever I need to reference a built-in, I’m always going to the source code. Which is fine, but I think I can do better on this front.

Other Projects

A few other things I worked on during the last fortnight:

I spent some time last weekend playing with Htmgo by making a simple world clock that would update every second. I ended up remaking this as a static web page backed by some WASM code. Knowing the time in UTC and some American cities could come in handy for my job. At least, that’s the theory: I haven’t had a need for it yet.
I also made some changes to Nano Journal to add support for uploading multiple attachments at once.

So that’s the update for this past fortnight.

Sat, 02 Nov 2024 09:39:26 +1000

Title design this morning. Trying to get as close as I can to the Cyberspace Raceway font as my pixel art skills will allow for.

Try-Catch In UCL - Some Notes

Mon, 21 Oct 2024 15:49:09 +1000

Stared working on a try command to UCL, which can be used to trap errors that occur within a block. This is very much inspired by try-blocks in Java and Python, where the main block will run, and if any error occurs, it will fall through to the catch block:

try {
  echo "Something bad can happen here"
} catch {
  echo "It's all right. I'll run next"
}

This is all I’ve got working at the moment, but I want to quickly write some notes on how I’d like this to work, lest I forget it later.

First, much like everything in UCL, these blocks should return a value. So it should be possible to do something like this:

set myResults (try {
  result-of-something-that-can-fail
} catch {
  "My default"
})
--> (result of the thing)

This is kind of like using or in Lua to fallback to a default, just that if the result fails with an error, the default value can be returned from the catch block. In might even be possible to simply this further, and have catch just return a value in cases where an actual block of code is unnecessary:

set myResults (try { result-of-something-that-can-fail } catch "My default")

One other thing to consider is how to represent the error. Errors are just treated out-of-band at the moment, and are represented as regular Go error types. It might be necessary to add a new error type to UCL, so that it can be passed through to the catch block for logging or switching:

try {
  do-something
} catch { |e|
  echo (cat "The error is " $e)
}

This could also be used as the return value if there is no catch block:

set myResult (try { error "my error" })
--> error: my error

Another idea I have is successive catch blocks, that would cascade one after the other if the one before it fails:

try {
  do-something
} catch {
  this-may-fail-also
} catch {
  echo "Always passes"
}

Unlike JavaScript or Python, I don’t think the idea of having catch blocks switching based on the error type would be suitable here. UCL is dynamic in nature, and having this static type checking feels a little wrong here. The catch blocks will only act as isolated blocks of execution, where an error would be caught and handled.

Finally, there’s finally, which would run regardless of which try or catch block was executed. I think, unlike the other two blocks, that the return value of a finally block will always be swallowed. I think this will work as the finally block should mainly be used for clean-up, and it’s the result of the try or catch blocks that are more important.

set res (try {
  "try"
} catch {
  "catch"
} finally {  
  "finally"
})
--> "try"

Anyway, this is the idea I have right now.

Update — I just realised that the idea of the last successful try block return an error, rather than letting it climb up the stack defeats the purpose of exceptions. So having something like the following:

try { error "this will fail" }

Should just unroll the stack and not return an error value. Although if there is a need to have an error value returned, then the following should work:

try { error "this will fail" } catch { |err| $err }
--> error: this will fail

Weekly Update - 20 Oct 2024

Sun, 20 Oct 2024 20:57:17 +1000

Yeah, I know, it’s been a while… again. A lot has been happening in life and there’ve been many days that I haven’t done any work on anything. Things are starting to settle down now, although I am expecting a few more bumpy days ahead, so we’ll see how we go with project work.

Cyber Burger

Yeah, I’m getting pretty tired of this one. I’m right in the trough of despair here, where the initial excitement has worn off and I just want to finish it. All the remaining work for it has been defined, which I think helps, and now it’s just a matter of plowing through the tasks.

Recent work done on this is the addition of power-ups and power-downs, which does things like bump the remain time remaining, give the player a “tractor beam” weapon, and clearing the play-area of items, amongst other things. I’m about 2/3 through this one: most of the these power-ups have been implemented, including the tractor beam. After that, the remaining work on this project is adjusting the difficulty curve, improving the main menu, and finally finishing off the website.

Despite how fatigued I feel about this, it has been fun learning how to build for the Pico-8. The expect fidelity of the pixel art is within my skillset, and it’s quite impressive what it offers in terms of audio effects. I haven’t explored the music options, and I probably won’t for this game, but something to look at for the next one (if there will be a next one).

Coasters

This was a bit of a side tangent to get my mind off working on Cyber Burger. It was a long weekend, and I was passively scrolling through the sample of Vintage Logos when I thought about recreating a game that was published in the weekend newspaper we read while I was growing up. You’ll be presented with two images and a clue, and you’d have to guess the word or phrase these alluded to. For example, if the clue was “insect” and the images were:

Then one possible answer would be “butterfly”.

The newspaper used stock photos, like the example above, but I was hoping to creates logos for fake companies as they would appear as if printed on cardboard coasters you’d find in pubs. The idea is that a new puzzle would be shown to the user a day, much like the word games you find online.

I coded up most of this in a single day and had a working version by the end of the weekend. I had a very simple Go server which served up a new puzzle every day at 00:00 UTC. The web frontend was mainly HTML, with some JavaScript for handling the player’s guesses. The puzzles themselves were defined in a CSV file which was imported into a Sqlite DB on server startup. This was a strange way of doing things, and was more of an accident as I was deploying this as a Docker container and I didn’t setup a persistent volume that survived restarts. I had plans of adding an admin section to the Go server so that I could design puzzles in advanced.

Fortunately I resisted this as I ended up only producing twelve puzzles. Once the original run was completed, I basically bundled them all on a static site. I was able to keep most of the original HTML and JavaScript as it was, with only the Go backend being junked. It’s still online, and maybe I’ll come back to it in the future, maybe using stock images in place of logos I have to create myself.

The new home page, made after the daily puzzles were at an end.

Example of one of the puzzles.

So although this was not a long lived side tangent, it was good to be able to get something finished. I was feeling quite dispirited with projects that I was just not finishing or releasing. Doing something small like this, and seeing it through, helped a lot. Maybe that’s the secret in how I can see through something from conception to completion: get the bulk of it done quickly, deploy it as soon as you can, and keep it small. That would limit the types of project I can deliver, but hey, at least I would deliver them.

Other Stuff

I added a few things to Ted, my terminal CSV editor, this past week. Like many of the other features of this project, this was driven mainly by needs that arise at work. I finally bit the bullet and embraced the support for headers. The first row of the CSV file is now treated as a header and is now fixed above the column, instead of the column numbers which were much less helpful. It should, in theory, be possible to read a CSV file without headers, but I hardly deal with those files that it’s probably just better to make headers the default.

TED with a very simple CSV file showing the headers.

I am also exploring ways to do more actions in bulk, such as add columns which are derived from the value of other columns. These are all quite experimental at the moment, and are more or less added in a quick and dirty fashion; just the bare minimum for what I’m trying to achieve. So not much to say here. A few other changes to other projects:

Nano Journal now keeps the current draft post in browser local storage, so that it’ll no longer get clobbered when you’re redirected to the login screen when trying to post an entry. This was a source of anxiety for me and meant I was less likely to write something off the cuff, so I’m glad this is now fixed.
Blogging Tool now has a new app which takes a clip videos produced by Pocketcasts — which is currently generated in an Instagram-style vertical video — and crops it as either a square or a letterbox.

The new "Post Podcast Clip" app in Blogging Tools.

Also, the app list has been styled a little better. Much more interesting than the bulleted list.

So, a lot of little things done this past month. Nothing worthy of a report on it’s own, but still, it’s something.

Sun, 06 Oct 2024 10:14:31 +1000

I’ve spent the last week working on a small puzzle game called Coasters, where you presented with two images and a clue, and you need to guess the word or phrase. One puzzle a day, sort of like Wordle. I’ve got 10 puzzles ready to go and I may add more but no promises. Check it out if you like.

Weekly Update - 22 Sept 2024

Sun, 22 Sep 2024 12:23:09 +1000

No preface today. Let’s move on to the update.

Cyber Burger

Cyber Burger now has sound!

I started added some basic sound effects to the laser and the items flying across the screen. They may change, depending on how I find them after a while, but it’s a start. I do like how Pico-8 makes these easy to make: select a waveform, then just draw out the pitch and volume graphically:

The laser fire sound effect, shown in the Pico-8 sound editor.

I also improved how items are spawned. Instead of the item types being completely random, they’re now taken from a list, which gets refilled and shuffled once it’s empty. This smoothes out the chance of seeing a particular item — no more waiting around for a bun bottoms to start your burger — while still making things random enough to be fun. The item types are distributed evenly, so every type will show up once per cycle. Maybe I’ll change this, but probably not, as it seems to be working reasonably well at the moment.

The Y location is now quantised a little so that items don’t completely overlap each other by a pixel or two. They can still be located on exactly the same Y position which… is not great — I may need to take a look at that. But the slice spacing between each item looks good. Also, items now spawn in from the left as well as the right.

Lastly, I changed how demerits work. The previous version was not adding any when the player made a mistake. This made them pretty useless, and the game felt quite easy. So now, whenever the player catches an item in the basket that they weren’t suppose to, they get a demerit. If their burger gets too large, then get 2 demerits. To balance things out, the player now has 6 demerit points instead of 3, and every completed burger would remove one demerit until the player has zero again. I think this will require some more rebalancing, but it makes play a little more interesting.

Not sure what I’ll work on next. I’ll need to finish (or redo) the sounds, and then come up with some more tasks to do. I’m thinking either working on the difficulty curve, making things harder as the game progresses; adding power-ups and power-downs; or working on the start screen and menu. More on either of these in the future.

Blogging Tool

For my “extracurricular” activity this week, I spend some more time on Blogging Tool. I added the notion of jobs, which are essentially background tasks for long running processes. These can be monitored from within the Jobs section, which is accessible from the main nav:

The new Jobs section, showing a job in progress.

Once completed, jobs can link to any resource — file or gallery — that they produced.

I’m recording jobs in the Sqlite3 database, including the ones that are currently running. Jobs have the ability to report progress by updating the “summary” message, which would update the job record in the database. I’m generally not too keen on having the database act as storage for something that could be updated quite frequently, but the alternative was keeping this progress information in memory or deploying something like Redis, both I felt were worse. Besides, I’m curious as to how well Sqlite3 handles status updates from a running job like this. I’m not expecting jobs to be updating there status too rapidly anyway, so it should be fine.

The Image App now with the Load URL and Send To Blog options.

The Image App got a few changes as well. It’s now possible to import an image from a URL (this is done within the browser so is subject to all the cross-origin rules that entails). I added this just so that I can touch up my own images that I already uploaded to Micro.blog, without having to download the image first. Speaking of, there’s now an option to send a processed image to a blog directly from the app, saving yet another download step. I’ve got this configured to Micro.blog but it uses Micropub to do this so it could, in theory, be any blogging system that supports that API.

The New Upload Zip app.

Complete with upload progress bar for slow upload connections (like mine).

The last thing that was added is the ability to upload a Zip file of images to a Micropub endpoint, and start creating a gallery. This has been a feature that I wanted for a while. It always took me ages to prepare a photo gallery in Micro.blog. I know there are apps which help here, but they’re for iOS and just don’t fit my workflow. Now, whenever I want to add a photo gallery, I can select the images from Google photos, download them as a Zip file, then directly upload them to Micro.blog in one hit. The upload can take as long as it needs to (this uses the new jobs system as well), and once they’re done, Blogging Tool will create a gallery for me so I can write the captions. To be able to do this without clicking around as much as I did is rather exciting.

This is not a new screen, but shows the gallery created once the images are uploaded.

Or at least it will be, when I find out how I can get large uploads working. Uploading a 27 MB Zip file in development was not an issue, but when I deployed it to my Dokku server and try it for real, I kept getting 502 errors. I’m not sure what’s causing these yet: both Nginx and Blogging Tool have been configured to accept files much larger than this, and there are no logs in either to suggest a problem. There might be a timeout somewhere that needs to be raised. In any case, hopefully this is the last hurdle I need to clear to get this working. Anyway, that’s all for this week. Oh, actually, one more thing: while working on the UI for the Upload Zip screen, I found that I never got around to changing the instructions on the Transcode Audio screen after copying and pasting the HTML from the New Gallery screen:

That's not what this screen is for!

Just goes to show how little I pay attention to this copy, even while working on my own apps. 😛

Weekly Update - 15 Sept 2024

Sun, 15 Sep 2024 10:15:41 +1000

Two projects to discuss this week.

Cyber Burger

I’ve decided to ditch game mode A, where the player is given a series of stages they need to clear. Instead, I’m changing this to be closer to an old-school arcade experience. In this mode, you start the game with a 45 second timer, and you need get as high a score as you can before the timer runs out. Your score depends roughly on how large and “interesting” your burger is. Every burger you make also adds 10 seconds to the clock, so the ultimate aim is to balance making interesting burgers for a higher score, vs. trying to avoid letting the clock run down to zero. Or at least that’s the idea. I need to do some rebalancing, as I think the game is a little easy.

I am wondering whether to award points based on how fast you make the burger. It was originally my intention, but I’m now wondering if that would be considered unfair, as you don’t get to choose the items flying across the screen. I still do need to fix the random number generator too, so as to avoid keeping the player from waiting around too long for a bun base to start building their burger. Resolving these two things will be my goal for the next week.

This working version has been pushed to the web for anyone to try out. If you do, please let me know if you have any feedback.

Nano Journal

The next project I spent some time on was Nano Journal, the web-based journalling app ~~stolen from~~ inspired by Kev Quirk’s journalling app.

I wasn’t intending to work on it this week, but I did have a motive to do so as I was expecting to do something that I wanted to document (I ended up not doing that thing).

The biggest changes I made were adding paging and post titles, plus making some improvements on the backend on how posts are stored and retrieved.

Current version of the index page. Note the posts with titles, which will appear beside the date. The paperclip indicates a post with attachments (usually images).

Setting the title is done using a slash-commands:

/title This is my post title
This is my post body

I also added an /append slash-command to add to the latest post, rather than create a new one. This also works for attachments too, allowing a somewhat clunky way of adding multiple attachments to a single post.

Viewing a post, now with paging.

I am still considering this a prototype, but I have found myself writing here more often than Day One. I definitely need to start thinking about the long-term storage of posts if I want to “productionise” this. Ultimately I want to get them saved in a private Git repository. Each post is just a markdown file stored on the file-system, which will make this easy to do, but I’m somewhat procrastinating on relearning how to use the various Go Git clients that are available.

One other thing that I’d like to do is improve instances where a post couldn’t be saved due to network issues. The way I’m thinking of doing this is to store the current draft in browser local storage, at least until it’s saved on the server. I’m also wondering if it’s worth adding the concept of draft posts. Not sure at this stage whether that juice is worth the squeeze, at least not yet. This was meant to be a relatively simple side-track. But I guess that’s the danger (beauty?) of starting something and finding it useful. 🤷

Anyway, that’s all for this week.

Project Updates

Sun, 08 Sep 2024 17:14:51 +1000

Well, it’s been three weeks since my last post here, and as hard as it was to write this update, not writing it would’ve been harder. So let’s just skip the preamble and go straight to the update.

Cyber Burger (That Pico-8 Game)

I’m terrible at being coy, I’ll just spill the beens. That game I’ve been working on is call Cyber Burger. It’s based on a DOS game I saw on YouTube, and it seemed like a fun project to try and work on, with some tweaks to the gameplay that I think would make it more forgiving.

In the weeks since my last update, I finished with the prototypes and started working on the game itself. The initial set of art is done and a very simple “Game A” mode has been implemented. In this mode, you are shown a burger you’ll need to make in your basket. You do so by shooting down the items flying across the screen and catching them in order. When you do, you get a “tip”, which basically amounts to points. If you make a mistake, you’re given a demerit. There are five rounds in total, and once the round is complete, you move on to the next one, with maybe slightly different items, or different item speeds, etc.

An example play session, with the new graphics.

I managed to make a HTML version of this which plays through round 1. I gave it to someone at work to play test and the results were… well, they weren’t bad but it didn’t set the world on fire.

I think I’m okay with that, but I do need to keep working on it. I think one thing that would help is adding sound. And I think it might help me deliver this earlier if I abandoned Mode A and start working on Mode B, which is closer to an arcade style of game that would allow for continuous play. These two things, I’ll aim to work on this next week.

Oh, and I’ll need to fix the item spawner. Waiting ages for an item you need is no good.

If you’re interested in giving it a try, you can do so by following this link (it runs in the browser). Feel free to send any feedback you may have.

UCL

The other thing I’ve been spending some time on over the last week or so was UCL. I’ve been using that work tool which has this language quite often recently and I’ve been running against a number of bugs and areas where quality of life changes could be made. Just small things, such as allowing the foreach command to be called with a proc name instead of requiring a block, much like the map function. Those have been addressed.

But the biggest thing I’ve been worked on was building out the core library. I added the following functions:

The `seq` function, used for generating a sequence of integers. I like how I built this: it’s effectively a virtual list — that can be indexed, iterated over, or calculated the length of — but does not take up linear space.
Comparator functions, like `eq`, `ne`, `gt`, etc. plus settling on a type system much like Python, where values are strongly type (can’t compare ints to strings) but are also dynamic.
Arithmetic functions, like `add`, `sub`, etc. which operate on integers, and the `cat` function use to concatenate strings (these functions do try to cohere values to the correct type)
Logical functions, like `and`, `or`, and `not`.

Along with this, I’ve started working through the strings package, which would add the various string functions you see, like trimming whitespace, splitting, joining, etc. I’ve got trimming and converting to upper and lower case, and my goal for next week is to add splitting to and joining from string lists. Once that’s done I’ll probably put this on the back-burner again so I can finish off Cyber Burger or work on something else.

Just a reminder that there’s also a playground for this too, although I apologise for the lack of documentation. I’ll need to get onto that as well.

Thu, 22 Aug 2024 07:41:13 +1000

I enjoyed reading Kev Quirk’s post about building a simple journal. I’m still using Day One, but I am still thinking of moving off it. So I was inspired to build a prototype similar to Kev’s, just to see if something similar works for me. Built using Go instead of PHP, but it also uses Simple CSS.

Project Seed - A Pico-8 Prototype

Sun, 18 Aug 2024 09:57:20 +1000

Oof, another long stretch between updates. This has not been a productive winter.

Much of the project I’ve been writing about here are, shall we say, “on ice”. UCL is still being used for the project it’s been built for, but there’s been no further work done on it recently. I think we can safely say Photo Bucket is dead, at least for now. Blogging Tool and that interactive fiction project is still ongoing, but both are running on a slow burn. I was hoping to get the fiction thing done by the end of winter, but it’s likely that timeline will slip. Maybe some time in spring.

What have I been doing in the meantime? Watching YouTube videos on old DOS games, actually. Hardly an activity worth writing about here. But it did get me wanting to try working on a game again. And I did get an idea for one while watching videos of someone going through a collection of shovelware titles.

This project is designated the codename “Project Seed”. I’m going to be a little cagey about the details, at least for now. But I will say that I’m planning to use Pico-8 for this. I bought a license for Pico-8 about 2 years ago (oof, did not expect it to have been that long ago) and I watched a few videos on how to use it, but I didn’t have a good idea for a project then. It is a fascinating bit of software, and I know it’s quite popular amongst hobbyists. One thing I like about it is that the games made with it are not expected to have great art. As someone who can’t draw to save himself, this works in my favour. So don’t expect anything resembling Celeste from me! 😄 Pico-8 also targets HTML5, which works for me.

Anyway, I have this idea, and I thought about starting a prototype to see how it feels. I downloaded Pico-8, spun up a new project, and started drawing some of the graphics. I’ve got the bare minimum so far: a user-controlled paddle, called the “basket”; a laser bullet, and a thing that needs to be shot.

The first few sprites. Must say I really like the Pico-8 palette.

Next was coding up the Lua code. Using the Pico-8 builtin editor was fine for a bit, but I eventually switch to Nova just for the screen size. I am still trying to adhere to the whole retro-style approach to Pico-8. The code I write is still bound to the 8192 token limit, and I’m trying to avoid using too much virtual memory, capping elements to only a handful. But, yeah, using Nova to write the logic is so much better.

Anyway, the first milestone was allowing the player to move the basket around and shoot laser bullets. Then it was to get one of the shootable items moving across the field. The idea is that the player will need to fire the laser to hit the shootable item. When it’s hit, it begins to fall, and the player needs to catch it in the basket.

Shooting the laser and catching the item in the basket.

This took about an hour or so, and already the glimpse of the core game mechanics are starting to show through. They’re just ridiculously primitive at this stage. I mean, the item really shouldn’t fall through the basket like that. But given that it’s a prototype, I’m okay with this so far.

Next experiment was spawning multiple items onto the field. This got off to an interesting start:

An "item" train.

But adding a bit of randomness to the Y position and the spawn delay managed to make things a little more natural again:

A more natural item spawner.

One thing I’m considering is whether to add some randomness to the item X velocity, and even have items move from right to left. But this will do for now.

At this stage, items were just being added to an array, which grew without bounds, and were not being released when they left the screen. Obviously not a good use of memory (even though this is running on an M2 Mac Mini and not retro hardware from the 1980’s, but that’s hardly the point of this exercise). Furthermore, the player is only able to shoot one bullet at a time, and those bullets weren’t being released either. So I set about resolving this, trying to do so in the spirit of a Pico-8 game. I’ve setup a fixed Lua array, which will grow up to a max size, and added a simple allocator which will search for an empty slot to put the next item in.

function next_slot(tbl,n)
  if \#tbl < n then
    return \#tbl+1
  else
    for i = 1,n do
      if tbl[i] == nil then
        return i
      end
    end
  end
  return nil
end

This makes releasing items really easy: just set that slot to nil. It does mean that I can’t use ipairs to iterate over items, though. Instead I have to use the following construct:

for i,item in pairs(items) do
  if item then
     -- Do thing with item
   end
end

It’s here that I wished Lua had a continue statement.

I used this for both the item and bullets: there can now be up to 8 items and 4 bullets on the screen at a time. After making those changes, the prototype started to play a little better:

So, a good start. But there are definitely things need to be fixed. The basket needs to be wider, for one. It’s fine for the prototype, and I’m okay with it the collision being pretty lenient, but it’s too narrow to make it fun.

But the biggest issue is that the collision logic sucks. Bullets are flying through the items, and items are falling through the basket. I’m using a point-in-rectangle approach to collision detection, with a few probing points for each item, but they obviously need to be adjusted.

So collision is what I’m hoping to work on next. More on this when I get around to it.

Current Project Update

Mon, 29 Jul 2024 07:12:14 +1000

Hmm, another long gap between posts. A little unexpected, but there’s an explanation for this: between setting up Forgejo and making the occasional update to Blogging Tools, I haven’t been doing any project work. Well, at least nothing involving code. What I have been doing is trying my hand at interactive fiction, using Evergreen by Big River Games.

Well, okay, it’s not completely without code: there is a bit of JavaScript involved for powering the “interactive” logic part. But a bulk of the effort is in writing the narrative, albeit a narrative that’s probably closer to a video game rather than a work of pure fiction.

Why? The main reason is to try something new. I had the occasional fancy to try my hand at fiction, like a short story or something. I have an idea for a novel — I mean, who doesn’t? — but the idea of writing it as a novel seems daunting at the moment (I’ve written it as short story for NanoWriteMo. It’s sitting online as an unedited draft somewhere. I should probably make a backup of it, actually). But the idea of writing something as interactive fiction seemed intriguing. I was never into text parser adventures, but I did enjoy the choose-your-own-adventure books growing up.

So what’s the story about, I hear you saying? Well, believe it or not, it’s about gardening. Yes, something I have zero experience in. And perhaps that’s what made it an interesting subject to explore.

I’ve been working on this for about a month now. I’m well past the everything-is-new-and-exciting phase, and I think I just made it through the oh-no-why-the-heck-am-I-even-doing-this pit of despair. I can see the finish line in terms of the narrative and the logic, and all that remains there should just be a matter of cleaning up, editing, and play testing. The biggest thing left to do is illustrations. I have zero artistic skills myself so I’m not quite sure what I’ll do here.

If you’re curious about it, here’s a sample. It’s about the first third of the story. It’s a little rough, and requires editing and proof-reading, and illustrations. But let me know what you think.

More Tools For Blogging Tool

Sun, 23 Jun 2024 09:54:56 +1000

Spent the last week working on Blogging Tool. I want to get as much done as a I can before motivation begins to wain, and it begins languishing like every other project I’ve worked on. Not sure I can stop that, but I think I can get the big ticket items in there so it’ll be useful to me while I start work on something else.

I do have plans for some new tools for Blogging Tool: making it easier to make Lightbox Gallery was just the start. This last week I managed to get two of them done, along with some cross-functional features which should help with any other tools I make down the road.

Move To Sqlite

First, a bit of infrastructure. I moved away from Rainstorm as the data store and replaced it with Sqlite 3. I’m using a version of Sqlite 3 that doesn’t use CGO as the Docker container this app runs in doesn’t have libc. It doesn’t have as much support out there as the more popular Sqlite 3 client, but I’ve found it to work just as well.

One could argue that it would’ve been fine sticking with Rainstorm for this. But as good as Rainstorm’s API is, the fact that it takes out a lock on the database file is annoying. I’m running this app using Dokku, which takes a zero-downtime approach to deployments. This basically means that the old and new app container are running at the same time. The old container doesn’t get shut down for about a minute, and because it’s still holding the lock, I can’t use the new version during that time as the new container cannot access the Rainstorm database file. Fortunately, this is not an issue with Sqlite 3.

It took me a couple of evenings to port the logic over, but fortunately I did this early, while there was no production data to migrate. I’m using Sqlc for generating Go bindings from SQL statements, and a home grown library for dealing with the schema migrations. It’s not as easy to use as the Rainstorm API but it’ll do. I’m finding working with raw SQL again to be quite refreshing so it may end up being better in the long run.

Current landing page.

Imaging Processing

Once that’s done, I focused on adding those tools I wanted. The first one to sit alongside the gallery tool, is something for preparing images for publishing. This will be particularly useful for screenshots. If you look carefully, you’d noticed that the screenshots on this site have a slightly different shadow than the MacOS default. It’s because I actually take a screenshot without the shadow, then use a CLI tool to add one prior to upload. I do this because the image margins MacOS includes with the shadow are pretty wide, which makes the actual screenshot part smaller than I like. Using the CLI tool is fine, but it’s not always available to me. So it seemed like a natural thing to add to this blogging tool.

So I added an image processing “app” (I’m calling these tools “apps” to distinguish them from features that work across all of them) which would take an image, and allows you to apply a processor on it. You can then download the processed image and use it in whatever you need.

The image processing tool, being used here to get the crop right for this particular screenshot.

This is all done within the browser, using the Go code from the CLI tool compiled to WASM. The reason for this is performance. These images can be quite large, and I’d rather avoid the network round-trip. I’m betting that it’ll be faster running it in the browser anyway, even if you consider the amount of time it takes to download the WASM binary (which is probably around a second or so).

One addition I did add was to allow processors to define parameters which are shown to the user as input fields. There’s little need for this now — it’s just being used in a simple meme-text processor right now — but it’s one of those features I’d like to at least get basic support for before my interest wains. It wouldn’t be the first time I stopped short of finishing something, thinking to my self that I’d add what I’ll need later, then never going back to do so. That said, I do have some ideas of processors which could use this feature for real, which I haven’t implemented yet. More on that in the future, maybe.

The "Cheeseburger" processor, and it's use of image parameters.

Audio Transcoding And Files

The other one I added deals with audio transcoding. I’ve gotten into the habit of narrating the long form posts I write. I usually use Quicktime Player to record these, but it only exports M4A audio files and I want to publish them as MP3s.

So after recording them, I need to do a transcode. There’s an ffmpeg command line invocation I use to do this:

ffmpeg -i in.m4a -c:v copy -c:a libmp3lame -q:a 4 out.mp3

But I have to bring up a terminal, retrieve it from the history (while it’s still in there), pick a filename, etc. It’s not hard to do, but it’s a fair bit of busy work

I guess now that I’ve written it here, it’ll be less work to remember. But it’s a bit late now since I’ve added the feature to do this for me. I’ve included a statically linked version of ffmpeg in the Docker container (it needs to be statically linked for the same reason why I can’t use CGO: there’s no libc or any other shared objects) and wrapped it around a small form where I upload my M4A.

The simple form used to transcode an M4A file.

The transcoding is done on the server (seemed a bit much asking for this to be done in the browser) but I’m hoping that most M4A files will be small enough that it wouldn’t slow things down too much. The whole process is synchronous right now, and I could’ve make the file available then and there, but it wouldn’t be the only feature I’m thinking of that would produced files that I’d like to do things with later. Plus, I’d like to eventually make it asynchronous so that I don’t have to wait for long transcodes, should there be any.

So along with this feature, I added a simple file manager in which these working files will go.

The files list. Click the link to download the file (although this may changed to be preview in the future).

They’re backed by a directory running in the container with metadata managed by Sqlite 3. It’s not a full file system — you can’t do things like create directories, for example. Nor is it designed to be long term storage for these files. It’s just a central place where any app can write files out as a result of their processing. The user can download the files, or potentially upload them to a site, then delete them. This would be useful for processors which could take a little while to run, or run on a regular schedule.

I don’t have many uses for this yet, apart from the audio transcoder, but having this cross-functional facility opens it up to features that need something like this. It means I don’t have to hand-roll it for each app.

Anyway, that’s the current state of affairs. I have one, maybe two, large features I’d like to work on next. I’ll write about them once they’re done.

Blogging Gallery Tool

Thu, 13 Jun 2024 17:19:38 +1000

Oof! It’s been a while, hasn’t it.

Not sure why I expected my side-project work to continue while I’m here in Canberra. Feels like a waste of a trip to go somewhere — well, not “unique”, I’ve been here before; but different — and expect to spend all your time indoors writing code. Maybe a choice I would’ve made when I was younger, but now? Hmm, better to spend my time outdoors, “touching grass”. So that’s what I’ve been doing.

But I can’t do that all the time, and although I still have UCL (I’ve made some small changes recently, but nothing worth writing about) and Photo Bucket, I spent this past fortnight working on new things.

The first was an aborted attempt at an RSS reader for Android that works with Feedbin. I did get something working, but I couldn’t get it onto my mobile, and frankly it was rather ugly. So I’ve set that idea aside for now. Might revisit it again.

But all my outdoor adventures did motivate me to actually finish something I’ve been wanting to do for a couple of years now. For you see, I take a lot of photos and I’d like to publish them on my Micro.blog in the form of a GLightbox gallery (see this post for an example). But making these galleries is a huge pain. Setting aside that I always forget the short-codes to use, it’s just a lot of work. I’m always switching back and forth between the Upload section in Micro.blog, looking that the images I want to include, and a text file where I’m working on the gallery markup and captions.

I’ve been wishing for some tool which would take on much of this work for me. I’d give it the photos, write the captions, and it would generate the markup. I’ve had a run at building something that would do this a few times already, including an idea for a feature in Photo Bucket. But I couldn’t get over the amount of effort it would take to upload, process, and store the photos. It’s not that it’d would be hard, but it always seemed like double handling, since their ultimate destination was Micro.blog. Plus, I was unsure as to how much effort I wanted to put into this, and the minimum amount of effort needed to deal with the images seemed like a bit of a hassle.

One of the earlier attempts at these. Images were hosted, and were meant to be rearranged by dragging. You can see why this didn't go anywhere.

It turns out the answer was in front of me this whole time. The hard part was preparing the markup so why couldn’t I build something that simply did that? The images would already be in Micro.blog; just use their URLs. A much simpler approach indeed.

So I started working on “Blogging Tools”, a web-app that’ll handle this part of making galleries. First, I upload the images to Micro.blog, then I copy the image tags into to this tool:

Create a new gallery by pasting the image tags from Micro.blog.

The tool will parse these tags, preserving things like the “alt” attribute, and present the images in the order they’ll appear in the gallery, with text boxes beside each one allowing me to write the caption.

The images get displayed alongside the captions, which you can edit on this screen.

Once I’m done, I can then “render” the gallery, which will produce the Hugo short-codes that I can simply copy and paste into the post.

Listing the galleries. Here you select "Render" from the Action column.

Copy and paste the snippet into Micro.blog.

This took me about a few evenings of work. It’s a simple Go app, using Fiber and Rainstorm, running in Docker. Seeing that the image files themselves are not managed by the tool, once I got the image parsing and rendering done, the rest was pretty straight forward. It’s amazing to think that removing the image handling side of things has turned this once “sizeable” tool into something that that was quick to build and, most importantly, finally exists. I do have more ideas for this “Blogging Tool”. The next idea is porting various command line tools that do simple image manipulation to WASM so I can do them in the browser (these tools were use to crop and produce the shadow of the screenshot in this post). I’m hoping that these would work on the iPad, so that I can do more of the image processing there rather than give up and go to a “real” computer. I should also talk a little about why I chose Rainstorm over Sqlite, or whether that was a good idea. Maybe be more on those topics later, but I’ll leave it here for now.

Bulk Image Selection

Sun, 26 May 2024 09:16:51 +1000

Some light housekeeping first: this is the 15th post on this blog so I thought it was time for a proper domain name. Not that buying a domain automatically means I’ll keep at it, but it does feel like I’ve got some momentum writing here now, so I’ll take the $24.00 USD risk. I’d also like to organise a proper site favicon too. I’ve got some ideas but I’ve yet to crack open Affinity Design just yet.

Anyway, I’ve been spending some time on Photo Bucket on and off this past week. I’ve fully implemented the new page model mentioned in the last post, and hooked it up to the switcher in the “Design” admin section. I’ve also built the public gallery and gallery item page.

Example of the landing page showing galleries (yes a lot of Jeff Wayne's War of the Worlds album covers in my test images.

Click one of the galleries to view the items within that gallery.

They’re a little on the simplistic side. That’s partly due to my minimalistic design sensibilities, but it’s also because I haven’t spent a lot of time on the public pages yet. I probably shouldn’t leave it too late, lest my impressions on how it looks drops to the point where I loose interest in working on this again. It’s a challenge, but I guess my counter is that I’ll probably be spending more time in the admin section, so as long as the experience is good enough there, I can probably go by with a very basic public site for now (but not for ever). Now that galleries can be shown on the landing page, I’d like to organise another deployment so that I can start showing images in galleries. But before I do, I’ll need an easy way to move all the existing images into a gallery. Clicking into 25 individual images just to select which gallery they should belong to doesn’t sound desirable to me. So I spent some time adding batch operations to the image admin page. The way it works is that by pressing Shift and clicking the images, you can now select them and perform batch operations, such as add them to a gallery (this is the only one I have now).

I do like how the selection indictor came out. It’s got some DaVinci Resolve vibes (I’ve been using DaVinci Resolve recently to edit some videos so I may have been inspired by their design language here) but I think I might need to use another highlight colour though: I think the black bleeds too easily into the images. Also, while I was recording the demo, I realise I broke the ability to rearrange gallery items. I may need to fix that before redeploying. Clicking “Gallery” brings up a model similar to the one used in the individual image page. It work’s slightly differently though: instead of choosing whether the images appear in the gallery or not, this one is used to choose which galleries to add the selected images to.
I’m not sure that this is the best modal for this. It was quick to add, but I get the feeling that using the same model in slightly different ways could confuse people. So I might do something else here. An idea I have is a modal more like the following:

A better modal for managing galleries for multiple images.

The idea is that all the galleries will be listed like before, but will have a three-segmented button to the right. The centre button will be selected by default, and will show how many of the selected images are currently within that particular gallery. To the left will be the option to remove those images from the gallery, and to the right will be the option to add all the remaining selected images to the gallery. These are identified with the number of the selected images each gallery will have when the user clicks “Save”: 0 for none, and the number of selected images for all. For good measure is an option to add all the selected images into a brand new gallery. This will require some backend work so I haven’t started work on this yet. Not sure if this too will be a bit confusing: may need some additional text explaining how it all works. I’m hoping that users would recognise it as operating similar to the Gallery model for a single image.

The Site Page Model

Sun, 19 May 2024 10:00:11 +1000

I opened up Photo Bucket this morning and found a bunch of commits involving pages. I had no idea why I added them, until I launched it and started poking around the admin section. I tried a toggle on the Design page which controlled whether the landing page showed a list of photos or galleries, and after finding that it wasn’t connected to anything, it all came flooding back to me. So while what I’m going to describe here isn’t fully implemented yet, I decided to write it down before I forget it again.

So here is where the story continues. Now that galleries have been added to the model, I want to make them available on the public site. For the first cut, I’m hoping to allow the admin (i.e. the site owner) the ability to switch the landing page between a grid of photos or a grid of galleries. This is that single toggle on the “Design” page I was talking about earlier:

The Design section showing the new "Show Gallers on Homepage" toggle.

Oh, BTW: I finally got around to highlighting the active section in the admin screen. I probably should’ve done this earlier, as deferring these “unnecessary aesthetics tasks” does affect how it feels to use this — and whether or not I’m likely to continue working on it.

Anyway, if this was before the new model I would’ve implemented this as a flag on the Design model. But I’d like to start actually building out how pages on the site are to be modelled. What I’m thinking is an architecture that looks a little like the following:

New architecture, with the new "Pages" model.

The site content will be encoded using a new Page model. These Pages will be used to define the contents of the landing page (each site will have at least this page by default) along with any additional pages the user would like to add. Galleries and photos will automatically have their own pages, and will not need to have any specific Page models to be present on the site. How these will look, plus the properties and stylings of the site itself, will be dictated by the Design model.

Each Page instance will have the following properties:

Slug — the path the page is available on. For the landing page, this will be `/`.
Type — what the page will contain.
other properties like title, description, etc. which have yet to be defined.

The “page type” is probably the most important property of a Page, as it will dictate what the contents of the page will be comprised of. The following page types will be supported at first:

Photos — grid of all the photos managed on the site.
Galleries — grid of all the galleries managed on the site.

The user will probably not have much control over how these pages will look, apart from styling, footers, etc. which will live on the Design model. But I’m also thinking of adding a page type which would just produce a standard HTML page from a Markdown body. This could be useful for About pages or anything else the user may want to add to their site. I haven’t thought about navigation but I think choosing whether to include the page on the site’s nav bar would be another useful property.

The result would be a sitemap that could look a little like the following, where all the user defined pages reference automatically created pages:

Example sitemap, with the Page instances with solid boarders reference automatically define pages with dotted boarders.

And this is what that single toggle should do. It should change the page type of the landing page between photo list and gallery list.

As you can probably guess, there’s currently no way to add additional pages at the moment. But I’m doing this work now so that it should be easier to do later.

Indexing In UCL

Fri, 10 May 2024 11:09:46 +1000

I’ve been thinking a little about how to support indexing in UCL, as in getting elements from a list or keyed values from a map. There already exists an index builtin that does this, but I’m wondering if this can be, or even should be, supported in the language itself.

I’ve reserved . for this, and it’ll be relatively easy to make use of it to get map fields. But I do have some concerns with supporting list element dereferencing using square brackets. The big one being that if I were to use square brackets the same way that many other languages do, I suspect (although I haven’t confirmed) that it could lead to the parser treating them as two separate list literals. This is because the scanner ignores whitespace, and there’s no other syntactic indicators to separate arguments to proc calls, like commas:

echo $x[4]      --> echo $x [4]
echo [1 2 3][2] --> echo [1 2 3] [2]

So I’m not sure what to do here. I’d like to add support for . for map fields but it feels strange doing that just that and having nothing for list elements.

I can think of three ways to address this.

Do Nothing — the first option is easy: don’t add any new syntax to the language and just rely on the index builtin. TCL does with lindex, as does Lisp with nth, so I’ll be in good company here.

Use Only The Dot — the second option is to add support for the dot and not the square brackets. This is what the Go templating language does for keys of maps or structs fields. They also have an index builtin too, which will work with slice elements.

I’d probably do something similar but I may extend it to support index elements. Getting the value of a field would be what you’d expect, but to get the element of a list, the construct .(x) can be used:

echo $x.hello     \# returns the "hello" field
echo $x.(4)       \# returns the forth element of a list

One benefit of this could be that the .(x) construct would itself be a pipeline, meaning that string and calculated values could be used as well:

echo $x.("hello")
echo $x.($key)
echo $x.([1 2 3] | len)
echo $x.("hello" | toUpper)

I can probably get away with supporting this without changing the scanner or compromising the language design too much. It would be nice to add support for ditching the dot completely when using the parenthesis, a.la. BASIC, but I’d probably run into the same issues as with the square brackets if I did, so I think that’s out.

Use Parenthesis To Be Explicit — the last option is to use square brackets, and modify the grammar slightly to only allow the use of suffix expansion within parenthesis. That way, if you’d want to pass a list element as an argument, you have to use parenthesis:

echo ($x[4])       \# forth element of $x
echo $x[4]         \# $x, along with a list containing "4"

This is what you’d see in more functional languages like Elm and I think Haskell. I’ll have see whether this could work with changes to the scanner and parser if I were to go with this option. I think it may be achievable, although I’m not sure how.

An alternative way might be to go the other way, and modify the grammar rules so that the square brackets would bind closer to the list, which would mean that separate arguments involving square brackets would need to be in parenthesis:

echo $x[4]         \# forth element of $x
echo $x ([4])      \# $x, along with a list containing "4"

Or I could modify the scanner to recognise whitespace characters and use that as a guide to determine whether square brackets following a value. At least one space means the square bracket represent a element suffix, and zero mean two separate values.

So that’s where I am at the moment. I guess it all comes down to what works best for the language as whole. I can live with option one but it would be nice to have the syntax. I rather not go with option three as I’d like to keep the parser simple (I rather not add to all the new-line complexities I’ve have already).

Option two would probably be the least compromising to the design as a whole, even if the aesthetics are a bit strange. I can probably get use to them though, and I do like the idea of index elements being pipelines themselves. I may give option two a try, and see how it goes.

Anyway, more on this later.

Tape Playback Site

Sun, 05 May 2024 22:14:02 +1000

Thought I’d take a little break from UCL today.

Mum found a collection of old cassette tapes of us when we were kids, making and recording songs and radio shows. I’ve been digitising them over the last few weeks, and today the first recorded cassette was ready to share with the family.

I suppose I could’ve just given them raw MP3 files, but I wanted to record each cassette as two large files — one per side — so as to not loose much of the various crackles and clatters made when the tape recorder was stopped and started. But I did want to catalogue the more interesting points in the recording, and it would’ve been a bit “meh” simply giving them to others as one long list of timestamps (simulating the rewind/fast-forward seeking action would’ve been a step too far).

Plus, simply emailing MP3 files wasn’t nearly as interesting as what I did do, which was to put together a private site where others could browse and play the recorded tapes:

The landing page, listing the available tapes (of which there's only one right now.

Playback of a tape side, with chapter links for seeking.

The site is not much to talk about — it’s a Hugo site using the Mainroad theme and deployed to Netlify. There is some JavaScript that moves the playhead when a chapter link is clicked, but the rest is just HTML and CSS. But I did want to talk about how I got the audio files into Netlify. I wanted to use `git lfs` for this and have Netlify fetch them when building the site. Netlify doesn’t do this by default, and I get the sense that Netlify’s support for LFS is somewhat deprecated. Nevertheless, I gave it a try by adding an explicit `git lfs` step in the build to fetch the audio files. And it could’ve been that I was using the LFS command incorrectly, or maybe it was invoked at the wrong time. But whatever the reason, the command errored out and the audio files didn’t get pulled. I tried a few more times, and I probably could’ve got it working if I stuck with it, but all those deprecation warnings in Netlify’s documentation gave me pause.

So what I ended up doing was turning off builds in Netlify and using a Github Action which built the Hugo site and publish it to Netlify using the CLI tool. Here’s the Github Action in full:

name: Publish to Netify
on:
  push:
    branches: [main]
jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: true
          fetch-depth: 0    
          lfs: true
      - name: Setup Hugo
        uses: peaceiris/actions-hugo@v3
        with:
            hugo-version: '0.119.0'
      - name: Build Site
        run: |
          npm install
          hugo
      - name: Deploy
        env:
          NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
          NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
        run: |          
          netlify deploy --dir=public --prod

This ended up working quite well: the audio files made it to Netlify and were playable on the site. The builds are also quite fast; around 55 seconds (an earlier version involved building Hugo from source, which took 5 minutes). So for anyone else interested in trying to serve LFS files via Netlify, maybe try turning off the builds and going straight to using Github Action and the CLI tool. That is… if you can swallow the price of LFS storage in Github. Oof! A little pricy. Might be that I’ll need to use something else for the audio files.

UCL: Brief Integration Update and Modules

Sat, 04 May 2024 21:14:27 +1000

A brief update of where I am with UCL and integrating it into Dynamo-browse. I did managed to get it integrated, and it’s now serving as the interpreter of commands entered in during a session.

It works… okay. I decided to avoid all the complexities I mentioned in the last post — all that about continuations, etc. — and simply kept the commands returning tea.Msg values. The original idea was to have the commands return usable values if they were invoked in a non-interactive manner. For example, the table command invoked in an interactive session will bring up the table picker for the user to select the table. But when invoked as part of a call to another command, maybe it would return the current table name as a string, or something.

But I decided to ignore all that and simply kept the commands as they are. Maybe I’ll add support for this in a few commands down the line? We’ll see. I guess it depends on whether it’s necessary.

Which brings me up to why this is only working “okay” at the moment. Some commands return a tea.Msg which ask for some input from the user. The table command is one; another is set-attr, which prompts the user to enter an attribute value. These are implemented as a message which commands the UI to go into an “input mode”, and will invoke a callback on the message when the input is entered.

This is not an issue for single commands, but it becomes one when you start entering multiple commands that prompt for input, such as two set-attr calls:

set-attr this -S ; set-attr that -S

What happens is that two messages to show the prompt are sent, but only one of them is shown to the user, while the other is simply swallowed.

Fixing this would require some re-engineering, either with how the controllers returning these messages work, or the command handlers themselves. I can probably live with this limitation for now — other than this, the UCL integration is working well — but I may need to revisit this down the line.

Modules

As for UCL itself, I’ve started working on the builtins. I’m planning to have a small set of core builtins for the most common stuff, and the rest implemented in the form of “modules”. The idea is that the core will most likely be available all the time, but the modules can be turned on and off by the language embedder based on what they need or are comfortable having.

Each module is namespaces with a prefix, such as os for operating system operations, or fs for file-system operations. I’ve chosen the colon as the namespace separator, mainly so I can reserve the dot for field dereferencing, but also because I think TCL uses the colon as a namespace separator as well (I think I saw it in some sample code). The first implementation of this was simply adding the colon to the list of characters that make up the IDENT token. This broke the parser as the colon is also use as the map key/value separator, and the parser couldn’t resolve maps anymore. I had to extend the “indent” parse rule to support multiple IDENT tokens separated by colons. The module builtins are simply added to the environment with there fully-qualified name, complete prefix and colon, and invoking them with one of these idents will just “flatten” all these colon-separated tokens into a single string. Not sophisticated, but it’ll work for now.

There aren’t many builtins for these modules at the moment: just a few for reading environment variables and getting files as list of strings. Dynamo-browse is already using this in a feature branch, and it’s allows me to finally add a long-standing feature I’ve been meaning to add for a while: automatically enabling read-only mode when accessing DynamoDB tables in production. With modules, this construct looks a little like the following:

if (eq (os:env "ENV") "prod") {
    set-opt ro
}

It would’ve been possible to do this with the scripting language already used by Dynamo-browse. But this is the motivation of integrating UCL: it makes these sorts of constructs much easier to do, almost as one would do writing a shell-script over something in C.

UCL: Breaking And Continuation

Wed, 01 May 2024 08:01:36 +1000

I’ve started trying to integrate UCL into a second tool: Dynamo Browse. And so far it’s proving to be a little difficult. The problem is that this will be replacing a dumb string splitter, with command handlers that are currently returning a tea.Msg type that change the UI in some way.

UCL builtin handlers return a interface{} result, or an error result, so there’s no reason why this wouldn’t work. But tea.Msg is also an interface{} types, so it will be difficult to tell a UI message apart from a result that’s usable as data.

This is a Dynamo Browse problem, but it’s still a problem I’ll need to solve. It might be that I’ll need to return tea.Cmd types — which are functions returning tea.Msg — and have the UCL caller detect these and dispatch them when they’re returned. That’s a lot of function closures, but it might be the only way around this (well, the alternative is returning an interface type with a method that returns a tea.Msg, but that’ll mean a lot more types than I currently have).

Anyway, more on this in the future I’m sure.

Break, Continue, Return

As for language features, I realised that I never had anything to exit early from a loop or proc. So I added break, continue, and return commands. They’re pretty much what you’d expect, except that break can optionally return a value, which will be used as the resulting value of the foreach loop that contains it:

echo (foreach [5 4 3 2 1] { |n|
  echo $n
  if (eq $n 3) {
    break "abort"
  }
})
--> 5
--> 4
--> 3
--> abort

These are implemented as error types under the hood. For example, break will return an errBreak type, which will flow up the chain until it is handled by the foreach command (continue is also an errBreak with a flag indicating that it’s a continue). Similarly, return will return an errReturn type that is handled by the proc object.

This fits quite naturally with how the scripts are run. All I’m doing is walking the tree, calling each AST node as a separate function call and expecting it to return a result or an error. If an error is return, the function bails, effectively unrolling the stack until the error is handled or it’s returned as part of the call to Eval(). So leveraging this stack unroll process already in place makes sense to me.

I’m not sure if this is considered idiomatic Go. I get the impression that using error types to handle flow control outside of adverse conditions is frowned upon. This reminds me of all the arguments against using expressions for flow control in Java. Those arguments are good ones: following executions between try and catch makes little sense when the flow can be explained more clearly with an if.

But I’m going to defend my use of errors here. Like most Go projects, the code is already littered with all the if err != nil { return err } to exit early when a non-nil error is returned. And since Go developers preach the idea of errors simply being values, why not use errors here to unroll the stack? It’s better than the alternatives: such as detecting a sentinel result type or adding a third return value which will just be yet another if bla { return res } clause.

Continuations

Now, an idea is brewing for a feature I’m calling “continuations” that might be quite difficult to implement. I’d like to provide a way for a user builtin to take a snapshot of the call stack, and resume execution from that point at a later time.

The reason for this is that I’d like all the asynchronous operations to be transparent to the UCL user. Consider a UCL script with a sleep command:

echo "Wait here"
sleep 5
echo "Ok, ready"

sleep could simply be a call to time.Sleep() but say you’re running this as part of an event loop, and you’d prefer to do something like setup a timer instead of blocking the thread. You may want to hide this from the UCL script author, so they don’t need to worry about callbacks.

Ideally, this can be implemented by the builtin using a construct similar to the following:

func sleep(ctx context.Context, arg ucl.CallArgs) (any, error) {
  var secs int
  if err := arg.Bind(&secs); err != nil {
    return err
  }

  // Save the execution stack
  continuation := args.Continuation()

  // Schedule the sleep callback
  go func() {
    <- time.After(secs * time.Seconds)

    // Resume execution later, yielding `secs` as the return value
    // of the `sleep` call. This will run the "ok, ready" echo call
    continuation(ctx, secs)
  })()

  // Halt execution now
  return nil, ucl.ErrHalt
}

The only trouble is, I’ve got no idea how I’m going to do this. As mentioned above, UCL executes the script by walking the parse tree with normal Go function calls. I don’t want to be in a position to create a snapshot of the Go call stack. That a little too low level for what I want to achieve here.

I suppose I could store the visited nodes in a list when the ErrHalt is raised; or maybe replace the Go call stack with an in memory stack, with AST node handlers being pushed and popped as the script runs. But I’m not sure this will work either. It would require a significant amount of reengineering, which I’m sure will be technically interesting, but will take a fair bit of time. And how is this to work if a continuation is made in a builtin that’s being called from another builtin? What should happen if I were to run sleep within a map, for example?

So it might be that I’ll have to use something else here. I could potentially do something using Goroutines: the script is executed on Goroutine and args.Continuation() does something like pauses it on a channel. How that would work with a builtin handler requesting the continuation not being paused themselves I’m not so sure. Maybe the handlers could be dispatched on a separate Goroutine as well?

A simpler approach might be to just offload this to the UCL user, and have them run Eval on a separate Goroutine and simply sleeping the thread. Callbacks that need input from outside could simply be sent using channels passed via the context.Context. At least that’ll lean into Go’s first party support for synchronisation, which is arguably a good thing.

UCL: The Simplifications Paid Off

Fri, 26 Apr 2024 11:13:49 +1000

The UCL simplifications have been implemented, and they seem to be largely successful.

Ripped out all the streaming types, and changed pipes to simply pass the result of the left command as first argument of the right.

"Hello" | echo ", world"
--> "Hello, world"

This has dramatically improved the use of pipes. Previously, pipes could only be used to connect streams. But now, with pretty much anything flowing through a pipe, that list of commands has extended to pretty much every builtins and user-defined procs. Furthermore, a command no longer needs to know that it’s being used in a pipeline: whatever flows through the pipe is passed transparently via the first argument to the function call. This has made pipes more useful, and usable in more situations.

Macros can still know whether there exist a pipe argument, which can make for some interesting constructs. Consider this variant of the foreach macro, which can “hang off” the end of a pipe:

["1" "2" "3"] | foreach { |x| echo $x }
--> 1
--> 2
--> 3

Not sure if this variant is useful, but I think it could be. It seems like a natural way to iterate items passed through the pipe. I’m wondering if this could extend to the if macro as well, but that variant might not be as natural to read.

Another simplification was changing the map builtin to accept anonymous blocks, as well as an “invokable” commands by name. Naturally, this also works with pipes too:

[a b c] | map { |x| toUpper $x }
--> [A B C]

[a b c] | map toUpper
--> [A B C]

As for other language features, I finally got around to adding support for integer literals. They look pretty much how you expect:

set n 123
echo $n
--> 123

One side effect of this is that an identifier can no longer start with a dash followed by a digit, as that would be parsed as the start of a negative integer. This probably isn’t a huge deal, but it could affect command switches, which are essentially just identifiers that start with a dash.

Most of the other work done was behind the scenes trying to make UCL easier to embed. I added the notion of “listable” and “hashable” proxies objects, which allow the UCL user to treat a Go slice or a Go struct as a list or hash respectively, without the embedder doing anything other than return them from a function (I’ve yet to add this support to maps just yet).

A lot of the native API is still a huge mess, and I really need to tidy it up before I’d be comfortable opening the source. Given that the language is pretty featureful now to be useful, I’ll probably start working on this next. Plus adding builtins. Really need to start adding useful builtins.

Anyway, more to come on this topic I’m sure.

Oh, one last thing: I’ve put together an online playground where you can try the language out in the browser. It’s basically a WASM build of the language running in a JavaScript terminal emulator. It was a little bit of a rush job and there’s no reason for building this other than it being a fun little thing to do.

You can try it out here, if you’re curious.

Simplifying UCL

Tue, 23 Apr 2024 08:21:44 +1000

I’ve been using UCL for several days now in that work tool I mentioned, and I’m wondering if the technical challenge that comes of making a featureful language is crowding out what I set out to do: making a useful command language that is easy to embed.

So I’m thinking of making some simplifications.

The first is to expand the possible use of pipes. To date, the only thing that can travel through pipes are streams. But many of the commands I’ve been adding simply return slices. This is probably because there’s currently no “stream” type available to the embedder, but even if there was, I’m wondering if it make sense to allow the embedder to pass slices, and other types, through pipes as well.

So, I think I’m going to take a page out of Go’s template book and simply have pipes act as syntactic sugar over sequential calls. The goal is to make the construct a | b essentially be the same as b (a), where the first argument of b will be the result of a.

As for streams, I’m thinking of removing them as a dedicated object type. Embedders could certainly make analogous types if they need to, and the language should support that, but the language will no longer offer first class support for them out of the box.

The second is to remove any sense of “purity” of the builtins. You may recall the indecision I had regarding using anonymous procs with the map command:

I’m not sure how I can improve this. I don’t really want to add automatic dereferencing of identities: they’re very useful as unquoted string arguments. I suppose I could add another construct that would support dereferencing, maybe by enclosing the identifier in parenthesis.

I think this is the wrong way to think of this. Again, I’m not here to design a pure implementation of the language. The language is meant to be easy to use, first and foremost, in an interactive shell, and if that means sacrificing purity for a map command that supports blocks, anonymous procs, and automatic dereferencing of commands just to make it easier for the user, then I think that’s a trade work taking.

Anyway, that’s the current thinking as of now.

Imports And The New Model

Sat, 20 Apr 2024 15:53:06 +1000

Well, I dragged Photo Bucket out today to work on it a bit.

It’s fallen by the wayside a little, and I’ve been wondering if it’s worth continuing work on it. So many things about it that need to be looked at: the public site looks ugly, as does the admin section; working with more than a single image is a pain; backup and restore needs to be added; etc.

I guess every project goes through this “trough of discontent” where the initial excitement has warn off and all you see is a huge laundry list of things to do. Not to mention the wandering eye looking at the alternatives.

But I do have to stop myself from completely junking it, since it’s actually being used to host the Folio Red Gallery. I guess my push to deploy it has entrapped me (well, that was the idea of pushing it out there in the first place).

Anyway, it’s been a while (last update is here) and the move to the new model is progressing. And it’s occurred to me that I haven’t actually talked about the new model (well, maybe I have but I forgot about it).

Previously, the root model of this data structure is the Store. All images belong to a Store, which is responsible for managing the physical storage and retrieval of them. These stores can have sub-stores, which are usually used to hold the images optimised for a specific use (serving on the web, showing as a thumbnails, etc). Separate to this was the public site Design which handed properties of the public site: how it should look, what the title, and description is, etc.

The "Store" model

There were some serious issues with this approach: images were owned by stores, and two images can belong to two different stores, but they all belonged to the same site. This made uploading confusing: which store should the image live on? I worked around this by adding the notion of a “primary store” but this was just ignoring the problem and defeated the whole multiple-store approach.

This is made even worse when one considers which store to use for serving the images. Down the line I was hoping to support virtual domain hosting, where one could setup different image sites on different domains that all pointed to the same instance. So imagine how that would work: one wanted to view images from alpha.example.com and another wanted to view images from beta.example.com. Should the domains live on the store? What about the site designs? Where should they live?

The result was that this model could only really ever support one site per Photo Bucket instance, requiring multiple deployments for different sites if one wanted to use a single host for separate photo sites.

So I re-engineered the model to simplify this dramatically. Now, the route object is the Site:

The "Site" model

Here, the Site owns everything. The images are associated with sites, not stores. Stores still exist, but their role is now more in-line with what the sub-stores did. When an image is uploaded, it is stored in every Store of the site, and each Store will be responsible for optimising it for a specific use-case. The logic used to determine which Store to use to fetch the image is still in place but now it can be assumed that any Store associated with a site will have the image.

Now the question of which Store an image should be added to is easy: all the them.

Non-image data, such as Galleries and Designs now live off the Site as well, and if virtual hosting is added, so would the domain that serves that Site.

At least one site needs to be present at all time, and it’s likely most instances will simply have a single Site for now. But this assumption solves the upload and hosting resolution issues listed above. And if multiple site support is needed, a simple site picker can be added to the admin page (the public pages can will rely on the request hostname).

This has been added a while ago, and as of today, has been merged to main. But I didn’t want to deal with writing the data migration logic for this, so my plan is to simply junk the existing instance and replace it with the brand new one. But in order to do so, I needed to export the photos from the old instance, and import them into the new one.

The export logic has been deployed and I’ve made an export it this morning. Today, the import logic was finished and merged. Nothing fancy: like the export it’s only invokable from the command line. But it’ll do the job for now.

Next steps is to actually deploy this, which I guess will be the ultimate test. Then, I’m hoping to add support for galleries in the public page so I can separate images on the Folio Red Gallery into projects. There’s still no way to add images in bulk to a gallery. Maybe this will give me an incentive to do that next.

UCL: Procs and Higher-Order Functions

Fri, 19 Apr 2024 16:10:59 +1000

More on UCL yesterday evening. Biggest change is the introduction of user functions, called “procs” (same name used in TCL):

proc greet {
    echo "Hello, world"
}

greet
--> Hello, world

Naturally, like most languages, these can accept arguments, which use the same block variable binding as the foreach loop:

proc greet { |what|
    echo "Hello, " $what
}

greet "moon"
--> Hello, moon

The name is also optional, and if omitted, will actually make the function anonymous. This allows functions to be set as variable values, and also be returned as results from other functions.

proc makeGreeter { |greeting|
    proc { |what|
        echo $greeting ", " $what
    }
}

set helloGreater (makeGreeter "Hello")
call $helloGreater "world"
--> Hello, world

set goodbye (makeGreeter "Goodbye cruel")
call $goodbye "world"
--> Goodbye cruel, world

I’ve added procs as a separate object type. At first glance, this may seem a little unnecessary. After all, aren’t blocks already a specific object type?

Well, yes, that’s true, but there are some differences between a proc and a regular block. The big one being that the proc will have a defined scope. Blocks adapt to the scope to which they’re invoked whereas a proc will close over and include the scope to which it was defined, a lot like closures in other languages.

It’s not a perfect implementation at this stage, since the set command only sets variables within the immediate scope. This means that modifying closed over variables is currently not supported:

\# This currently won't work
proc makeSetter {
    set bla "Hello, "
    proc appendToBla { |x|
        set bla (cat $bla $x)
        echo $bla
    }
}

set er (makeSetter)
call $er "world"
\# should be "Hello, world"

Higher-Order Functions

The next bit of work is finding out how best to invoke these procs in higher-order functions. There are some challenges here that deal with the language grammar.

Invoking a proc by name is fine, but since the grammar required the first token to be a command name, there was no way to invoke a proc stored in a variable. I quickly added a new call command — which takes the proc as the first argument — to work around it, but after a while, this got a little unwieldy to use (you can see it in the code sample above).

So I decided to modify the grammar to allow any arbitrary value to be the first token. If it’s a variable that is bound to something “invokable” (i.e. a proc), and there exist at-least one other argument, it will be invoked. So the above can be written as follows:

set helloGreater (makeGreeter "Hello")
$helloGreater "world"
--> Hello, world

At-least one argument is required, otherwise the value will simply be returned. This is so that the value of variables and literal can be returned as is, but that does mean lambdas will simply be dereferenced:

"just, this"
--> just, this

set foo "bar"
$foo
--> bar

set bam (proc { echo "BAM!" })
$bam
--> (proc)

To get around this, I’ve added the notion of the “empty sub”, which is just the construct (). It evaluates to nil, and since a function ignores any extra arguments not bound to variables, it allows for calling a lambda that takes no arguments:

set bam (proc { echo "BAM!" })
$bam ()
--> BAM!

It does allow for other niceties, such as using a falsey value:

if () { echo "True" } else { echo "False" }
--> False

With lambdas now in place, I’m hoping to work on some higher order functions. I’ve started working on map which accepts both a list or a stream. It’s a buggy mess at the moment, but some basic constructs currently work:

map ["a" "b" "c"] (proc { |x| toUpper $x }) 
--> stream ["A" "B" "C"]

(Oh, by the way, when setting a variable to a stream using set, it will now collect the items as a list. Or at least that’s the idea. It’s currently not working at the moment.)

A more refined approach would be to treat commands as lambdas. The grammar supports this, but the evaluator doesn’t. For example, you cannot write the following:

\# won't work
map ["a" "b" "c"] toUpper

This is because makeUpper will be treated as a string, and not a reference to an invokable command. It will work for variables. You can do this:

set makeUpper (proc { |x| toUpper $x })
map ["a" "b" "c"] $makeUpper

I’m not sure how I can improve this. I don’t really want to add automatic dereferencing of identities: they’re very useful as unquoted string arguments. I suppose I could add another construct that would support dereferencing, maybe by enclosing the identifier in parenthesis:

\# might work?
map ["a" "b" "c"] (toUpper)

Anyway, more on this in the future I’m sure.

UCL: First Embed, and Optional Arguments

Thu, 18 Apr 2024 08:39:12 +1000

Came up with a name: Universal Control Language: UCL. See, you have TCL; but what if instead of being used for tools, it can be more universal? Sounds so much more… universal, am I right? 😀

Yeah, okay. It’s not a great name. But it’ll do for now.

Anyway, I’ve started integrating this language with the admin tool I’m using at work. This tool I use is the impetus for this whole endeavour. Up until now, this tool was just a standard CLI command usable from the shell. But it’s not uncommon for me to have to invoke the tool multiple times in quick succession, and each time I invoke it, it needs to connect to backend systems, which can take a few seconds. Hence the reason why I’m converting it into a REPL.

Anyway, I added UCL to the tool, along with a readline library, and wow, did it feel good to use. So much better than the simple quote-aware string splitter I’d would’ve used. And just after I added it, I got a flurry of requests from my boss to gather some information, and although the language couldn’t quite handle the task due to missing or unfinished features, I can definitely see the potential there.

I’m trying my best to only use what will eventually be the public API to add the tool-specific bindings. The biggest issue is that these “user bindings” (i.e. the non-builtins) desperately need support for producing and consuming streams. They’re currently producing Go slices, which are being passed around as opaque “proxy objects”, but these can’t be piped into other commands to, say, filter or map. Some other major limitations:

No commands to actually filter or map. In fact, the whole standard library needs to be built out.
No ability to get fields from hashes or lists, including proxy objects which can act as lists or hashes.

One last thing that would be nice is the ability to define optional arguments. I actually started work on that last night, seeing that it’s relatively easy to build. I’m opting for a style that looks like the switches you’d find on the command line, with option names starting with dashes:

join "a" "b" -separator "," -reverse
--> b, a

Each option can have zero or more arguments, and boolean options can be represented as just having the switch. This does mean that they’d have to come after the positional arguments, but I think I can live with that. Oh, and no syntactic sugar for single-character options: each option must be separated by whitespace (the grammar actually treats them as identifiers). In fact, I’d like to discourage the use of single-character option names for these: I prefer the clarity that comes from having the name written out in full (that said, I wouldn’t rule out support for aliases). This eliminates the need for double dashes, to distinguish long option names from a cluster of single-character options, so only the single dash will be used.

I’ll talk more about how the Go bindings look later, after I’ve used them a little more and they’re a little more refined.

Tool Command Language: Lists, Hashs, and Loops

Wed, 17 Apr 2024 21:04:22 +1000

A bit more on TCL (yes, yes, I’ve gotta change the name) last night. Added both lists and hashes to the language. These can be created using a literal syntax, which looks pretty much looks how I described it a few days ago:

set list ["a" "b" "c"]
set hash ["a":"1" "b":"2" "c":"3"]

I had a bit of trouble working out the grammar for this, I first went with something that looked a little like the following, where the key of an element is optional but the value is mandatory:

list_or_hash  --> "[" "]"        \# empty list
                | "[" ":" "]"    \# empty hash
                | "[" elems "]"  \# elements

elems --> ((arg ":")? arg)*      \# elements of a list or hash

arg --> <anything that can be a command argument>

But I think this confused the parser a little, where it was greedily consuming the key arg and expecting the : to be present to consume the value.

So I flipped it around, and now the “value” is the optional part:

elems --> (arg (":" arg)?)*

So far this seems to work. I renamed the two fields “left” and “right”, instead of key and value. Now a list element will use the “left” part, and a hash element will use “left” for the key and “right” for the value.

You can probably guess that the list and hash are sharing the same AST types. This technically means that hybrid lists are supported, at least in the grammar. But I’m making sure that the evaluator throws an error when a hybrid is detected. I prefer to be strict here, as I don’t want to think about how best to support it. Better to just say either a “pure” list, or a “pure” hybrid.

Well, now that we have collections, we need some way to iterate over them. For that, I’ve added a foreach loop, which looks a bit like the following:

\# Over lists
foreach ["a" "b" "c"] { |elem|
  echo $elem
}

\# Over hashes
foreach ["a":"1" "b":"2"] { |key val|
  echo $key " = " $val
}

What I like about this is that, much like the if statement, it’s implemented as a macro. It takes a value to iterate over, and a block with bindable variables: one for list elements, or two for hash keys and values. This does mean that, unlike most other languages, the loop variable appears within the block, rather than to the left of the element, but after getting use to this form of block from my Ruby days, I can get use to it.

One fun thing about hashes is that they’re implemented using Go’s map type. This means that the iteration order is random, by design. This does make testing a little difficult (I’ve only got one at the moment, which features a hash of length one) but I rarely depend on the order of hash keys so I’m happy to keep it as is.

This loop is only the barest of bones at the moment. It doesn’t support flow control like break or continue, and it also needs to support streams (I’m considering a version with just the block that will accept the stream from a pipe). But I think it’s a reasonably good start.

I also spend some time today integrating this language in the tool I was building it for. I won’t talk about it here, but already it’s showing quite a bit of promise. I think, once the features are fully baked, that this would be a nice command language to keep in my tool-chest. But more of that in a later post.

Backlog Proc: A Better JQL

Mon, 15 Apr 2024 10:47:39 +1000

Backlog Proc is a simple item backlog tracker I built for work. I’d like to link them to Jira tickets, so that I know whether a particular backlog item actually has tasks written for them, and what the status of each of those tasks are. I guess these are meant to be tracked by epics, but Jira’s UI for handling such things is a mess, and I’d like to make notes that are only for my own eyes.

Anyway, I’m was using JQL to select the Jira tickets. And it worked, but the language is a bit verbose. Plus the tool I’m running the queries in, jira-cli, requires that I add the project ID along with the things like the epic or fix version.

So I’m started working on a simpler language, one that’s just a tokenised list of ticket numbers. For example, instead of writing:

project = "ABC" and key in (ABC-123 ABC-234 ABC-345)

One could just write:

ABC-123 ABC-234 ABC-345

And instead of writing:

(project = "ABC" and epicLink = "ABC-818") OR (project = "DEF" and epicLink = "DEF-222")

One could just write:

epic:(ABC-818, DEF-222)

(note here the use of OR, in that the sets are unionised; I’m not sure how this would scale for the other constructs).

Key characteristics is that the parser would be able to get the project ID from the query, instead of having the query writer (i.e. me) explicitly add it.

I can also do introspection, such as get the relevant projects, by “unparsing” the query. An advantage of controlling the parser and language. Can’t do that with JQL.

But, of-course, I can’t cover all possible bases with this language just yet, so I’ll need a way to include arbitrary JQL.. So I’ve also added a general “escape” clause to do this:

jql:"project in (bla)"

A Few Other Things

A few other things that is needed for Backlog Proc:

The landing screen needs to show all the active items regardless of what project they belong to. Bouncing around projects is a bit of a hassle (to be fair, I haven’t spent any time styling the screens beyond what Buffalo generated).
Add back ticket drafts. I enjoyed the flow of drafting all the tickets in one hit, then clicking “Publish” and seeing them come out as a fully specified Jira epic.
Add the notion of logs, with a “log type”. Such types can include things like “todo” (i.e thing for me to work on), “decision” (a product decision that was agreed on), and “log” (work that was done).
A way to quickly view the tickets in Jira so I can do batch operations over them.

Tool Command Language: Macros And Blocks

Sat, 13 Apr 2024 16:44:15 +1000

More work on the tool command language (of which I need to come up with a name: I can’t use the abbreviation TCL), this time working on getting multi-line statement blocks working. As in:

echo "Here"
echo "There"

I got a little wrapped up about how I can configure the parser to recognise new-lines as statement separators. I tried this in the past with a hand rolled lexer and ended up peppering NL tokens all around the grammar. I was fearing that I needed to do something like this here. After a bit of experimentation, I think I’ve come up with a way to recognise new-lines as statement separators without making the grammar too messy. The unit tests verifying this so far seem to work.

// Excerpt of the grammar showing all the 'NL' token matches.
// These match a new-line, plus any whitespace afterwards.

type astStatements struct {
    First *astPipeline   `parser:"@@"`
    Rest  []*astPipeline `parser:"( NL+ @@ )*"`
}

type astBlock struct {
    Statements []*astStatements `parser:"LC NL? @@ NL? RC"`
}

type astScript struct {
    Statements *astStatements `parser:"NL* @@ NL*"`
}

I’m still using a stateful lexer as it may come in handy when it comes to string interpolation. Not sure if I’ll add this, but I’d like the option.

Another big addition today was macros. These are much like commands, but instead of arguments being evaluated before being passed through to the command, they’re deferred and the command can explicitly request their evaluation whenever. I think Lisp has something similar: this is not that novel.

This was used to implement the if command, which is now working:

set x "true"
if $x {
  echo "Is true"
} else {
  echo "Is not true"
}

Of course, there are actually no operators yet, so it doesn’t really do much at the moment.

This spurred the need for blocks. which is a third large addition made today. They’re just a group of statements that are wrapped in an object type. They’re “invokable” in that the statements can be executed and produce a result, but they’re also a value that can be passed around. It jells nicely with the macro approach.

Must say that I like the idea of using macros for things like if over baking it into the language. It can only add to the “embed-ability” of this, which is what I’m looking for.

Finally, I did see something interesting in the tests. I was trying the following test:

echo "Hello"
echo "World"

And I was expecting a Hello and World to be returned over two lines. But only World was being returning. Of course! Since echo is actually producing a stream and not printing anything to stdout, it would only return World.

I decided to change this. If I want to use echo to display a message, then the above script should display both Hello and World in some manner. The downside is that I don’t think I’ll be able to support constructs like this, where echo provides a source for a pipeline:

\# This can't work anymore
echo "Hello" | toUpper

I mean, I could probably detect whether echo is connected to a pipe (the parser can give that information). But what about other commands that output something? Would they need to be treated similarly?

I think it’s probably best to leave this out for now, and have a new construct for providing literals like this to a pipe. Heck, maybe just having the string itself would be enough:

"hello" | toUpper

Anyway, that’s all for today.

Tool Command Language

Fri, 12 Apr 2024 07:45:30 +1000

I have this idea for a tool command language. Something similar to TCL, in that it’s chiefly designed to be used as an embedded scripting language and chiefly in an interactive context.

It’s been an idea I’ve been having in my mind for a while, but I’ve got the perfect use case for it. I’ve got a tool at work I use to do occasional admin tasks. At the moment it’s implemented as a CLI tool, and it works. But the biggest downside is that it needs to form connections to the cluster to call internal service methods, and it always take a few seconds to do so. I’d like to be able to use it to automate certain actions, but this delay would make doing so a real hassle.

Some other properties that I’m thinking off:

It should be able to support structured data, similar to how Lisp works
It should be able to support something similar to pipes, similar to how the shell and Go’s template language works.

Some of the trade-offs that come of it:

It doesn’t have to be fast. In fact, it can be slow so long as the work embedding and operating it can be fast.
It may not be completely featureful. I’ll go over the features I’m thinking of below, but I say upfront that you’re not going to be building any cloud services with this. Administering cloud servers, maybe; but leave the real programs to a real language.

Some Notes On The Design

The basic concept is the statement. A statement consists of a command, and zero or more arguments. If you’ve used a shell before, then you can imagine how this’ll look:

firstarg "hello, world"
--> hello, world

Each statement produces a result. Here, the theoretical firstarg will return the first argument it receives, which will be the string "hello, world"

Statements are separated by new-lines or semicolons. In such a sequence, the return value of the last argument is returned:

firstarg "hello" ; firstarg "world"
--> world

I’m hoping to have a similar approach to how Go works, in that semicolons will be needed if multiple statements share a line, but will otherwise be unnecessary. I’m using the Participal parser library for this, and I’ll need to know how I can configure the scanner to do this (or even if using the scanner is the right way to go).

The return value of statements can be used as the arguments of other statements by wrapping them in parenthesis:

echo (firstarg "hello") " world"
--> hello world

This is taken directly from TCL, except that TCL uses the square brackets. I’m reserving the square brackets for data structures, but the parenthesis are free. It also gives it a bit of a Lisp feel.

Pipelines

Another way for commands to consume the output of other commands is to build pipelines. This is done using the pipe | character:

echo "hello" | toUpper
--> HELLO

Pipeline sources, that is the command on the left-most side, can be either commands that produce a single result, or a command that produces a “stream”. Both are objects, and there’s nothing inherently special about a stream, other than there some handling when used as a pipeline. Streams are also designed to be consumed once.

For example, one can consider a command which can read a file and produce a stream of the contents:

cat "taleOfTwoCities.txt"
--> It was the best of times,
--> it was the worst of times,
--> …

Not every command is “pipe savvy”. For example, piping the result of a pipeline to echo will discard it:

echo "hello" | toUpper | echo "no me"
--> no me

Of course, this may differ based on how the builtins are implemented.

Variables

Variables are treated much like TCL and shell, in that referencing them is done using the dollar sign:

set name "josh"
--> "Josh"
echo "My name is " $name
--> "My name is Josh"

Not sure how streams will be handled with variables but I’m wondering if they should be condensed down to a list. I don’t like the idea of assigning a stream to a variable, as streams are only consumed once, and I feel like some confusion will come of it if I were to allow this.

Maybe I can take the Perl approach and use a different variable “context”, where you have a variable with a @ prefix which will reference a stream.

set file (cat "eg.text")

echo @file
\# Echo will consume file as a stream

echo $file
\# Echo will consume file as a list

The difference is subtle but may be useful. I’ll look out for instances where this would be used.

Attempting to reference an unset variable will result in an error. This may also change.

Other Ideas

That’s pretty much what I have at the moment. I do have some other ideas, which I’ll document below.

Structured Data Support: Think lists and hashes. This language is to be used with structured data, so I think it’s important that the language supports this natively. This is unlike TCL which principally works with strings and the notion of lists feels a bit tacked on to some extent.

Both lists and hashes are created using square brackets:

\# Lists. Not sure if they'll have commas or not
set l [1 2 3 $four (echo "5")]

\# Maps
set m [a:1 "b":2 "see":(echo "3") (echo "dee"):$four]

Blocks: Yep, containers for a groups of statements. This will be used for control flow, as well as for definition of functions:

set x 4
if (eq $x 4) {
  echo "X == 4"
} else {
  echo "X != 4"
}

foreach [1 2 3] { |x|
  echo $x
}

Here the blocks are just another object type, like strings and stream, and both if and foreach are regular commands which will accept a block as an argument. In fact, it would be theoretically possible to write an if statement this way (not sure if I’ll allow setting variables to blocks):

set thenPart {
  echo "X == 4"
}

if (eq $x 4) $thenPart

The block execution will exist in a context, which will control whether a new stack frame will be used. Here the if statement will simply use the existing frame, but a block used in a new function can push a new frame, with a new set of variables:

proc myMethod { |x|
  echo $x
}

myMethod "Hello"
--> "Hello

Also note the use of |x| at the start of the block. This is used to declare bindable variables, such as function arguments or for loop variables. This will be defined as part of the grammar, and be a property of the block.

Anyway, that’s the current idea.

Photo Bucket Update: Exporting To Zip

Sat, 09 Mar 2024 09:33:57 +1000

Worked a little more on Photo Bucket this week. Added the ability to export the contents of an instance to a Zip file. This consist of both images and metadata.

I’ve went with lines of JSON file for the image metadata. I considered a CSV file briefly, but for optional fields like captions and custom properties, I didn’t like the idea of a lot of empty columns. Better to go with a format that’s a little more flexible, even if it does mean more text per line.

As for the images, I’m hoping the export to consist of the “best quality” version. What that means will depend on the instance. The idea is to have three tiers of image quality managed by the store: “original”, “web”, and “thumbnail”. The “original” version is the untouched version uploaded to the store. The “web” version is re-encoded from the “original” and will be slightly compressed with image metadata tags stripped out. The “thumbnail” version will be a small, highly compressed version suitable for the thumbnail. There is to be a decision algorithm in place to get an image given the desired quality level. For example, if something needed the “best quality” version of an image, and the “original” image is not available, the service will default to the “web” version (the idea is that some of these tiers will be optional depending on the need of the instances).

This is all partially working at the moment, and I’m hoping to rework all this when I replace how stores and images relate to each other (This is what I’m starting on now, and why I built export now since this will be backwards incompatible). So for the moment the export simply consists of the “web” version.

I’ve got unit tests working for this as well. I’m trying a new approach for unit testing in this project. Instead of using mocks, the tests are actually running against a fully instantiated versions of the services. There exists a servicestest package which does all the setup (using temporary directories, etc) and tear down of these services. Each individual unit test gets the services from this package and will run tests against a particular one.

This does mean all the services are available and exercised within the tests, making them less like unit tests and more like integrations tests. But I think I prefer this approach. The fact that the dependent services are covered gives me greater confidence that they’re working. It also means I can move things around without changing mocks or touching the tests.

That’s not to say that I’m not trying to keep each service their own component as much as I can. I’m still trying to follow best practice of component design: passing dependencies in explicitly when the services are created, for example. But setting them all up as a whole in the tests means I can exercise them while they serve the component being tested. And the dependencies are explicit anyway (i.e. no interfaces) so it makes sense keeping it that way for the tests as well. And it’s just easier anyway. 🤷

Anyway, starting rework on images and stores now. Will talk more about this once it’s done.

Photo Bucket Update: More On Galleries

Tue, 05 Mar 2024 21:43:05 +1000

Spent a bit more time working on Photo Bucket this last week¹, particularly around galleries. They’re progressing quite well. I’m made some strides in getting two big parts of the UI working now: adding and removing images to galleries, and re-ordering gallery items via drag and drop.

I’ll talk about re-ordering first. This was when I had to bite the bullet and start coding up some JavaScript. Usually I’d turn to Stimulus for this but I wanted to give HTML web components a try. And so far, they’ve been working quite well.

The gallery page is generated server-side into the following HTML:

<main>
  <pb-draggable-imageset href="/_admin/galleries/1/items" class="image-grid">
    <pb-draggable-image position="0" item-id="7">
      <a href="/_admin/photos/3">
        <img src="/_admin/img/web/3">
      </a>
    </pb-draggable-image>
        
    <pb-draggable-image position="1" item-id="4">
      <a href="/_admin/photos/4">
        <img src="/_admin/img/web/4">
      </a>
    </pb-draggable-image>
        
    <pb-draggable-image position="2" item-id="8">
      <a href="/_admin/photos/1">
        <img src="/_admin/img/web/1">
      </a>
    </pb-draggable-image>        
  </pb-draggable-imageset>
</main>

Each <pb-draggable-image> node is a direct child of an <pb-draggable-imageset>. The idea is that the user can rearrange any of the <pb-draggable-image> elements within a single <pb-draggable-imageset> amongst themselves. Once the user has moved an image onto to another one, the image will signal its new position by firing a custom event. The containing <pb-draggable-imageset> element is listening to this event and will respond by actually repositioning the child element and sending a JSON message to the backend to perform the move in the database.

A lot of this was based on the MDN documentation for drag and drop and it follows the examples quite closely. I did find a few interesting things though. My first attempt at this was to put it onto the <pb-draggable-image> element, but I wasn’t able to get any drop events when I did. Moving the draggable attribute onto the <a> element seemed to work. I not quite sure why this is. Surely I can’t think of any reason as to why it wouldn’t work. It may had something else, such as how I was initialising the HTTP components.

Speaking of HTML components, there was a time where the custom component’s connectedCallback method was being called before the child <a> elements were present in the DOM. This was because I had the <script> tag in the the HTML head and configured to be evaluated during parsing. Moving it to the end of the body and loading it as a module fixed that issue. Also I found that moving elements around using element.before and element.after would actually call connectedCallback and disconnectedCallback each time, meaning that any event listeners registered within connectedCallback would need to be de-registered, otherwise events would be handled multiple times. This book-keeping was slightly annoying, but it worked.

Finally, there was moving the items with the database. I’m not sure how best to handle this, but I have that method that seems to work. What I’m doing is tracking the position of each “gallery item” using a position field. This field would be 1 for the first item, 2 for the next, and so on for each item in the gallery. The result of fetching items would just order using this field, so as long as they’re distinct, they don’t need to be a sequence incrementing by 1, but I wanted to keep this as much as possible.

The actual move involves two update queries. The first one will update the positions of all the items that are to shift left or right by one to “fill the gap”. The way it does this is that when an item is moved from position X to position Y, the value of position between X and Y would be changed by +1 if X > Y, or by –1 if Y > X. This is effectively the same as setting position X to X + 1, and so on, but done using one UPDATE statement. The second query just sets the position of item X to Y.

So that’s moving gallery items. I’m not sure how confident I am with this approach, but I’ve been testing this, both manually and by writing unit tests. It’s not quite perfect yet: I’m still finding bugs (I found some while coming up with these screencasts). Hopefully, I’ll be able to get to the bottom of them soon.

The second bit of work was to actually add and remove images in the gallery themselves. This, for the moment, is done using a “gallery picker” which is available in the image details. Clicking “Gallery” while viewing an image will show the list of galleries in the system, with toggles on the left. The galleries an image already belongs to is enabled, and the user can choose the galleries they want the image to be in by switching the toggles on and off. These translate to inserts and remove statements behind the scenes.

The toggles are essentially just HTML and CSS, and a bulk of the code was taken from this example, with some tweaks. They look good, but I think I may need to make them slightly smaller for mouse and keyboard.

I do see some downside with this interaction. First, it reverses the traditional idea of adding images to a gallery: instead of doing that, your selecting galleries for an image. I’m not sure if this would be confusing for others (it is modelled on how Google Photos works). Plus, there’s no real way to add images in bulk. Might be that I’ll need to add a way to select images from the “Photos” section and have a dialog like this to add or remove them all from a gallery. I think this would go far in solving both of these issues.

So that’s where things are. Not sure what I’ll work on next, but it may actually be import and export, and the only reason for this is that I screwed up the base model and will need to make some breaking changes to the DB schema. And I want to have a version of export that’s compatible with the original schema that I can deploy to the one and only production instance of Photo Bucket so that I can port the images and captions over to the new schema. More on this in the future, I’m sure.

Apparently I’m more than happy to discuss work in progress, yet when it comes to talking about something I’ve finished, I freeze up. 🤷 ↩︎

Photo Bucket Galleries and Using the HTML Popover API

Mon, 19 Feb 2024 22:05:48 +1000

Spent a bit more on Photo Bucket this evening. Tonight I started working on galleries, which’ll work more or less like albums.

At the moment you can create a gallery and add a photo to it. Most of the work so far has been backend so the UI is pretty rough. Eventually you’ll be able to do things like rearrange photos within galleries, but for the moment they’ll just be added to the end.

I did need to add some modals to the UI, such as the New Gallery model that’s shown for the gallery name. This gave me the opportunity to try out the new popover API. And yeah, it does exactly what it says on the tin: add the popover attribute to an element and it becomes a working popover (at least in the latest version of Vivaldi). Must say it’s impressive that this is now possible with HTML alone.

The initial version of the modals used a <div> for the popover target. And while that worked, there were some small annoyances. First was that the form within the popover didn’t get focus when the popover was displayed. It would be nice to click “New” and start typing out the gallery name. But this is a small thing that’s easily solvable with JavaScript, so it’s no big deal.

The second, slightly larger one, was that dismissing the popover by clicking outside of it will not eat the input. If you were to click a sidebar link while the New Gallery model is opened, you’ll end up on that newly selected page. I’m not a fan of this. Dismissing the popover feels like its own user gesture, and I fear the user accidentally activating things when all they’re trying to do is dismiss the popover (it’s not in place now, but I am planning to dim the background when the Create Gallery modal is visible).

Fortunately, there’s a simple solution to this. It turns out that replacing the <div> element with a <dialog> element would solves both problems. It works seamlessly with the new popover attributes, yet showing the dialog will give focus to the form, and will eat the click when the user dismisses it.

Perfect. Looks like I’ve delayed the need for JavaScript a little longer (it will come eventually; it always will).

Mon, 12 Feb 2024 21:25:51 +1000

Spent some time this evening working on my image hosting tool. It’s slowly coming along, but wow do I suck at UI design (the “Edit Photo” screen needs some rebalancing).

Fri, 26 Jan 2024 15:08:32 +1000

Working on one of the admin sections of the project I was alluding to yesterday. Here’s a screencast of how it’s looking so far.

The styling and layout is not quite final. I’m focusing more on functionality, and getting layout and whitespace looking good always takes time. But compared to how it looked before I started working on it this morning, I think it’s a good start.

Wed, 10 Jan 2024 20:49:46 +1000

In the end it took significantly more time to write about it then to actually do it, but the dot product approach seems to work.

Download video

Tue, 09 Jan 2024 06:55:23 +1000

So I guess today’s beginning with a game of “guess the secret password requirements.” 😒

Fri, 22 Dec 2023 12:29:15 +1000

Making some progress in learning Elm for building frontends. Started working on a Connections clone, which I’m calling “Clonections”. This is what I’ve got so far:

It’s been fun using Elm to build this. So far I’m liking the language. Of course, now I’ll have to come up with puzzles for this. 😐

Sun, 03 Dec 2023 20:09:27 +1000

Spent a little more time working on my idea for Dynamo-Browse this week. Managed to get it somewhat feature complete this weekend:

Download Video

I probably should say a few words on what it actually is. The idea is to make it quick and easy to run pre-canned queries based on the currently selected item and table.

Let’s say you’ve got a table with customer information, and another table with subscription information, and they’re linked with some form of customer ID. If you wanted to see the subscriptions of a customer, up until now, you’d have to copy the customer ID to the paste-board, change the currently viewed table, then run a query to select the subscription with that customer ID. It’s not difficult but it’s extremely tedious.

This change is meant to streamline this. Now, in a script function, you can define a “related item” provider which, if matched against the currently displayed table, will be given the currently selected item, and will return a list of queries that will display items related to the current item (depending on whatever definition of “related” will be). This will be presented to the user as a list. When the user chooses the item, the query will run and the results will be displayed.

Here’s an example of the script used for the screencasts:

ext.related_items("business-addresses", func(item) {
    return [
        {"label": "Customer", "query": `city=$city`, "args": {"city": "Austin"}},
        {"label": "Payment", "query": `address^="3"`},
        {"label": "Thing", "table": "inventory", 
            "query": `pk=$pk`, "args": {"pk": "01fca33a-5817-4c27-8a8f-82380584e69c"}},
    ]
})

ext.related_items("inventory", func(item) {
    sk := string(item.attr("sk"))
    return [
        {"label": "SK: " + sk, "table": "business-addresses", 
            "query": `pk^=$sk`, "args": {"sk": sk}},
    ]
})

Notice how the last business-addresses item specifies the “inventory” table, and that the “inventory” provider actually uses an attribute of the item. Here’s a screencast of that working:

Download Video

This feature has been on the idea board for a while. I was trying to work out how best to handle the pre-canned queries, especially considering that they will likely be different for each item and table. Some ideas I had were adding additional UI elements that the user could use to configure these queries. These would go into the workspace file, a sort of an embedded database which is created for each session. This was pretty crappy, especially when you consider that workspaces usually only last until the user exists. It was only a few weeks ago when I considered using the scripting facilities to implement this (which, honestly, shows how much it remains under-utilised).

Anyway, I’ve only just finished development of it. I’d still like to try it for the various related items I tend to use during my day-to-day. We’ll see how well it works out.

Idea For Mainboard Mayhem: A Remote Pickup

Sun, 19 Nov 2023 20:21:56 +1000

Sort of in-between projects at the moment so I’m doing a bit of light stuff on Mainboard Mayhem. I had an idea for a new element: a remote control which, when picked up, will allow the player to toggle walls and tanks using the keyboard, much like the green and blue buttons.

I used ChatGGT to come up with some artwork, and it produced something that was pretty decent.

Prompt: pixel art of a remote control with a single red button styled like the tiles found in Chips Challange, rotated 45 degrees to the right.

Only issue was that the image was huge — 1024 x 1024 — and the tiles in Mainboard Mayhem were only 32 x 32.

I tried shrinking it down in Acorn, using various scaling algorithms. The closest that worked was bringing it down slowly to about 128 x 128 using Nearest Neighbour, than trying to go all the way down to 32 x 32 using Lanczos. That worked, but it required true 32 bit colour to be recognisable, and I wanted to preserve the 16 colour palette used by the original Chips Challenge.

So using the original image as a reference, I bit the bullet and drew my own in Acorn. You can see it here in this test level:

They're the elements that look like remote controls.

It turn out okay. At least it’s recognisable. Anyway, I coded it up and gave it a bit of a try:

Yeah, it works well. When the player has the appropriate colour remote, they can hit either Z or X to toggle the green walls or blue tanks respectively. I really should add some indicators in the status bar to show which button to press.

Not sure what I’ll do after this. The fun part was coming up with the element. But I guess I’ll have to come up with a few puzzles that use it.

Sun, 15 Oct 2023 15:52:50 +1000

More work on Mainboard Mayhem today. Had a bit more success getting the Windows build into a releasable state.

First thing was the app icon. That blog post I talked about yesterday worked: I was able to set the icon of the executable. I did make a slight adjustment though. The post suggested using ImageMagick to produce the ICO file, but I wasn’t happy with how they looked. There were a lot of artefacts on the smaller icon sizes.

So I looked around for an alternative, and found this package by Lea Anthony. He’s the maintainer of Wails, a cross-platform toolkit for making browser-based GUI apps in Go, sort of like Electron but without bundling Chrome. In fact, most of the build for Mainboard Mayhem was put together by reading the Wails source code, so I trust he knows what his doing. And sure enough, his package produced a nicely scaled ICO file from a source PNG image. Better yet, it was distributed as a Go package, so I could no need to install and shell-out to run it: I could just integrated it directly into the project’s build tool.

Using rsrc to generate the SYSO file with the icon worked as expected: Go did pick it up and embed it into the executable. I did have some trouble getting the Go compiler to pick up these files at first. In short, they need to be in the same directory as the main package. So if you’re running go build ./cmd/thing, make sure the SYSO files are in ./cmd/thing. Other than that, no real issues here.

A beautiful site: Mainboard.exe with the embedded app icon

One last thing I had to deal with was the console window. Running a Go app in Windows shows the console by default. Perfectly fine for command line tools, but less so for games:

Mainboard Mayhem with that annoying console window. Even the log messages are dull (well, unless you're working on the app).

So I had to find a way to hide the console on launch. Since Mainboard Mayhem is using SDL, I’m actually using MinGW to cross-compile the Windows release on an Ubuntu build runner. The documentation for MinGW suggests adding -mwindows as a linker option to hide the console:

# What I was doing before, which didn't work
CGO_ENABLED=1 \
CC="x86_64-w64-mingw32-gcc" \
GOOS="windows" \
CGO_LDFLAGS="-mwindows -L…" \
go build -o dist/cclm/mainboard.exe ./cmd/cclm'

This didn’t actually work when I tried it: launching the app kept bringing up the console. Turns out what I should’ve done was follow the advice of many Stack Overflow answers, and set -ldflags "-H=windowsgui" on the Go command:

# This works
CGO_ENABLED=1 \
CC="x86_64-w64-mingw32-gcc" \
GOOS="windows" \
CGO_LDFLAGS="-L…" \
go build -ldflags "-H=windowsgui" -o dist/cclm/mainboard.exe ./cmd/cclm'

This works even without the -mwindows switch. Not completely sure why though. I guess MinGW is not actually being used for linking? Or maybe -m only works with C header files? Don’t know. 🤷 But doesn’t matter: the console no longer shows up on launch.

Mainboard Mayhem without the console window. A much nicer experience now.

Finally, there was testing it all, and for this I just bit the bullet and set-up a Windows 10 virtual machine in Azure. The rate is something like $0.16 AUD an hour, an easy decision compared to spending time trying to get a VM with Windows 10 running on my machine.

One remaining thing that’s slightly annoying is Windows Defender refusing to launch it after download, doing effectively the same thing as Gatekeeper on MacOS does:

Gatekeeper a.la. Microsoft.

I’m sure there’s a way around it but it’s probably not worth learning about it at this stage. It’s easy enough to dismiss: click “More Info” and the click “Run Anyway”:

Clicking 'More Info' gives you a way to launch the app.

But other than that, I think the Windows version of Mainboard Mayhem is ready. I’ve updated the website to include the Windows archive if anyone’s interested.

Sat, 14 Oct 2023 21:20:04 +1000

Spent some time today on Mainboard Mayhem, trying to finish the Windows build. I’ve actually got Windows version of the game being built for a while now. I just haven’t published them, mainly because I haven’t got the app icon set-up yet.

But this week, Golang Weekly had a link to a blog post by Mahmud Ridwan on how to do so. It looked pretty straightforward, so I thought I’d give it a try.

And yeah, the instructions themselves were easy enough, and I wish I could say if they worked or not. But in order to test it, I need a Windows machine. And I don’t have one, and I wasn’t about to get one just for this.

So I tried setting up Windows in a VM using UTM. I got this far:

Yeah, this infinite spinner has been staring at me pretty much all day. I got a Windows 10 installer ISO using CrystalFetch, and it seemed to work. But it just doesn’t want to boot up for the first time.

Not actually sure what the problem is. The error message seems to suggest that it’s having trouble connecting to the internet. Might be that? Or maybe the installation didn’t complete properly? Could be anything. 🤷

So no luck getting this tested yet. I’m wondering if it might be easier to forget virtualisation and just launch a Windows instance in the cloud somewhere instead.

Mainboard Mayhem

Wed, 30 Aug 2023 22:27:50 +1000

Project update on Mainboard Mayhem, my Chip’s Challenge fan game. I didn’t get it finished in time for the release deadline, which was last weekend. I blame work for that. We’re going through a bit of a crunch at the moment, and there was a need to work on the weekend.

The good news is that there wasn’t much left to do, and after a few more evenings, I’m please to say that it’s done. The game is finish, and ready for release.

So here it is: Mainboard Mayhem: A Chip’s Challenge fan game (and yes, that’s its full title).

At the moment it’s only available for MacOS. It should work on both Intel and Apple Silicon Macs, although I’ve only tested on my M2 Mac Mini running Ventura.

It’s good to finally see this project done. It’s been in development for about last ten years, and I spent half of that time wondering whether it was worth getting it finished it at all. Not committing to anything meant any work I did do on it was pretty aimless, and I always felt like I was wasting my time. Giving myself three weeks to either kill it, or release it helped a lot. I’ll start making deadlines for all the other unfinished projects I’m working on.

As to what that next project will be, I not sure at this stage. Part of me wants to wait until this crunch time ends, but I suspect I’ll get antsy before then and start work on something else. I’ll keep you posted one way or the other.

But for now, if you happen to give it a try, thank you and I hope you enjoy it.

Sun, 20 Aug 2023 11:00:37 +1000

Project update for Mainboard Madness. Well, today’s the deadline for getting the thing code complete, and what a surprised, it’s not finished.

To be fair, it’s pretty close. All the levels are more or less done, and the beats of the in-game lore have been added. It all just needs tightening up a little. I spent today working on the end-game phase, which mainly involved coding up the credit sequence, and making sure I include credits for those involved in the original game (and who’s artwork I lifted).

The work remaining is to finish one or two game elements, adding a proper app icon, and finishing off the website. I’m wondering whether to add sound, but I feel bad enough taking the artwork from the original game, I rather not take the sound effects as well. That will mean the game will remain silent for the time being, but I can probably live with that for now.

I think we’re still on track for getting this finished by this time next week. Last dash to the finish line, then I can put this 9 year project to rest for a while.

Sun, 13 Aug 2023 13:36:44 +1000

Small project update on my Chips Challenge fan game.

Started working on the final level. I was dreading this a little, thanks to my awful level design skills, but I made a great start to it this morning and it’s actually coming along pretty well. It’s a good opportunity to use all the elements that I didn’t get a chance to use in any of the other puzzles, and it’s also shaping up to be one that has a bit of climax.

I’ve also started working on the website, which is little more than just a landing page. This meant finally coming up with a name. I’ve chosen “Mainboard Mayhem” which is… okay, but it’s one that’s been rattling around in my head for a while, and I really couldn’t use anything close to “Chips Challenge”. I’m already using the tile-set from the original game, I rather not step on any more intellectual property.

Anyway, one more week of development left to go. Still need to setup the app icon, finish all the levels, and maybe add a menu. Then I think we’re code complete.

Sat, 05 Aug 2023 13:43:19 +1000

Working on my Chips Challenge “fan game” this morning. Added the notion of “lower thirds,” which will show text at the bottom of the play field. I’m hoping to use it for narrative or way-finding, like here in this hub level:

Also working on puzzle design. There’s about 19 or so “real” puzzles but I’m wondering if it’s worth adding a few tutorial ones for those that have never played the original Chip Challenge before. I’ve done about 5 such puzzles and I think I need to add maybe 3 or 4 more to cover everything I’m hoping to demonstrate. I wish I liked puzzle design more than I like tinkering on the engine.

Of course, the big question is why I’m working on this at all. There is, for lack of a better word, a vision for this, in terms of narrative and structure, but this project has been in development on and off for about 9 years or so, and I’m wondering if it’s time to just stop working on it altogether. I really am starting to get sick of it, in a way. And yet, this project has shown remarkable staying power over that time that I feel like if I don’t actually wrap it up, it’ll just continued to be worked on. It feels like the only way to end this project is to finish it, in one way or another.

So I’ll set myself a dead-line: something releasable in two weeks, and actually released a week after that. After that, no more! I’ll work on something else.

Sat, 29 Jul 2023 10:28:48 +1000

Attempting to design an app icon for a Chips Challenge fan game I’m working on. Going for something that looks like the fireball sprite in the original game with a hint more realism and tinted in the colour blue. For reference, here’s the original fireball sprite:

And here’s my attempt:

I started with Stable Diffusion to get the base image:

Prompt: a blue plasma fireball shaped like a throwing star with four points on a white background, pixel art

Then imported into Acorn to rotate it, colourise it, and distort it to look a bit closer to the original sprite.

Desaturating the original image got rid of the purple centre, then applying the Glowhoo and Hue Adjust effect recolourised it to the blue I was looking for (I’m not sure what the Glowhoo effect does, but it seems to adjust the colour based on the pixel intensity, so it was good enough for what I wanted). Finally, I added a Twirl Distortion effect to achieve the slight warp in the star.

And yeah, it’s not going to win any design awards, but it’s good enough for now.

Oh, and just for kicks, here was my first attempt of producing the sprite using Affinity Designer.

That’s definitely not going to win any design awards. 😂

Wed, 26 Jul 2023 22:43:28 +1000

Success! Managed to get a Go app built, signed, and notarised all from within a GitHub Action. It even cross-compiles to ARM, which is something considering that it’s using SDL. Here’s the test app being downloaded and launched in a VM (ignore the black window, the interesting part is the title).

Sun, 23 Jul 2023 09:33:00 +1000

Spent most of the weekend going down various rabbit holes to get a Go application signed and notarised as a MacOS app. I’m trying to do this in a way that would make this easy to automate using GitHub Actions. This means things like no implicit access to the system keychain: I want to make a temporary keychain, add my secret stuff to it, then delete it once signing and notarisation is done.

It also means no XCode GUI either: command line stuff only. Not that I had much hope of using XCode here anyway, since this is a Go application.

But that’s fine, preferable even. I’ve never liked all the manual steps needed to get code signing work with XCode. What are you going to do when you change systems? Would you remember all the steps you took several years ago, when you last setup developer certificates?

So this is why I’m trying to get it working with the terminal. But it’s not easy. Lots of esoteric commands that I need to learn and be made aware of. Just hope it’s not a huge waste of time.

Sun, 21 May 2023 21:52:03 +1000

I’ve been working on Micropub Checkins over the last week. It’s been a bit of a rush trying to get it into a usable state for an upcoming trip. And by “usable”, I mean a form that I can tolerate, and when it comes to projects like this, I can tolerate quite a lot. It can have a really dodgy UI (which this does) and miss some really important features that are annoying to work around; but if it works, and doesn’t loose data, I’ll be fine with it.

The main screen showing the recent check-ins. Note the lock next to some of them. These won't be published until the locks are removed.

The last week was dedicated to making the act of checking in distinct from publishing it. Until now, check-ins were published the minute they were entered, meaning that you cannot check-in somewhere unless you’re comfortable with people knowing where you are the minute you do. Yes, some people like it that way, but not me. And I’m aware that this’ll only be the case if people are following my check-in blog, which I’m doubtful of.

So pressing the floating action button and choosing a check-in type now starts the flow of a new check-in that will get saved in an SQLite database. You can edit the check-in whenever you like, so long as it’s not published. Currently there’s no real way of deleting a check-in unless it’s been published. This is a bit dodgy, but it’s a good example of how tolerant I am with working around these feature gaps for the moment.

The newly styled edit screen. Notice the rating field, which will appear for eateries.

Check-ins can be published by tapping the upward facing button on the main screen. Any check-in with a lock is private and will not be published until you toggle the “Ready to publish” switch in the properties. Doing so will not change the date of the check-in: it will still have the date and time that check-in was created.

The targets are currently hard-coded but they can be turned on or off. I had a bit of trouble publishing a check-in to two targets, so I'm not sure if I'll keep multi-target publishing.

On the subject of publishing, I had some issues with Dart’s date and time methods. The method on the DateTime class used to produce an ISO-8501 date-stamp don’t include the time-zone if the date and time is not in UTC. This is important as I want the post date and time to be as close to the check-in time as possible, and in the time-zone of the phone. DateTime knows all this, including what the time-zone we’re in. So why didn’t the developers include it in the ISO-8501 date-time string?

This is really strange. Fortunately, ChatGPT stepped in to help out, writing a function which will add the time-zone offset to the ISO-8501 date-time string:

String formatTimeZoneOffset(Duration offset) {
  String sign = offset.isNegative ? '-' : '+';
  int hours = offset.inHours.abs();
  int minutes = (offset.inMinutes.abs() % 60);

  return '$sign${_padZero(hours)}:${_padZero(minutes)}';
}

Honestly, ChatGPT has been so helpful over the past week with this project, I probably should give it a credit if I get this polished enough to release.

Sat, 13 May 2023 12:25:17 +1000

Back working on Micropub Checkin. Re-engineered the home page to now include a list of what would eventually be check-ins — both historical and soon to be published — complete with the check-in type emoji as the icon:

Main screen for Micropub Checkin

The same list of emoji icons now adorn the check-in type picker as well (except for the airplane one which seems to always be shown as what I can only describe as the “Wingding” representation):

The check-in type picker

I went around a bit trying to work out how best to use these emojis icons in the leading slot of the ListTile widget. I expored trying to convert them to IconData, but it turns out just using a Text widget with a large font worked well. I wrapped in in a Widget type with a fixed font-size and so far it looks quite good, at least in the emulator:

class EmojiIcon extends StatelessWidget {
  final String emoji;

  const EmojiIcon({super.key, required this.emoji});

  Widget build(BuildContext context) {
    return Text(emoji, style: TextStyle(fontSize: 26.0));
  }
}

Also started working on a Cubit to handle state for the main page. I had a bit of trouble working ont where the soon-to-be database call to get the list of checkins should go in the cubit. After asking ChatGPT, it looks like the initializer is the best place for it:

class CheckinListCubit extends Cubit<CheckinListState> {

  CheckinListCubit(): super(LoadingCheckinListState()) {
    loadCheckinList();
  }

  void loadCheckinList() async {
    var listOfCheckins = await read_database(); 
    emit(FoundCheckinListState(checkins));
  }
}

I’ve got some scaffolding code in place to simulate this, and so far it seems to work.

I need to start working on the database layer and having the ability to edit and delete check-ins before they’re published. I think I’ll tackle that next.

Building F5 To Run

Sun, 16 Apr 2023 10:36:18 +1000

At the risk of talking about something that I’ve only just started, I’d thought today I write about what I’m working on right now.

I’ve been going through my digital archives this weekend, trying to get it into something more permenant than the portable USB drives it’s currently stored on. Amongst all that stuff is a bunch of QBasic apps and games I wrote way back when I was a kid. Over the years it’s laid dormant but I do like revising them from time to time.

Is it a form of nostalgia? An attempts to live past glories? Maybe? I was pretty proud of them at the time, much like anyone else that’s proud of their early stuff while they’re leaning to code. And I know about the risk of living in the past at the expense of the present. But I also know that if I get rid of them, I’d regret it. I already regret loosing the things from the archive so far, due to bad disks or just missing things while copying them from portable hard-drive to portable hard-drive. I don’t want to loose any more.

So in an act of posterity, I’d figured it’s time to coat them in amber and put them online. So that’s what I’m doing now.

These apps run without issue in DosBox, and hearing about how the Wayback Machine has managed to make a bunch of DOS games playable within the browser, I wondered if I could do something similar. Anything that deals with virtualisation is always going to be a little bit involved. I guess one thing going for these is that they were written for a pretty slow machine and a pretty thin OS that would be trivial for modern hardware to emulate. The apps themselves, even compiled to an EXE file, are not very taxing on the hardware back then either. But I still expected to do a bit of heavy lifting myself.

How wrong I was! After a tiny bit of research — and by tiny I mean one Ecosia search — I managed to find a JavaScript library called JS-Dos which provides a DosBox emulator that’s runnable from a browser. All I need to do is prepare a bundle on what I want to run (more on that below) and with a bit of JavaScript, I can start a DosBox machine in the browser and mount it to a HTML element. The library does all the work.

How To Use JS-Dos

It’s still early days, but here’s what I learnt about using the library so far.

First, the library comes as a NPM package, or can be loaded from their CDN in the form of a <script> import. I first tried using the NPM package, but I didn’t know the appropriate import statement to use, and the documentation was not forthcoming on this front.

So I went with the CDN approach. I’m using Hugo Pipes to fetch the remote JavaScript file and make a local bundle so I can host it from the site itself. It comes with some CSS which I also need to get (note, I’m using parenthesis instead of curly braces here as I’m not sure how to include two curly braces in a code-block).

(( $jsDosCSS := resources.GetRemote "https://js-dos.com/v7/build/releases/latest/js-dos/js-dos.css" ))
<link rel="stylesheet" href="(( $jsDosCSS.RelPermalink ))">
        
(( $jsDosJS := resources.GetRemote "https://js-dos.com/v7/build/releases/latest/js-dos/js-dos.js" ))
<script src="(( $jsDosJS.RelPermalink ))" defer></script>

I also needed to get an appropriate wdosbox emulator. This comes in the form of a WASM file, plus a small JavaScript file which I assume is some simple bootstrapper. I’ve downloaded these and stored them in the static/emulators directory of my Hugo project. The JSDos library loads them on demand and I needed to set the URL path prefix for these two files so that JSDos knows where to get them:

emulators.pathPrefix = '/emulators/';

Next, I needed to build a bundle. These are the DOS programs that are launched with DosBox. They’re effectively just Zip files holding some metadata, the DOS executable, and any files needed for the program to run. There’s some basic information about how to make them, and there’s even an online tool which will take on a lot of the tedious work. I’ve used it to make a couple of test bundles and it works quite well. I’d like to eventually make my bundles myself but I’ll stick with the tool for the time being, at least until I’ve got a DosBox configuration that I’m happy with. One thing the tool does is give you the ability to define an overlay so that these DOS apps are usable from within a mobile browsers. I’ll see if I can get away from needing these overlays at this stage. I’m not expecting anyone with a mobile app to try these out.

The contents of .jsdos/dosbox.conf for the test bundle `logo-2.jsdos`

[sdl]
autolock=false

fullscreen=false
fulldouble=false
fullresolution=original
windowresolution=original
output=surface
sensitivity=100
waitonerror=true
priority=higher,normal
mapperfile=mapper-jsdos.map
usescancodes=true
vsync=false
[dosbox]
machine=svga_s3

language=
captures=capture
memsize=16
[cpu]
core=auto
cputype=auto
cycles=max

cycleup=10
cycledown=20
[mixer]
nosound=false
rate=44100

blocksize=1024
prebuffer=20

[render]
# frameskip: How many frames DOSBox skips before drawing one.
#    aspect: Do aspect correction, if your output method doesn't support scaling this can slow things down!.
#    scaler: Scaler used to enlarge/enhance low resolution modes.
#              If 'forced' is appended, then the scaler will be used even if the result might not be desired.
#            Possible values: none, normal2x, normal3x, advmame2x, advmame3x, advinterp2x, advinterp3x, hq2x, hq3x, 2xsai, super2xsai, supereagle, tv2x, tv3x, rgb2x, rgb3x, scan2x, scan3x.

frameskip=0
aspect=false
scaler=none

[midi]
#     mpu401: Type of MPU-401 to emulate.
#             Possible values: intelligent, uart, none.
# mididevice: Device that will receive the MIDI data from MPU-401.
#             Possible values: default, win32, alsa, oss, coreaudio, coremidi, none.
# midiconfig: Special configuration options for the device driver. This is usually the id of the device you want to use.
#               See the README/Manual for more details.

mpu401=intelligent
mididevice=default
midiconfig=

[sblaster]
#  sbtype: Type of Soundblaster to emulate. gb is Gameblaster.
#          Possible values: sb1, sb2, sbpro1, sbpro2, sb16, gb, none.
#  sbbase: The IO address of the soundblaster.
#          Possible values: 220, 240, 260, 280, 2a0, 2c0, 2e0, 300.
#     irq: The IRQ number of the soundblaster.
#          Possible values: 7, 5, 3, 9, 10, 11, 12.
#     dma: The DMA number of the soundblaster.
#          Possible values: 1, 5, 0, 3, 6, 7.
#    hdma: The High DMA number of the soundblaster.
#          Possible values: 1, 5, 0, 3, 6, 7.
# sbmixer: Allow the soundblaster mixer to modify the DOSBox mixer.
# oplmode: Type of OPL emulation. On 'auto' the mode is determined by sblaster type. All OPL modes are Adlib-compatible, except for 'cms'.
#          Possible values: auto, cms, opl2, dualopl2, opl3, none.
#  oplemu: Provider for the OPL emulation. compat might provide better quality (see oplrate as well).
#          Possible values: default, compat, fast.
# oplrate: Sample rate of OPL music emulation. Use 49716 for highest quality (set the mixer rate accordingly).
#          Possible values: 44100, 49716, 48000, 32000, 22050, 16000, 11025, 8000.

sbtype=sb16
sbbase=220
irq=7
dma=1
hdma=5
sbmixer=true
oplmode=auto
oplemu=default
oplrate=44100

[gus]
#      gus: Enable the Gravis Ultrasound emulation.
#  gusrate: Sample rate of Ultrasound emulation.
#           Possible values: 44100, 48000, 32000, 22050, 16000, 11025, 8000, 49716.
#  gusbase: The IO base address of the Gravis Ultrasound.
#           Possible values: 240, 220, 260, 280, 2a0, 2c0, 2e0, 300.
#   gusirq: The IRQ number of the Gravis Ultrasound.
#           Possible values: 5, 3, 7, 9, 10, 11, 12.
#   gusdma: The DMA channel of the Gravis Ultrasound.
#           Possible values: 3, 0, 1, 5, 6, 7.
# ultradir: Path to Ultrasound directory. In this directory
#           there should be a MIDI directory that contains
#           the patch files for GUS playback. Patch sets used
#           with Timidity should work fine.

gus=false
gusrate=44100
gusbase=240
gusirq=5
gusdma=3
ultradir=C:\ULTRASND

[speaker]
# pcspeaker: Enable PC-Speaker emulation.
#    pcrate: Sample rate of the PC-Speaker sound generation.
#            Possible values: 44100, 48000, 32000, 22050, 16000, 11025, 8000, 49716.
#     tandy: Enable Tandy Sound System emulation. For 'auto', emulation is present only if machine is set to 'tandy'.
#            Possible values: auto, on, off.
# tandyrate: Sample rate of the Tandy 3-Voice generation.
#            Possible values: 44100, 48000, 32000, 22050, 16000, 11025, 8000, 49716.
#    disney: Enable Disney Sound Source emulation. (Covox Voice Master and Speech Thing compatible).

pcspeaker=true
pcrate=44100
tandy=auto
tandyrate=44100
disney=true

[joystick]
# joysticktype: Type of joystick to emulate: auto (default), none,
#               2axis (supports two joysticks),
#               4axis (supports one joystick, first joystick used),
#               4axis_2 (supports one joystick, second joystick used),
#               fcs (Thrustmaster), ch (CH Flightstick).
#               none disables joystick emulation.
#               auto chooses emulation depending on real joystick(s).
#               (Remember to reset dosbox's mapperfile if you saved it earlier)
#               Possible values: auto, 2axis, 4axis, 4axis_2, fcs, ch, none.
#        timed: enable timed intervals for axis. Experiment with this option, if your joystick drifts (away).
#     autofire: continuously fires as long as you keep the button pressed.
#       swap34: swap the 3rd and the 4th axis. can be useful for certain joysticks.
#   buttonwrap: enable button wrapping at the number of emulated buttons.

joysticktype=auto
timed=true
autofire=false
swap34=false
buttonwrap=false

[serial]
# serial1: set type of device connected to com port.
#          Can be disabled, dummy, modem, nullmodem, directserial.
#          Additional parameters must be in the same line in the form of
#          parameter:value. Parameter for all types is irq (optional).
#          for directserial: realport (required), rxdelay (optional).
#                           (realport:COM1 realport:ttyS0).
#          for modem: listenport (optional).
#          for nullmodem: server, rxdelay, txdelay, telnet, usedtr,
#                         transparent, port, inhsocket (all optional).
#          Example: serial1=modem listenport:5000
#          Possible values: dummy, disabled, modem, nullmodem, directserial.
# serial2: see serial1
#          Possible values: dummy, disabled, modem, nullmodem, directserial.
# serial3: see serial1
#          Possible values: dummy, disabled, modem, nullmodem, directserial.
# serial4: see serial1
#          Possible values: dummy, disabled, modem, nullmodem, directserial.

serial1=dummy
serial2=dummy
serial3=disabled
serial4=disabled

[dos]
#            xms: Enable XMS support.
#            ems: Enable EMS support.
#            umb: Enable UMB support.
# keyboardlayout: Language code of the keyboard layout (or none).

xms=true
ems=true
umb=true
keyboardlayout=auto

[ipx]
# ipx: Enable ipx over UDP/IP emulation.

ipx=true
[autoexec]
echo off
mount c .
c:

type jsdos~1/readme.txt
echo on

LOGO.EXE

I’m keeping the bundles in the static/bundles directory, which sits alongside the emulator WASM file. They’re not huge binaries but I’m still using git lfs to manage them. Best to keep the the Git repository relatively sane.

Finally, it’s just a matter of adding some JavaScript to start DosBox, load the bundle, and mount it onto a HTML element:

Dos(document.querySelector("#element-to-mount")).run("/bundles/bundle-to-load.jsdos");

And that’s pretty much it.

A test webpage with JSDos running Logo 2, which is one of my Basic programs

After a few hours, I’ve managed to get a test version of this working. There are a few things that need working on: the app I’m trying require the Alt key, which is not readily available of Apple keyboards, so I may need to do something about that (JSDos comes with a virtual keyboard with Ctrl and Alt so it’s not a complete show-stopper)¹. And I’ll need to get styling and more maintainable JavaScript written for this (I’m using StimulusJS for the JavaScript²). But I’m impressed by how well this works, given the minimal amount of effort from my part. Shoulders of giants and all that.

I’ve since learnt that Alt is bound to the Option key in MacOS. ↩︎
One thing I leant about Hugo is that it’s bundled with ESBuild, meaning that it’s resource pipeline supports NPM packages. My understanding is that this is invokable using the js.Build construct. This is super useful to know. ↩︎

Updates To Dynamo-Browse

Sun, 02 Apr 2023 12:36:32 +1000

In the off-chance that anyone other than me is reading this, it’s likely that there will be no update next week due to the Easter weekend. It may not be the only weekend without an update either. If I find that I didn’t get much done for a particular week, I probably won’t say anything and leave the well to fill up for the next one (although I do have some topics planned for some of those weekends).

In fact, I was not expecting to say much this week, given that work was going through a bit of a crunch period. But the needs of work finally pushed me to add a few features to Dynamo-Browse that were sorely lacking. So that’s where most of this week’s side-project budget went to.

Dynamo-Browse

A lot of work done to query expressions in Dynamo-Browse this week, some of it touching on a few topics I mentioned in a previous update.

I’ve finally got around to finishing the between keyword, so that it works with the query planner and actually produces a DynamoDB query when used with a range[^sort] key. This means no more falling back on table scans. It’s still in a branch as of this post, but I feel much less embarrassed with merging it now, given that this support has been added.

I’ve also made a decision about how to deal with multiple index candidates. Now, when Dynamo-Browse finds that multiple indices can apply for a specific query expression, it will produce an error, requesting you to specify which index to use. This can be done by adding a using suffix to an expression, which specifies how the query should be evaluated:

color="blue" using index("color-item-index")

This can be used at any time to override the index Dynamo-Browse should use, even if only one index candidate was found. It can also be used to force the query to run as a table scan if you don’t want to use an index at all:

color="blue" using scan

Ideally you shouldn’t need to use this suffix that often. The whole purpose of query expressions was to eliminate the need for specifying details of how the query should be evaluated. But we don’t live in a perfect world, and it makes sense adding this to deal with cases where it helps to be specific.

This is also in a branch, but I’m hoping this would be merged soon as well.

Unified Expression Types and Values

A relatively large change made this week was how how values and types are represented within query expressions.

A query expression, once parsed, can be executed in multiple contexts. It can be used to generate a conditional expression for a DynamoDB query or scan, or it can be evaluated within the app itself to produce a result or alter the fields of a DynamoDB record in memory. Each of these contexts have a different set of types the expression operates on. When interpreting the expression in order to produce a result, the expression operates on types that implement types.AttributeType. This fits nicely with what the expression has to work with, which is usually the raw DynamoDB records returned by the Go client. The context used to produces conditional expressions, however, operate on a sort of hybrid type hieararchy, that supports both AttributeType and Go types. This is because to the client used to build the expression accept native Go values, which are sometimes available — particularly if they show up in the expression as a literal — but sometimes not.

But here’s the problem: I want to be able to add functions to the expression language that can be used in both contexts. I’ll get into what sort of functions I’m thinking of in a minute, but the issue is that with two sets of type hierarchies, I’d have to implement the functions twice.

Another problem is that an evaluation context operating on AttributeTypes feels very inefficient. Numbers are represented as string, and new attribute values are created on the heap. This is probably not too bad in the grand scheme of things, but it would be nice to use native Go values here, even if it’s just to avoid going from strings to numbers constantly.

So I spent most of yesterday trying to fix this. I built a new private Go interface called exprValue and added as implementing subtypes all the types supported by DynamoDB — strings, numbers, booleans, lists, etc. Values of these type implement this new interface, and can be converted to Go values or DynamoDB AttributeType values depending on the need.

Most of the evaluation logic was changed to use these types, including the builtin functions, and already I’m seeing some dramatic improvements of what’s possible now. I can define a function once and it can be evaluated both in the evaluation and query building context (provided that it’s only operating on constant values in the query building context). It also addressed some long standing issues I’ve had with the expression language, such as adding support for using a list with the in keyword; something that was not possible before:

pk in $someList

This could potentially be helpful with the “fan-out” one I mentioned a few weeks ago.

This is still early days, but I think it’s been a huge improvement to what was there before. And it’s super satisfying cleaning out all this tech-debt, especially if it means I can add features easily now.

Date Functions In The Expression Language

Now with the new type hierarchy in place, the time has come to start adding functions. What existed to date were the operators and functions that DynamoDB’s conditional expression language supported, and little else. It’s time to go beyond that. And to be honest, this was always the plan, especially given that operators like “begins with” (^=) have been there since the start.

This first thing I’m pondering now is time and date functions. The immediate issue is one of representation: in that I don’t want to settle on any specific one. I’ve seen dates stored as both string date-stamps, usually in ISO 8601, or as integer seconds from the Unix epoch, and it would be good to operate on both of these, in addition to other possible representations, like milliseconds from the Unix epoch, to some other string encoding scheme.

So what I’m thinking is an abstract date-type, probably something backed by Go’s builtin date.Time type. This will neither be a number or a string, but can be converted to one, maybe by using a keyword like as:

now() as "S"   -- represent date as ISO-8601 timestamp
now() as "N"   -- represent date as Unix timestamp
now()          -- error: need to convert it to something

Or maybe some other mechanism.

The idea is that all the builtin functions will operate on this type, but will prevent the user from assuming a particular representation, and will force them to choose one.

I think this is something that will fit nicely with the new type hierarchy system, but for now (pun unintended), I’ll stick with Unix timestamp, just so that I can use something that is easy to implement. But to make it crystal clear that this is temporary, any such functions will have an annoying prefix.

So two new functions were added this week: the _x_now() function, which returns the current time as seconds from the Unix epoch as a number; and the _x_add(), which returns the sum of two numbers. Much like the time functions, I’d like to eventually add arithmetic operators like + to the expression language, but I needed something now and I didn’t have much time to work on that.

Attribute Commands

Finally, a few random notes about commands dealing with attribute values.

The set-attr command can now accept the switch -to, which can be used to set the attribute to the result of a query expression. No more copying-and-pasting values, and operating on them outside Dynamo-Browse. The good thing about this is that the previous attribute values are available in the value expression, so you can use this switch to set the value of attribute based on other attributes in a row. This comes in super handy with bulk changes. I’ve used this to adjust the value of TTLs in a table I’m working in. To set the TTL to be 10 minutes into the future, I just marked the rows, entered the command set-attr -to ttl, and use the expression _x_add(_x_now(), 600). Super useful.

Also, I’ve found a bug where the del-attr command does not work with marked items. It’ll only delete attributes from the item that’s selected (i.e. in pink). I haven’t got around to fixing this, but I hope to very soon.

I think that’s all for this week. Until next time.

Updating Bocce Scorecard

Sun, 26 Mar 2023 10:13:43 +1000

I didn’t get to a lot of side-project work this week, but I did have to make a large change to a project we use to track scores for our “bocce club”. So I’d though I’d say a few words about that today.

We had our bocce “grand final” a few weeks ago, and one of the matches resulted in a tie between two players. Unfortunately, the Bocce Scorecard web-app I build could not properly handle these, which meant that I had to fix it.

I’ll be honest in saying that that this was never really well fleshed out in the code, and there was actually a small bug which didn’t handle the ranking of players well. But I was pushing to keep this app as the de-facto source of truth for these matches, and there was a bit riding on this being correct (there’s a $4 trophy involved). So I had to get this fix before our next match, which was today.

Now, I been having trouble coming up with a good description of what the rules should be so I’d figured a simple example would suffice.

Imagine that there are four players: Tom, Dick, Harry, and Sally. They play several bocce matches during a season — which roughly corresponds to one calendar year — plus three “grand final” matches at the end. Each player would be awarded a number of “season points” (we informally call them “cookies”) based on how well they did in the match. The person with the most season points at the end of the last grand final match wins the season, and gets to take home the trophy.

In regular matches, the wining player is awarded one season point, while the remaining players get nothing:

Player	Score	Season Points
Tom	11	1
Dick	8	0
Sally	6	0
Harry	3	0

In grand final matches, the winning player is awarded 5 points, the one coming in second gets 2, and the one coming in third gets 1:

Player	Score	Season Points
Tom	11	5
Dick	8	2
Sally	6	1
Harry	3	0

Season points for grand final matches are distributed this way so that a single grand final match is roughly equivalent to an entire season of regular matches. This means that if someone is coming in last during the regular season (fun fact: that person’s usually me) they still has a chance to win the trophy if they do well during the grand final.

Now, let’s say that our four players are playing a grand final match, and Dick and Sally tie for second place. What should happen is that both Dick and Sally should be awarded half the season points they would get for both the second and third rank, given that they are evenly match for these two positions. In other words, they should both get 1.5 season points (1 + 2 = 3 / 2 = 1.5). Harry, who came last, still gets zero.

Player	Score	Season Points
Tom	11	5
Dick	7	1.5
Sally	7	1.5
Harry	3	0

This was the rule that I needed to change.

What I found when I started working on this is that the rule definitions themselves needed to be closer to how the players are ranked. What was previously done was that the players were sorted based on their match score, and then the rules were applied to each one by checking the win condition and awarding the points if they match it. But this didn’t fit nicely with this new approach to ties.

So instead of the conditions and awards approach, I simplified the rule definitions such that it simply defines the number of season points based on the players rank. This effectively makes it a simple map between rank and points. For normal matches the mapping would look like this:

Rank	Season Points
1	1

and for grand final matches, like this:

Rank	Season Points
1	5
2	2
3	1

Now, when a match is over, the logic that awards the season points first sorts the players based on their match score, and then groups the players into buckets such that all the players with same match score are lumped together in the same bucket. Ranks are then assigned to the players in descending score order. If two players have the same score, they will be given two ranks (e.g. Dick and Sally would have both rank two and three). Finally, season points are awarded with the rule definition and the following formula:

season_points(player) = sum[over player_ranks](rules.rank_scores[rank]) / no_of_players_in_bucket

This new logic works for ties between any number of players with any ranks.

But the introduction of division now means that the season points can be a decimal, and the database row that holds the season points is an integer type. I didn’t want to make it a floating point, so I took a page from Stripe and simply changed the representation of the season scores such that 1 season point is represented as 100 in the database. This is exposed in the rules configuration, which now looks like this:

{
  "rank_scores": [
    {
      "points": 500,
      "rank": 1
    },
    {
      "points": 200,
      "rank": 2
    },
    {
      "points": 100,
      "rank": 3
    }
  ]
}

although all the non-admin screens properly represents the score as a decimal number.

I managed to get all finished and pushed to the server, but there was one other thing I think I’d like to get done down the line. My friends have been asking me about the outcome of previous seasons recently and I’d like to make it easier for them to view it themselves. The data exists, but it’s super hacky to get: you need to “open” a previous season so that the leader board is shown on the home page, then close it again once the info is seen. This can only be done by the admin user (i.e. me) and the screens to do it leave a lot to be desired:

The current season admin section.

What I’m thinking is adding a “Seasons” section in the web-app. Clicking “Seasons” in the nav will bring up the following screen:

Mockup of a new season browser section.

The game variant will appear the top as a tab, and below them are all the current and past seasons arranged in descending chronological order. Clicking the > will bring up the season results display:

Drilling down into a season brings up the details, complete with a leader board and list of matches played during that season.

This will show the final outcome of the season, any metadata associated with the season, and the matches of the season, along with the winner. Clicking the location will bring up the particular bocce session so that all the matches played that day can be seen.

We’ll see when I get around to building this. It’s actually been a while since I’ve last touched this project while making such a large feature.

Oh, and since it’s been a while, this usually means I needed to upgrade Buffalo, the framework this app is using. Doing this usually means that you’ll need to change your app in some way to handle the new build process. This time, it’s moving the main.go file, previously in the project directory, into a cmd/app directory. When you see output like this:

leonmika@Stark bocce-scorecard % buffalo build -o /tmp/app
Usage:
  buffalo build [flags]

Aliases:
  build, b, bill, install

Flags:
      --build-flags strings        Additional comma-separated build flags to feed to go build
      --clean-assets               will delete public/assets before calling webpack
      --dry-run                    runs the build 'dry'
      --environment string         set the environment for the binary (default "development")
  -e, --extract-assets             extract the assets and put them in a distinct archive
  -h, --help                       help for build
      --ldflags string             set any ldflags to be passed to the go build
      --mod string                 -mod flag for go build
  -o, --output string              set the name of the binary
  -k, --skip-assets                skip running webpack and building assets
      --skip-build-deps            skip building dependencies
      --skip-template-validation   skip validating templates
  -s, --static                     build a static binary using  --ldflags '-linkmode external -extldflags "-static"'
  -t, --tags string                compile with specific build tags
  -v, --verbose                    print debugging information

ERRO[0000] Error: open cmd/app/main.go: no such file or directory

You’ll need to create a cmd/app directory and move main.go into the cmd/app directory.

This will get the build working again but it will break buffalo dev as it could no longer find the main file in the project directory. To fix that, you’ll need to open up .buffalo.dev.yml and add the following property:

build_target_path: "./cmd/app"

This will get the dev build working again.

I don’t know why the dev command honours this config, yet the build command chooses to look at a hard coded path. Wouldn’t it have been easier to express this in a single configuration file?

And let’s not leave Node out of the cold. If you’re trying to run buffalo build and you’re getting this error:

#21 12.21 node:internal/crypto/hash:71
#21 12.21   this[kHandle] = new _Hash(algorithm, xofLen);
#21 12.21                   ^
#21 12.21 
#21 12.21 Error: error:0308010C:digital envelope routines::unsupported
#21 12.21     at new Hash (node:internal/crypto/hash:71:19)
#21 12.21     at Object.createHash (node:crypto:133:10)
#21 12.21     at BulkUpdateDecorator.hashFactory (/src/bocce_scorecard/node_modules/webpack/lib/util/createHash.js:145:18)
#21 12.21     at BulkUpdateDecorator.update (/src/bocce_scorecard/node_modules/webpack/lib/util/createHash.js:46:50)
#21 12.21     at RawSource.updateHash (/src/bocce_scorecard/node_modules/webpack/node_modules/webpack-sources/lib/RawSource.js:77:8)
#21 12.21     at NormalModule._initBuildHash (/src/bocce_scorecard/node_modules/webpack/lib/NormalModule.js:888:17)
#21 12.21     at handleParseResult (/src/bocce_scorecard/node_modules/webpack/lib/NormalModule.js:954:10)
#21 12.21     at /src/bocce_scorecard/node_modules/webpack/lib/NormalModule.js:1048:4
#21 12.21     at processResult (/src/bocce_scorecard/node_modules/webpack/lib/NormalModule.js:763:11)
#21 12.21     at /src/bocce_scorecard/node_modules/webpack/lib/NormalModule.js:827:5 {
#21 12.21   opensslErrorStack: [ 'error:03000086:digital envelope routines::initialization error' ],
#21 12.21   library: 'digital envelope routines',
#21 12.21   reason: 'unsupported',
#21 12.21   code: 'ERR_OSSL_EVP_UNSUPPORTED'
#21 12.21 }

You’ll need to enable the legacy OpenSSL provider using a Node option:

export NODE_OPTIONS=--openssl-legacy-provider

Yeah, building to a framework is always fun. 😏

So that’s it for this weeks update. I spent some time on Dynamo-Browse this week as well, but I haven’t actually finished that work and this log entry is long enough, so I might say more about that next week.

Updates To Dynamo-Browse And CCLM

Sun, 19 Mar 2023 21:46:18 +1000

I started this week fearing that I’d have very little to write today. I actually organised some time off over the weekend where I wouldn’t be spending a lot of time on side projects. But the week started with a public holiday, which I guess acted like a bit of a time offset, so some things did get worked on.

That said, most of the work done was starting or continuing things in progress, which is not super interesting at this stage. I’ll hold off on talking about those until there’s a little more there. But there were a few things that are worth mentioning.

Dynamo-Browse

I found a bug in the query planner. It had to do with which index it chose when planning a query with only a single attribute. If a table has multiple GSIs that have that same attribute as the partition key (with different attributes for sort keys), the index the planner choose became effectively random. Because each index may have different records, running that query could give incomplete results.

I think the query planner needs to be fixed such that any ambiguity in which index to be use would result in an error. I try to avoid putting an unnecessary need for the user to know that a particular query required a particular index. But I don’t think there’s any getting around this: the user would have to specify.

But how to allow the user to specify the index to use?

The fix for the script API was reasonably simple: just allow the script author to specify the index to use in the form of an option. That’s effectively what I’ve done by adding an optional index field to the session.query() method. When set, the specific index would be used regardless of which index the query planner would choose.

I’m not certain how best to solve this when the user is running a query interactively. My current idea is that a menu should appear, allowing the user to select the index to use from a list. This could also include a “scan” option if no index is needed. Ideally this information will be stored alongside the query expression so that pressing R would rerun the query without throwing up the prompt again.

Another option is allowing the user to specify the index within the expression in some way. Maybe in the form of a hint, as in having the user explicitly specify the sort key in a way that does’t affect the output. This is a little hacky though — sort of like those optimisations you need to do in SQL queries to nudge the planner in a particular execution plan.

Another option is having the user specify the index specifically in the query. Maybe as an annotation:

color="blue" @index('color-item-index')

or as a suffix:

color="blue" using index('color-item-index')

Anyway, this will be an ongoing thing I’m sure.

One other thing I started working on in Dynamo-Browse is finally working on support for the between keyword:

age between 12 and 24

This maps directly to the between statement in DynamoDB’s query expression language, so getting scan support for this was relatively easy. I do need to make the query planner know of this though, as this operation is supported in queries if it’s used with the sort key. So this is still on a branch at the moment.

Finally, I’ve found myself using this tool a lot this last week and I desperately need something akin to what I’ve been calling a “fanout” command. This is a way to take the results of one query and use them in someway in another query — almost like sub-queries in regular SQL. What I’ve been finding myself wishing I could use this for is getting the IDs of the row from a query run over the index, and just running a query for rows with those ID over the main table. At the moment I’m left with copying the ID from the first result set, and just making a large pk in (…) expression, which is far from ideal.

I’m not sure whether I’d like to do this as a command, or extend the query expression in some way. Both approaches have advantages and disadvantages. That’s probably why I haven’t made any movement on this front yet.

CCLM

I did spend the Monday working on CCLM. I coded up a small script which took some of the ideas from the blog post on puzzle design I mention last week that I could run to get some ideas. So far it’s only producing suggestions with two game elements, but it’s enough of a starting point for making puzzles:

leonmika@Stark cclm % go run ./cmd/puzzleidea
bear trap
directional walls

After running it on Monday I had a go at starting work on a new level. It became clear reasonably soon after I started that I needed a new game element. So I added one, which I’ve called “kindling”. By default it looks like a pile of wood, and is perfectively safe to walk on:

But if a fireball runs into it, it catches alight and spreads to any adjacent kindling tiles, turning them into fire tiles.

I had an idea for this for a while. I even went to the extend of producing the graphics for this element. But needing it for this puzzle finally bought me around to finishing the work. I actually manage to make most of the changes without any changes to the Go code at all: the existing tile definition configuration was almost powerful enough to represent this tile.

One other minor thing I fixed was the alignment of the info panels on the right side of the screen. Dealing with the unaligned numbers got a bit much eventually. The cursor position, marker position, and tag numbers are properly aligned now.

Anyway, that’s all for this week.

Dev Log - 2023-03-12

Sun, 12 Mar 2023 09:12:28 +1000

Preamble

When I moved Working Set over to Micro.blog, I’d thought I’d be constantly writing micro-posts about what I’m working on, as a form of working in public. I found that didn’t really work for me, for a few reasons.

I’ve got a strange relationship with this blog. I wanted a place online to write about the projects I’ve been working on, but every time I publish something here, I always get the feeling that I’m “showing off” in some way: ooh, look what I’ve done, aren’t I cleaver? And okay, I’d be lying if there’s not a part of me that wants others to see how I spend my time. If I didn’t want that, I’d be content with these posts existing in a private journal.

And maybe this is a form of self-justification but I’d like to think that there’s a bit of that feeling in every developer that keeps a public blog on what they do. Maybe not exactly “showing off”, but I’m sure they feel proud on what they work on and they want to talk about it. And there’s really nothing wrong with that. In fact, the posts I tend to enjoy the most are those from other devs talking about the projects they’re working on.

So yeah, I admit that having others see what I’m working on would be nice. They say write what you want to read, and this is my attempt at doing just that.

But that only explains why I write about it on a public blog instead of a private journal. Why I should want to write these posts at all is that I’d like to keep a record of the the projects I work on. Nostalgia is one reason: seeing a project progress over time or remembering projects long since abandoned. But another might be a way to track where I’m spending my time. This is theoretical at the moment, but if there ever came a time when I wanted to find this out, I have to have the record written somewhere.

But not as micro-posts. I think a fixed weekly cadence is more appropriate. I tried this a couple of years ago, and although it worked for a while, I fell out of the habit. But after seeing the weekly entries by Jonathan Hays, I’ve been inspired to try it again.

So that’s why I’m trying these weekly update. They’ll be frequent enough to be useful to act like diary entries, but not so frequent that they will bother people who aren’t interested. They’ll be long enough to warrant a title, making it easy for people to skip it. And they’ll be any anything related to a side project I’m working on: either current or abandoned, public or completely private. And I’m giving myself permission not to feel bad about it.

Anyway, we’ll see how we go.

Update 6th July: I’ve gone back to just making posts when I have something to write about.

Dynamo-Browse

Big week for Dynamo-Browse: I finally got v0.2.0 out the door. This is the release with scripting support (yes, it finally shipped). The scripting implementation has been finished for a while. The thing that was blocking it’s release was all the documentation I had to write: both the section in the manual and the API reference.

The build was also a bit of an issue. The release builds are built using GitHub actions. To get them published as Homebrew casks, the actions need to push them to another repository. The secret token used to access this repository expired, and I had to create another one. Not difficult, but the fact that I had to create a whole new secret instead of rotate the existing one was a little annoying. Getting the permissions right, and being forced to choose a different name (“Deploy Homebrew formulas v2”) didn’t help matters either.

But got there in the end. The v0.2.0 release is now available on dynamobrowse.app and GitHub.

I’ll reduce the time I spend on this for a little while. We’ll see how long that lasts. I use this tool for work so often and I’ve got a whole list of features I’d like to see added to it.

CCLM

I got the editor up and running again last week and I spent Saturday designing a level with the working name “The Beach”. I’m a huge fan of the Developing series on the Game Makers Toolkit YouTube channel, and the latest video was about how difficult it was for Mark to design levels for his video game. I found I had the exact same problem for designing levels for mine (although I think the lack of effort I put into it doesn’t help). He pointed to a blog post by an indie game designer that had some useful tips to help with puzzle design. The one about using two elements that interact with it.

The one I worked on was for a custom element that will change boxes with a square indicator to blank tiles when pressing the yellow button. I’ve had this element around for a while but I haven’t actually used it in a level yet. I’d thought I’ll be time to do so, but the level I came up with seems a little simple. Not sure what I’d do about it. I could either rearrange it so that it appears earlier in the level set, or I can make it a little more difficult in some way.

Client Project

One thing that releasing Dynamo-Browse has given me is the opportunity to do a small client project. I’ve talked about this on lmika.org and the latest update is that I think I’ve convinced him to consider a static site, seeing that it would be easier for him to run (don’t need to worry about plugins) and would be easier for me to build (I don’t know how to use Wordpress, especially not their new block editor).

This week was basically coming up with a site layout. I had the opportunity to use Figma for the first time. Works reasonably well, but I’m wondering if Balsamiq Mockups was probably a better choice for a rough outline of what the site is to look like. But that’s all moot: a layout was put together and sent to the client for him to get some feedback.

Anyway, still early days here. I’m looking at possible Hugo templates to build the site in and possible hosting solutions that would work with the client. I’m not aware of options for static hosting other than the AWS, Cloudflare or Azures of the world. Not sure it will work for the client, although it’s totally possible that I’m just not looking in the right places.

So that’s it. Update one done. Although next week I’ll be taking some leave so update two might be slightly shorter (at least there’ll be no preamble) so it may be less about current updates. I guess we’ll find out together.

Wed, 08 Mar 2023 20:52:12 +1000

Completed the release of Dynamo-Browse 0.2.0. Most of the work in the last week was updating the manual, especially the scripting API. Some more updates need to be made for the query expressions as well, but I’ll publish what I have now and update that over time.

Sat, 04 Mar 2023 15:55:32 +1000

Here’s a bit of a blast from the past. I managed to get ccedit working again. This was the original level editor for workingset.net/2022/12/2… my Chips Challenge “fan game” I’ve been working on.

I’ve been designing a few levels for it recently, but since moving to a new Mac, the level editor I was using was going to be difficult to port. It’s QT application and the QT bindings were a pain to setup, and I rather not go through that again. I was using a Mac at the time I started working on it, but I wasn’t yet ready to go all in on MacOS. So to hedge my bets, I decided to go with QT as the UI toolkit.

This was 5 years ago and I’m unlikely to go back to Linux, so choosing QT was a bit of a bad decision. I think if I had my time again, I’d go with something like AppKit.

Anyway, the level editor still works but I have to log into a screen share to use it. I’d like to be able to edit levels on the machine I’m using now.

The code for the original level editor was still around but it hasn’t been touched in ages. It’s basically an SDL application — the same graphics library I’m using for the actual game itself — and the SDL v2 bindings I’m using are still maintained, so updating those were quite easy¹.

One thing I did have to pull out was the Lua VM². The editor was using old C Lua bindings. Better Lua VMs written in pure Go are now available, so I didn’t want to keep using these old bindings anymore. In fact, I didn’t want to use Lua at all. Lua was originally used for the level scripts, but I replaced this in favour of another language (which is no longer maintained 😒, but I’m not changing it again).

The original CCLM Editor

So far the editor boots up, but that’s about it. I can move the cursor around but I can’t add new tiles or load existing levels. There seems to be some weird things going on with the image name lookup. I originally thought image name were case insensitive, but after looking at the image name lookup logic in the game itself, I’m not so sure.

How much time I’d like to spend on this is still a bit of a question. It all depends whether I’d like to release the game itself in some fashion. There are still questions about whether I’m allowed to, given that the graphics are not my own. Still need to think about that.

But in any case, good to see the old editor again.

The level editor was actually using older SDL v1 bindings, but it was relatively easy to port them over to v2, although some gaps are still present. ↩︎
Lua was actually the second language used by the editor. The first was a Go native TCL interpretor. ↩︎

Wed, 22 Feb 2023 21:45:13 +1000

Spent some time closing off the Dynamo-Browse shortlist. I think I’ve got most of the big ticket items addressed. Here’s a brief update on each one:

Fix the activity indicator that is sometimes not clearing when a long running task is finished.

How long running tasks are dealt with has been completely overhauled. The previous implementation had many opportunities for race conditions, which was probably the cause of the activity indicator showing up when nothing was happening. I rewrote this using a dedicated goroutine for handling these tasks, and the event bus for sending events to the other areas of the app, including the UI layer. Updates and status changes are handled with mutexes and channels, and it just feels like better code as well.

It will need some further testing, especially in real world use against a real DynamoDB database. We’ll see if this bug rears its unpleasant head once more.

Fix a bug in which executing a query expression with just the sort key does nothing. I suspect this has something to do with the query planner somehow getting confused if the sort key is used but the partition key is not.

Turns out that this was actually a problem with the “has prefix” operator. It was incorrectly determining that an expression of the form sort_key ^= "string" with no partition key could be executed as a query instead of a scan. Adding a check to see if the partition key also existed in the expression fixed the problem.

Also made a number of other changes to the query expression. Added the ability to use indexed references, like this[1] or that["thing"]. This has been a long time coming so it’s good to see it implemented. Unfortuntly this only works reliably when a single level is used, so this[1][2] will result in an error. The cause of this is a bug in the Go SDK I’m using to produce the query expressions that are run against the database. If this becomes a problem I look at this again.

I also realised that true and false were not treated as boolean literals, so I fixed that as well.

Finally, the query planner now consider GSIs when it’s working out how to run a query expression. If the expression can be a query over a GSI, it will be executed as one. Given the types of queries I need to run, I’ll be finding this feature useful.

Fix a bug where set default-limits returns a bad value.

This was a pretty simple string conversion bug.

Add a way to describe the table, i.e. show keys, indices, etc. This should also be made available to scripts. Add a way to “goto” a particular row, that is select rows just by entering the value of the partition and optionally the sort key.

These I did not do. The reason is that they’ll make good candidates for scripts and it would be a good test to see if they can be written as one. I think the “goto” feature would be easy enough. I added the ability to get information about the current table in the script, and also for scripts to add new key bindings, so I don’t force any issues here.

The table description would be trickier. There’s currently no real way to display a large block of text (except the status bar, but even there it’s a little awkward). So a full featured description might be difficult. But the information is there, at least to a degree, so maybe something showing the basics would work.

Anyway, the plan now is to use this version for a while to test it out. Then cut a release and update the documentation. That’s a large enough task in and of itself, but I’d really like to get this finished so I can move onto something else.

Sat, 21 Jan 2023 08:18:18 +1000

Looking at the “backlog” of things to work on for Dynamo-Browse before I set it aside. I’ll fix a few bugs and add a few small features that I’ve found myself really wanting. The short list is as follows:

Fix the activity indicator that is sometimes not clearing when a long running task is finished.
Fix a bug in which executing a query expression with just the sort key does nothing. I suspect this has something to do with the query planner somehow getting confused if the sort key is used but the partition key is not.
Fix a bug where set default-limits returns a bad value.
Add a way to describe the table, i.e. show keys, indices, etc. This should also be made available to scripts.
Add a way to “goto” a particular row, that is select rows just by entering the value of the partition and optionally the sort key.

I’ll start with these and see how I go.

Oh, and one more thing: I will need to kill my darlings, namely the other commands in the “audax” repository that I’ve hacked togeather. They’re mildly useful — one of them is used to browse SSM parameters and another is used to view JSON log files — but they’re unloved and barely functional. I’ll move them out of the “audax” repository and rename this repo to “dynamo-browse”, just to make it less confusing for everyone.

Fri, 20 Jan 2023 15:19:26 +1000

I think I’ll take a little break from Dynamo-Browse. There’s a list of small features that are on my TODO list. I might do one or two of them over the next week, then cut and document a release, and leave it for a while.

I’m still using Dynamo-Browse pretty much every day at work, but it feels a little demotivating being the only person that’s using it. Even those at work seem like they’ve moved on. And I can understand that: it’s not the most intuitive bit of software out there. And I get the sense that it’s time to do something new. Maybe an online service or something. 🤔

Sat, 14 Jan 2023 08:21:55 +1000

Finally bit the bullet and got scripting working in Dynamo-Browse. It’s officially in the tool, at least in the latest development version. It’s finally good to see this feature implemented. I’ve been waffling on this for a while, as the last several posts can attest, and it’s good to see some decisions made.

In the end I went with Tamarin as the scripting language. It was fortunate that the maintainer released version 1.0 just as I was about to merge the scripting feature branch into main. I’ve been trying out the scripting feature at work and so far I’ve been finding it to work pretty well. It helps that the language syntax is quite close to Go, but I also think that the room to hide long-running tasks from the user (i.e. no promises everywhere) dramatically simplifies how scripts are written.

As for the runtime, I decided to have scripts run in a separate go-routine. This means they don’t block the main thread and the user can still interact with the tool. This does mean that the script will need to indicate when a long running process is occurring — which they can do by displaying a message in the status line — but I think this is a good enough tradeoff to avoid having a running script lock-up the app. I still need to add a way for the user to kill long-running scripts (writing a GitHub ticket to do this now).

At the moment, only one script can run at any one time, sort of like how JavaScript in the browser works. This is also intentional, as it will prevent a whole bunch of scripts launching go-routines and slowing down the user experience. I think it will help in not introducing any potential synchronisation issues with parallel running scripts accessing the same memory space. No need to build methods in the API to handle this. Will this mean that script performance will be a problem? Not sure at this stage.

I’m also keeping the API intentionally small at this stage. There are methods to query a DynamoDB table, get access to the result set and the items, and do some basic UI and OS things. I’m hoping it’s small enough to be useful, at least at the start, without overwhelming script authors or locking me into an API design. I hope to add methods to the API over time.

Anyway, good to see this committed to.

Poking Around The Attic Of Old Coding Projects

Wed, 28 Dec 2022 15:27:00 +1000

I guess I’m in a bit of a reflective mood these pass few days because I spent the morning digging up an old project that was lying dormant for several years. It’s effectively a clone of Chips Challenge, the old strategy game that came with the Microsoft Entertainment Pack. I was a fan of the game when I was a kid, even though I didn’t get through all the levels, and I’ve tried multiple times to make a clone of it.

The earliest successful clone I can think of was back when I was using Delphi, which I think was my teens. It’s since been lost but I do recall having a version that work and was reasonably true to the original game as possible. It wasn’t a particularly accurate clone: I do recall some pretty significant bugs, and the code itself was pretty awful. But it was nice to be able to do things like design my own levels (I wasn’t as internet savvy back then and I didn’t go looking for level editors for the Microsoft’s release of Chips Challenge). Eventually I stopped working on it, and after a few updates to the family computer, plus a lack of backups or source control, there came a time where I lost it completely.

Years later, I made another attempt at building a clone. I was dabbling in .Net at the time and I think I was working on it as an excuse to learn C#. I think I got the basics of the game and associated level editor working but I didn’t get much further than that. Either I got bored and stopped working on it.

I started the latest clone nine years ago. I can’t remember the original motivation. I was just getting into Go at the time and I think it was both to learn how to build something non-trivial in the language, and to determine how good Go was for building games. Although this is probably just a rationalisation: I’m sure the real reason was to work on something fun on the side.

Screenshot of CCLM, the latest clone.

Over the first five years of its life or so, I worked on it on and off, adding new game elements (tiles, sprites, etc.) and capabilities like level scripts. One thing I am particularly proud of was building a mini-language for selecting game elements using something akin to CSS selectors. Want to select all wall tiles? Use the selector SOLD. How about configuring a water tile to only allow gliders and the player but only if they have the flipper? Set the Immune attribute of the water tile to GLID,PLYR:holding(flippers). This was particularly powerful when working on tile and sprite definitions.

A sample of how some of the tiles are defined, including usage of the selectors.

I didn’t put as much effort into content however. As of today, there are only 18 or so unique levels, and about half of them are ones that I consider good. I certainly put little effort into the graphics. Many of the tile images were just taken from the original tile-set and any additional graphics were basically inspirations from that. This blatant copyright violation is probably why this project won’t see the light of day.

The level editor, and one of the rare times when it's not crashing.

I’m impressed on how Go maintains its backwards capability: moving from 1.13 to 1.19 was just a matter of changing the version number in the .mod file. I haven’t updating any of the libraries, and I’m sure the only reason why it still builds is because I haven’t dared to try.

I’ll probably shouldn’t spend a lot of time on this. But it was fun to revisit this for a while.

One final thing: I might write more about projects I’ve long since abandoned or have worked on and haven’t released, mainly for posterity reasons but also because I like reflecting on them later. You never know what you’d wish you documented until you’ve lost the chance to do so.

Tue, 27 Dec 2022 15:15:06 +1000

Spent the day restyling the Dynamo-Browse website. The Terminal theme was fun, but over time I found the site to be difficult to navigate. And if you consider that Dynamo-Browse is not the most intuitive tool out there, an easy to navigate user manual was probably important. So I replaced that theme with Hugo-Book, which I think is a much cleaner layout. After making the change, and doing a few small style fixes, I found it to be a significant improvement.

I also tried my hand at designing a logo for Dynamo-Browse. The blue box that came with the Terminal theme was fine for a placeholder, but it felt like it was time for a proper logo now.

I wanted something which gave the indication of a tool that worked on DynamoDB tables while also leaning into it’s TUI characteristics. My first idea was a logo that looked like the DynamoDB icon in ASCII art. So after attempting to design something that looks like it in Affinity Designer, and passing it through an online tool which generated ASCII images from PNG, this was the result:

I tried adjusting the colours of final image, and doing a few things in Acorn to thicken the ASCII characters themselves, but there was no getting around the fact that the logo just didn’t look good. The ASCII characters were too thin and too much of the background was bleeding through.

So after a break, I went back to the drawing board. I remembered that there were actually Unicode block characters which could produce filled-in rectangles of various heights, and I wondered if using them would be a nice play on the DynamoDB logo. Also, since the Dynamo-Browse screen consists of three panels, with only the top one having the accent colour, I thought having a similar colour banding would make a nice reference. So I came up with this design:

And I must say, I like it. It does look a little closer to low-res pixel art than ASCII art, but what it’s trying to allude to is clear. It looks good in both light mode and dark mode, and it also makes for a nice favicon.

That’s all the updates for the moment. I didn’t get around to updating the screenshots, which are in dark-mode to blend nicely with the dark Terminal theme. They actually look okay on a light background, so I can probably hold-off on this until the UI is changed in some way.

Fri, 23 Dec 2022 10:25:27 +1000

I’ve been resisting using mocks in the unit tests of Dynamo-Browse, but today I finally bit the bullet and started adding them. There would have just been too much scaffolding code that I needed to write without them. I guess we’ll see if this was a wise decision down the line.

Thinking About Scripting In Dynamo-Browse, Again

Thu, 22 Dec 2022 14:39:16 +1000

I’m thinking about scripting in Dynamo-Browse. Yes, again.

For a while I’ve been using a version of Dynamo-Browse which included a JavaScript interpreter. I’ve added it so that I could extend the tool with a few commands that have been useful for me at work. That branch has fallen out of date but the idea of a scripting feature has been useful to me and I want to include it in the mainline in some way.

The scripting framework works, but there are a few things that I’m unhappy about. The first is around asynchronicity and scheduling. I built the scripting API around the JS event-loop included in interpreter. Much like a web-browser or Node.js, this event-loop allows the use of promises for operations that can be dispatched asynchronously. The interpreter, however, is not ES6 compatible, which means no async/await keywords. The result is that many of the scripts that I’ve been writing are littered with all these then() chains. Example:

const session = require("audax:dynamo-browse/session");
const ui = require("audax:dynamo-browse/ui");
const exec = require("audax:x/exec");

plugin.registerCommand("cust", () => {
    ui.prompt("Enter UserID: ").then((userId) => {
        return exec.system("lookup-customer-id.sh", userId);
    }).then((customerId) => {
        let userId = output.replace(/\s/g, "");        
        return session.query(`pk="CUSTOMER#${customerId}"`, {
            table: `account-service-dev`
        });
    }).then((custResultSet) => {
        if (custResultSet.rows.length == 0) {
            ui.print("No such user found");
            return;            
        }
        session.resultSet = custResultSet;
    });
});

Yeah, I know; this is nothing new if you’ve been doing any web-dev or Node in the past, but it still feels a little clunky exposing the execution details to the script writer. Should that be something they should be worried about? Feels like the tool should take on more here.

The second concern involves modules. The JavaScript interpreter implements the require() function which can be used to load a JS module, much like Node.js. But the Node.js standard library is not available. That’s not really the fault of the maintainers, and to be fair to them, they are building out native support for modules. But that support isn’t there now, and I would like to include some modules to do things like access the file system or run commands. Just adding them with non-standard APIs and with name that are the same as equivalent modules in node Node.js feels like a recipe for confusion.

Further exacerbating this is that script authors may assume that they have access to the full Node standard library, or even NPM repositories, when they won’t. I certainly don’t want to implement the entire Node standard library from scratch, and even if the full library was available to me, I’m not sure the use of JavaScript here warrants that level of support.

Now, zooming out a little, this could all be a bit of a non-issue. I’ve haven’t really shared this functionality with anyone else, so all this could be of no concern to anyone else other than myself. But even so, I’m am thinking of options other than JavaScript. A viable alternative might be Lua — and Go has a bunch of decent interpreters — but I’m not a huge fan of Lua as a language. Also, Lua’s table structure being used for both arrays and structures seems like a source of confusion, especially when dealing with JSONish data structures like DynamoDB items.

One interpreter that has caught my eye is Tamarin. It’s early in its development already it’s showing some promise. It offers a Go like syntax, which is nice, along with native literals for lists, sets and maps, which is also nice. There’s a bit of a standard library already in place to do things like string and JSON operations. There’s not really anything that interacts with the operating system, but this is actually an advantage as it will mean that I’m free to write these modules myself to do what I need. The implementation looks simple enough which means that it will probably play nicely with Go’s GC and scheduler.

How the example above could look in Tamarin is given below. This assumes that the asynchronous aspects are completely hidden from the script author, resulting in something a little easier to read:

ext.command("cust", func() {
    var userId = ui.prompt("Enter UserID: ")
    
    var commandOut = exec.system("lookup-customer-id.sh", userId).unwrap()
    var userId = strings.trim_space(commandOut)
    
    var custResultSet = session.query(`pk=$1`, {
        "table": "account-service-dev",
        "args": [userId]
    }).unwrap()
    
    if len(custResultSet.items.length) == 0 {
        ui.print("No such user found")
        return
    }
    
    session.set_result_set(custResultSet)
})

So it looks good to me. However… the massive, massive downside is that it’s not a well-known language. As clunky as JavaScript is, it’s at least a language that most people have heard of. That’s a huge advantage, and one that warrants thinking twice about it before saying “no, thanks”. I’m not expecting Dynamo-Browse to be more popular than sliced bread (I can count on one hand the number of user’s I know are using it), but it would be nice if scripting was somewhat approachable.

One thing in Tamarin’s favour is that it’s close enough to Go that I think others could pick it up relatively easily. It’s also not a huge language — the features can be describe on a single Markdown page — and the fact that the standard library hasn’t been fully fleshed out yet can help keep things small.

I guess the obvious question is why not hide the scheduling aspects from the JavaScript implementation? Yeah, I think that’s a viable thing to do, although it might be a little harder to do than Tamarin. That will still leave the module exception issue though.

So that’s where I am at the moment. I’m not quite sure what I’ll do here, but I might give Tamarin a go, and see if it result in scripts that are easier to read and write.

One could argue that this pontificating over something that will probably only affect me and a handful of other people is even worth the time at all. And yeah, you’re probably right in thinking that it isn’t. To that, I guess the only thing I can respond with is that at least I got a blog post out of it. 🙂

One last concern I do have, regardless of what language I choose, is how to schedule the scripts. If I’m serious about not exposing the specifics of which thread/goroutine is waiting for what to the script author, I’d like to run the script in a separate goroutine from the main event loop. The concern is around start-up time: launching a goroutine is fast, but if I want to execute a script in response to a keystroke, for example, would it be fast enough? I guess this is something I should probably test first, but if it is a little unresponsive, the way I’m thinking of tackling this is having a single running goroutine waiting on a channel for script scheduling events. That way, the goroutine will always be “warm” and they’ll be no startup time associated with executing the script. Something to think about.

Project Exploration: A Check-in App

Sun, 18 Dec 2022 08:22:40 +1000

I’m in a bit of a exploratory phase at the moment. I’ve set aside Dynamo-Browse for now and looking to start something new. Usually I need to start two or three things before I find something that grabs me: it’s very rare that I find myself something to work on that’s exciting before I actually start working on it. And even if I start something, there’s a good chance that I won’t actually finish it.

So it might seem weird that I’d write about these explorations at all. Mainly I do it for prosperity reasons: just something to record so that when look back and think “what was that idea I had?”, I have something to reflect on.

With that out of the way, I do have a few ideas that are worth considering. Yesterday, I’ve started a new Flutter project in order to explore an idea for an app that’s been rattling around in my head for about a month or two.

Basically the idea is this: I keep a very simple check-in blog where I post check-ins for locations that seem useful to recall down the line. At the moment I post these check-ins manually, and usually a few hours after I leave in order to maintain some privacy of my movements. I’ve been entertaining the idea of an app that would do this: I can post a check-in when I arrive but it won’t be published several hours after I leave. The check-ins themselves follow a consistent format so the actual act of filling in the details can be handled by the app by presenting these properties like a regular form. At the same time, I wanted to try something with the Micropub format and possibly the IndieAuth protocol.

So I started playing around with a Flutter app that could do this. I got to the point where I can post new check-ins and they will show up in a test blog. Here are some screenshots:

The home screen, which obviously needs a bit of work. Pressing the floating action button will create a new checkin.

Selecting the type of check-in.

Each type will have different properties. After selecting the check-in type, they will be presented to you in a form.

How it will ultimately look on the blog.

It’s still very early stages at this moment. For one thing, check-ins are published as soon as you press “Checkin”. Eventually they should be stored on the device and then published at a later time. The form for entering checkins should also be slightly different based on the checkin type. Restaurants and cafes should offer a field to enter a star rating, and airport checkins should provide a way to describe your route.

I’m not sure if I will pursue this any further. We’ll see how I feel. But it was good to get something up and running in a couple of hours.

Dynamo-Browse Running With iSH

Sun, 27 Nov 2022 09:15:30 +1000

Bit of a fun one today. After thinking about how one could go about setting up a small dev environment on the iPad, I remembered that I actually had iSH installed. I’ve had for a while but I’ve never really used it since I never installed tools that would be particularly useful. Thinking about what tools I could install, I was curious as to whether Dynamo-Browse could run on it. I guess if Dynamo-Browse was a simple CLI tool that does something and produces some output, it wouldn’t be to difficult to achieve this. But I don’t think I’d be exaggerating if I said Dynamo-Browse is a bit more sophisticated than your run-of-the-mill CLI tool. Part of this is finding out not only whether building it was possible, but whether it will run well.

Answering the first question involves determining which build settings to use to actually produce a binary that worked. Dynamo-Browse is a Go app, and Go has some pretty decent cross-compiling facilities so I had no doubt that such settings existed. My first thought was a Darwin ARM binary since that’s the OS and architecture of the iPad. But one of the features of iSH is that it actually converts the binary through a JIT before it runs it. And it turns out, after poking around a few of the included binaries using file, that iSH expects binaries to be ELF 32-bit Linux binaries.

Thus, setting GOOS to linux and GOARCH to 386 will produced a binary that would run in iSH:

GOOS=linux GOARCH=386 go build -o dynamo-browse-linux ./cmd/dynamo-browse/

After uploading it to the iPad using Airdrop and launching in in iSH, success!

So, is runs; but does is run well? Well, sadly no. Loading and scanning the table worked okay, but doing anything in the UI was an exercise in frustration. It takes about two seconds for the key press to be recognised and to move the selected row up or down. I’m not sure what the cause of this is but I suspect it’s the screen redrawing logic. There’s a lot of string manipulation involved which is not the most memory efficient. I’m wondering if building the app using TinyGo would improve things.

But even so, I’m reasonably impress that this worked at all. Whether it will mean that I’ll be using iSH more often for random tools I build remains to be seen, but at least the possibility is there.

Update: While watching a re:Invent session on how a company moved from Intel to ARM, they mentioned a massive hit to performance around a function that calculates the rune length of a Unicode character. This is something that this application is constantly doing in order to layout the elements on the screen. So I’m wondering if this utility function could be the cause of the slowdown.

Update 2: Ok, after thinking about this more, I think the last update makes no sense. For one thing, the binary iSH is running is an Intel one, and although iSH is interpreting it, I can’t imagine the slowdowns here are a result of the compiled binary. For another, both Intel and ARM (M1) builds of Dynamo-Browse work perfectly well on desktops and laptops (at least on MacOS systems). So the reason for the slowdown must be something else.

Nee Audax Toolset

Mon, 21 Nov 2022 21:38:23 +1000

I’ve decided to retire the Audax Toolset name, at least for the moment. It was too confusing to explain what it actually was, and with only a single tool implemented, this complexity was unnecessary.

The project is now named after the sole tool that is available: Dynamo-Browse. The site is now at dynamobrowse.app, although the old domain will still work. I haven’t renamed the repository just yet so there will still be references to “audax”, particularly the downloads section. I’m hoping to do this soon.

I think I’m happy with this decision. It really now puts focus on the project, and removes the feeling that another tool had to be built just to satisfy the name.

Sun, 13 Nov 2022 08:22:38 +1000

Most of what’s going on with Audax and Dynamo-Browse is “closing the gap” between the possible queries and scans that can be performed over a DynamoDB table, and how they’re represented in Dynamo-Browse query expression language. Most of the constructs of DynamoDB’s conditions expression language can now be represented. The last thing to add is the size() function, and that is proving to be a huge pain.

The reason is that the IR representation is using the expression builder package to actually put the expression together. These builders uses Go’s type system to enforce which constructs work with each other one. But this clashes with how I built the IR representation types, which are essentially structs implement a common interface. Without having an overarching type to represent an expression builder, I’m left with either using a very broad type like any, or completely ditching this package and doing something else to build the expression.

It feels pretty annoying reaching the brick wall just when I was finishing this off. But I guess them’s the breaks.

One other thing I’m still considering is spinning out Dynamo-Browse into a separate project. It currently sits under the “Audax” umbrella, with the intention of releasing other tools as part of the tool set. These tools actually exist¹ but I haven’t been working on them and they’re not in a fit enough state to release them. So the whole Audax concept is confusing and difficult to explain with only one tool available at the moment.

I suppose if I wanted to work on the other tools, this will work out in the end. But I’m not sure that I do, at least not now. And even if I do, I’m now beginning to wonder if building them as TUI tools would be the best way to go.

So maybe the best course of action is to make Dynamo-Browse a project in it’s own right. I think it’s something I can resurrect later should I get to releasing a second tool.

Edit at 9:48: I managed to get support for the size function working. I did it by adding a new interface type with a function that returns a expression.OperandBuilder. The existing IR types representing names and values were modified to inherit this interface, which gave me a common type I could use for the equality and comparison expression builder functions.

This meant that the IR nodes that required a name and literal value operand — which are the only constructs allowed for key expressions — had to be split out into separate types from the “generic” ones that only worked on any OperandBuilder node. But this was not as large a change as I was expecting, and actually made the code a little neater.

Dynamo-Browse was actually the second TUI tool I made as part of what was called “awstools”. The first was actually an SQS browser. ↩︎

Audax Toolset Version 0.1.0

Sat, 22 Oct 2022 10:20:36 +1000

Audax Toolset version 0.1.0 is finally released and is available on GitHub. This version contains updates to Dynamo-Browse, which is still the only tool in the toolset so far.

Here are some of the headline features.

Adjusting The Displayed Columns

Consider a table full of items that look like the following:

pk           S    00cae3cc-a9c0-4679-9e3a-032f75c2b506
sk           S    00cae3cc-a9c0-4679-9e3a-032f75c2b506
address      S    3473 Ville stad, Jersey , Mississippi 41540
city         S    Columbus
colors       M    (2 items)
  door       S    MintCream
  front      S    Tan
name         S    Creola Konopelski
officeOpened BOOL False
phone        N    9974834360
ratings      L    (3 items)
  0          N    4
  1          N    3
  2          N    4
web          S    http://www.investorgranular.net/proactive/integrate/open-source

Let’s say you’re interested in seeing the city, the door colour and the website in the main table which, by default, would look something like this:

There are a few reasons why the table is laid out this way. The partition and sort key are always the first two columns, followed by any declared fields that may be used for indices. This is followed by all the other top-level fields sorted in alphabetical order. Nested fields are not included as columns, and maps and list fields are summarised with the number of items they hold, e.g. (2 items). This makes it impossible to only view the columns you’re interested in.

Version 0.1.0 now allows you to adjust the columns of the table. This is done using the Fields Popup, which can be opened by pressing f.

While this popup is visible you can show columns, hide them, or move them left or right. You can also add new columns by entering a Query Expression, which can be used to reveal the value of nested fields within the main table. It’s now possible to change the columns of the table to be exactly what you’re interested in:

Read-only Mode And Result Limits

Version 0.1.0 also contains some niceties for reducing the impact of working on production tables. Dynamo-Browse can now be started in read-only mode using the -ro flag, which will disable all write operations — a useful feature if you’re paranoid about accidentally modifying data on production databases.

Another new flag is -default-limit which will change the default number of items returned from scans and queries from 1000 to whatever you want. This is useful to cut down on the number of read capacity units Dynamo-Browse will use on the initial scans of production tables.

These settings are also changeable from while Dynamo-Browse using the new set command:

Progress Indicators And Cancelation

Dynamo-Browse now indicates running operations, like scans or queries, with a spinner. This improves the user experience of prior versions of Dynamo-Browse, which gave no feedback of running operations whatsoever and would simply “pop-up” the result of such operations in a rather jarring way.

With this spinner visible in the status bar, it is also now possible to cancel an operation by pressing Ctrl-C. You have the option to view any partial results that were already retrieved at the time.

Other Changes

Here are some of the other bug-fix and improvements that are also included in this release:

Audax toolset is now distributed via Homebrew. Check out the Downloads page for instructions.
A new mark command to mark all, unmark all, or toggle marked rows. The unmark command is now an alias to mark none.
Query expressions involving the partition and sort key of the main table are now executed as a DynamoDB queries, instead of scans.
The query expression language now supports conjunction, disjunction, and dot references.
Fixed a bug which was not properly detecting whether MacOS was in light mode. This was making some highlighted colours hard to see while in dark mode.
Fixed the table-selection filter, which was never properly working since the initial release.
Fixed the back-stack service to prevent duplicate views from being pushed.
Fixed some conditions which were causing seg. faults.

Full details of the changes can be found on GitHub. Details about the various features can also be found in the user manual.

Finally, although it does not touch on any of the features described above, I recorded a introduction video on the basics of using Dynamo-Browse to view items of a DynamoDB table:

No promises, but I may record further videos touching on other aspects of the tool in the future. If that happens, I’ll make sure to mention them here.¹

Or you can like, comment or subscribe on YouTube if that’s your thing 😛. ↩︎

Thu, 20 Oct 2022 21:01:23 +1000

Putting the final touches on the website for the upcoming release of Audax Toolset v0.1.0, and I’m finding myself a bit unhappy with it. Given that Dynamo-Browse is the only tool in this “suite”, it feels weird putting together a landing page with a whole lot of prose about this supposed collection of tools. There’s no great place to talk more about Dynamo-Browse right there on the landing page.

Part of me is wondering whether it would be better focusing the site solely on Dynamo-Browse, and leave all this Audax Toolset stuff on the back-burner, at least until (or unless) another tool is made available through this collection. I’m wondering if I’ll need to rearrange the codebase to do this, and spin out the other commands currently in development into separate repositories.

Bridging The Confidence Gap

Thu, 20 Oct 2022 13:06:14 +1000

I had to do some production work with DynamoDB this morning. It wasn’t particularly complicated work: run a query, get a couple of rows, change two attributes on each one. I could have used Dynamo-Browse to do this. But I didn’t. Despite building a tool designed for doing these sorts of things, and using it constantly for all sorts of non-prod stuff, I couldn’t bring myself to use it on a production database.

I’m wondering why this is. It’s not like I have any issues with using Dynamo-Browse for the non-prod stuff. Sure there a bugs, and things that the tool can’t do yet, but I haven’t (yet) encountered a situation where the tool actually corrupted data. I also make sure that the code that touches the database is tested against a “real” instance of DynamoDB (it’s actually a mock server running in Docker).

Yet I still don’t have the confidence to use it on a production database. And okay, part of me understands this to a degree. I’ve only been using this tool for a handful of months, and when it comes to dealing with production data, having a sense of caution, bordering on a small sense of paranoia, is probably healthy. But if I can’t be confident that Dynamo-Browse will work properly when it matters the most, how can I expect others to have that confident?

So I’d like to get to the bottom of this. After a bit of thought, I think there are three primary causes of my lack of confidence here.

The first is simple: there are features missing in the tool that are needed to do my job. Things such as running a query over an index, or the ability to impose a conditional check on updates. This is a legitimate concern in my opinion: if you need to do something, and Dynamo-Browse doesn’t support it, then you can’t use Dynamo-Browse to do that thing, plain and simple. It’s also the easiest concern to address: either add the feature, or say that feature is unsupported and don’t expect people to use Dynamo-Browse if they need it.

I think the second cause is a lack of insight into what the tool is actually doing. Even though I built the tool and tested it during development, there’s always this feeling of uncertainty in the back of my head while I’m using it when the stakes are high. That feeling of “when I do this, although I think the tool will behave this way, how will the tool actually behave?” Part of this, I think, comes from a decade and a half of being burned by technology as part of my job, and growing a healthy (?) sense of distrust for it.¹

I’m sure some of this will resolve itself as I continue to use Dynamo-Browse, but the only approach I know of gaining assurance that the tool is working as intended is if the tool tells me what it’s doing. Logging will help here, but I think some features that provide the user the ability to check what’s going to happen prior to actually doing it would be useful as well.

The third cause is probably a lack of controlled end-to-end testing. There does exists a suite of unit tests with a decent (maybe not good, but decent) level of coverage, but it does not extend to the UI layer, which is a little problematic. It might be that more testing of the application as a whole would help here.

This means more manual testing, but it might also be possible to setup some automated testing of the entire tool end-to-end here as well. What’s going for me is the fact that Dynamo-Browse runs in the terminal, and is responsible for rendering the UI and handling the event loop instead of palming this off to the OS. Looking at some of the features that Bubble Tea offers, it might be possible to run the application headless, and simulate use by pumping in keyboard events. Verifying that the UI is such may be a little difficult, but what I can test is what is actually read from and written to the database, which is what I’m mostly interested in.

I’m still unsure as to what I’ll do to address these three causes of concern. This is still a hobby project that I do in my spare time, and some of the mundane tasks, like more manual testing, sound unappealing. On the other hand, I do want to use this tool for my job, and I want others to use it as well. So I can’t really complain that others choose not to use it if I cannot feel comfortable using it when it matters most. So I probably should do some of each. And who knows, I may actually get some enjoyment out of doing it. I certainly would get some enjoyment from knowing that others can rely on it.

Someone once said that an insane person is one that does the same thing twice and expects different results. That someone has never worked in technology. ↩︎

Things Dynamo-Browse Need

Thu, 29 Sep 2022 11:17:56 +1000

I’m working with dynamo-browse a lot this morning and I’m coming up with a bunch of usability shortcomings. I’m listing them here so I don’t forget.

The query language needs an in operator, such as pk in ("abc", "123"). This works like in in all the other database services out there, in that the expression effectively becomes `pk = “abc” or pk = “123”. Is this operator supported natively in DynamoDB? 🤔 Need to check that.

Likewise, the query language needs something to determine whether an attribute contains a substring. I believe DynamoDB expressions support this natively, so it probably makes sense to use that.

Longer term, it would be nice to include the results of the current result set in the expression. For example, assuming the current result set has these records:

pk    sk    thing     place   category
11    aa    rock      home    geology
22    bb    paper     home    art
33    cc    scissors  home    utensils

and you want to effectively query for the rows where pk is equal to the set of pk in the current result set, having a way to do that in the expression language would save a lot of copy-and-pasting. An example might be pk in @pk or something similar, which could produce a result set of the form:

pk    sk    thing      place  category
11    aa    rock       home   geology
11    ab    sand       beach  ~
22    bb    paper      home   art
22    bc    cardboard  shops  ~
33    cc    scissors   home   utensils
33    cd    spoon      cafe   ~

Another way to do this might be to add support for filters, much like the expressions in JQ. For example, the second result set could be retrieved just by using a query expression of the form place = "home" | fanout pk, which could effectively do the following pesudo-code:

firstResultSet := query("place = 'home')
secondResultSet := newResultSet()
for pk in resultSet['pk'] {
    secondResultSet += query("pk = ?", pk)
}

For dealing with workspaces, a quick way to open up the last workspace that you had opened would be nice, just in case you accidentally close Dynamo-Browse and you want to restore the last session you had opened. Something like screen -r. I think in this case having workspace files stored in the temporary directory might be a problem. Maybe having some way to set where workspace files are stored by default would work? 🤔

For dealing with marking rows, commands to quickly mark or unmark rows in bulk. There’s already the unmark command, but something similar for marking all rows would be good. This could be extended to only mark (or unmark) certain rows, such as those that match a particular substring (then again, using filters would work here as well).

It might be nice for refresh to keep your current row and column position, instead of going to the top-left. And it would also be nice to have a way to refresh credentials (could possiably be handled using the scripting framework).

And finally, if I have any doubt that the feature to hide or rearrange columns would not be useful, there was an instance today where I wish this feature was around. So, keep working on that.

I’m sure there will be more ideas. If so, I’ll post them here.

Overlay Composition Using Bubble Tea

Sat, 24 Sep 2022 13:58:24 +1000

Working on a new feature for Dynamo-Browse which will allow the user to modify the columns of the table: move them around, sort them, hide them, etc. I want the feature to be interactive instead of a whole lot of command incantations that are tedious to write. I also kind of want the table whose columns are being manipulated to be visible, just so that the affects of the change would be apparent to the user while they make them.

This is also an excuse to try something out using Bubble Tea — the TUI framework I’m using — which is to add the ability to display overlays. These are UI elements (the overlay) that appear above other UI elements (the background). The two are composed into a single view such that it looks like the overlay is obscuring parts of the background, similar to how windows work in MacOS.

Bubble Tea doesn’t have anything like this built in, and the way views are rendered in Bubble Tea doesn’t make this super easy. The best way to describe how views work is to think of them as “scanlines.” Each view produces a string with ANSI escape sequences to adjust the style. The string can contain newlines which can be used to move the cursor down while rendering. Thus, there’s no easy way position the cursor at an arbitrary position and render characters over the screen.¹

So, I thought I’d tackle this myself.

Attempt 1

On the surface, the logic for this is simple. I’ll render the background layer up to the top most point of the overlay. Then for each scan line within the top and bottom border of the overlay, I’ll render the background up to the overlay’s left border, then the contents of the overlay itself, then the rest of the background from the right border of the overlay.

My first attempt was something like this:

line := backgroundScanner.Text()
if row >= c.overlayY && row < c.overlayY+c.overlayH {
    // When scan line is within top & bottom of overlay
    compositeOutput.WriteString(line[:c.foreX])
    
    foregroundScanPos := row - c.overlayY
    if foregroundScanPos < len(foregroundViewLines) {
        displayLine := foregroundViewLines[foregroundScanPos]
        compositeOutput.WriteString(lipgloss.PlaceHorizontal(
            c.overlayW,
            lipgloss.Left,
            displayLine,
            lipgloss.WithWhitespaceChars(" ")),
        )
    }
    
    compositeOutput.WriteString(line[c.overlayX+c.overlayW:])
} else {
    // Scan line is beyond overlay boundaries of overlay
    compositeOutput.WriteString(line)
}

Here’s how that looked:

Yeah, not great.

Turns out I forgot two fundamental things. One is that the indices of Go strings works on the underlying byte array, not runes. This means that attempting to slice a string between multi-byte Unicode runes would produce junk. It’s a little difficult to find this in the Go Language Guide apart from this cryptic line:

A string’s bytes can be accessed by integer indices 0 through len(s)-1

But it’s relatively easy to test within the Go playground:

package main

import "fmt"

func main() {
    fmt.Println("世界😀😚🙃😇🥸😎"[1:6])                   // ��界
    fmt.Println(string([]rune("世界😀😚🙃😇🥸😎")[1:6]))   // 界😀😚🙃😇
}

The second issue is that I’m splitting half-way through an ANSI escape sequence. I don’t know how long the escape sequence is to style the header of the item view, but I’m betting that it’s longer than 5 bytes (the overlay is to be position at column 5). That would explain why there’s nothing showing up to the left of the overlay for most rows, and why the sequence 6;48;5;188m is there.

Attempt 2

I need to modify the logic so that zero-length escape sequences are preserved. Fortunately, one of Bubble Tea’s dependency is reflow, which offers a bunch of nice utilities for dealing with ANSI escape sequences. The function that looks promising is truncate.String, which will truncate a string at a given width.

So changing the logic slightly, the solution became this:

// When scan line is within top & bottom of overlay
compositeOutput.WriteString(truncate.String(line, uint(c.overlayX)))

foregroundScanPos := r - c.overlayY
if foregroundScanPos < len(foregroundViewLines) {
    displayLine := foregroundViewLines[foregroundScanPos]
    compositeOutput.WriteString(lipgloss.PlaceHorizontal(
        c.overlayW,
        lipgloss.Left,
        displayLine,
        lipgloss.WithWhitespaceChars(" "),
    ))
}

rightStr := truncate.String(line, uint(c.foreX+c.overlayW))
compositeOutput.WriteString(line[len(rightStr):])

The results are a little better. At least the left side of the overlay looked OK now:

But there are still problems here. Notice the [0m at the right side of the overlay on the selected row. I can assure you that’s not part of the partition key; take a look at the item view to see for yourself. And while you’re there, notice the header of the item view? That should be a solid grey bar, but instead it’s cut off at the overlay.

I suspect that rightStr does not have the proper ANSI escape sequences. I’ll admit that the calculation used to set rightStr is a bit of a hack. I’ll need to replace it with a proper way to detect the boundary of an ANSI escape sequence. But it’s more than just that. If an ANSI escape sequence starts off at the left side of the string, and continues on “underneath” the overlay, it should be preserved on the right side of the overlay as well. The styling of the selected row and the item view headers are examples of that.

Attempt 3

So here’s what I’m considering: we “render” the background underneath the overlay to a null buffer while recording any escape sequences that were previously set on the left, or were changed underneath the overlay. We also keep track of the number of visible characters that were seen. Once the scan line position reached the right border of the overlay, we replay all the ANSI escape sequences in the order that were found, and then render the right hand side of the scan line from the first visible character.

I was originally considering rendering these characters to a null reader, but what I actually did was simply count the length of visible characters in a loop. The function to do this looks like this:

func (c *Compositor) renderBackgroundUpTo(line string, x int) string {
    ansiSequences := new(strings.Builder)
    posX := 0
    inAnsi := false

    for i, c := range line {
        if c == ansi.Marker {
            ansiSequences.WriteRune(c)
            inAnsi = true
        } else if inAnsi {
            ansiSequences.WriteRune(c)
            if ansi.IsTerminator(c) {
                inAnsi = false
            }
        } else {
            if posX >= x {
                return ansiSequences.String() + line[i:]
            }
            posX += runewidth.RuneWidth(c)
        }
    }
    return ""
}

And the value set to rightStr is changed to simply used this function:

rightStr := c.renderBackgroundUpTo(line, c.foreX+c.overlayW)

Here is the result:

That looks a lot better. Gone are the artefacts from splitting in ANSI escape sequences, and the styling of the selected row and item view header are back.

I can probably work with this. I’m hoping to use this to provide a way to customise the columns with the table view. It’s most likely going to power other UI elements as well.

Well, there might be a way using ANSI escape sequences, but that goes against the approach of the framework. ↩︎

Intermediary Representation In Dynamo-Browse Expressions

Fri, 16 Sep 2022 22:37:20 +1000

One other thing I did in Dynamo-Browse is change how the query AST produced the actual DynamoDB call.

Previously, the AST produced the DynamoDB call directly. For example, if we were to use the expression pk = "something" and sk ^= "prefix", the generated AST may look something like the following:

The AST will then be traversed to determine whether this could be handled by either running a query or a scan. This is called “planning” and the results of this will determine which DynamoDB API endpoint will be called to produce the result. This expression may produce a call to DynamoDB that would look like this:

client.Query(&dynamodb.QueryInput{
    TableName: "my-table-name",
    KeyConditionExpression: "#0 = :0 and beings_with(#1, :1)",
    ExpressionAttributeNames: map[string]string{
        "#0": "pk",
        "#1": "sk",
    },
    ExpressionAttributeValues: map[string]types.AttributeValue{
         ":0": &types.StringAttributeValue{ Value: "something" },
         ":1": &types.StringAttributeValue{ Value: "prefix" },
    },
})

Now, instead of determining the various calls to DynamoDB itself, the AST will generate an intermediary representation, something similar to the following:

The planning traversal will now happen off this tree, much like it did over the AST.

For such a simple expression, the benefits of this extra step may not be so obvious. But there are some major advantages that I can see from doing this.

First, it simplifies the planning logic quite substantially. If you compare the first tree with the second, notice how the nodes below the “and” node are both of type “binOp”. This type of node represents a binary operation, which can either be = or ^=, plus all the other binary operators that may come along. Because so many operators are represented by this single node type, the logic of determining whether this part of the expression can be represented as a query will need to look something like the following:

First determine whether the operation is either = or ^= (or whatever else)
If it’s = and the field on the left is either a partition or sort key, this can be represented as a query
If it’s ^=, first determine whether the operand is a string, (if not, fail) and then determine whether the field on the left is a sort key. If so, then this can be query.
Otherwise, it will have to be a scan.

This is mixing various stages of the compilation phase in a single traversal: determining what the operator is, determining whether the operands are valid (^= must have a string operand), and working out how we can run this as a query, if at all. You can imagine the code to do this being large and fiddly.

With the IR tree, the logic can be much simpler. The work surrounding the operand is done when the AST tree is traverse. This is trivial: if it’s = then produce a “fieldEq”; if it’s ^= then produce a “fieldBeginsWith”, and so on. Once we have the IR tree, we know that when we encounter a “fieldEq” node, this attribute can be represented as a query if the field name is either the partition or sort key. And when we encounter the “fieldBeginsWith” node, we know we can use a query if the field name is the sort key.

Second, it allows the AST to be richer and not tied to how the actual call is planned out. You won’t find the ^= operator in any of the expressions DynamoDB supports: this was added to Dynamo-Browse’s expression language to make it easier to write. But if we were to add the “official” function for this as well — begins_with() — and we weren’t using the IR, we would need to have the planning logic for this in two places. With an IR, we can simply have both operations produce a “fieldBeginsWith” node. Yes, there could be more code encapsulated by this IR node (there’s actually less) but it’s being leverage by two separate parts of the AST.

And since expressions are not directly mappable to DynamoDB expression types, we can theoretically do things like add arithmetic operations or a nice suite of utility functions. Provided that these produce a single result, these parts of the expression can be evaluated while the IR is being built, and the literal value returned that can be used directly.

It felt like a few other things went right with this decision. I was expecting this to take a few days, but I was actually able to get it built in a single evening. I’m also happy about how maintainable the code turned out to be. Although there are two separate tree-like types that need to be managed, both have logic which is much simpler than what we were dealing with before.

All in all, I’m quite happy with this decision.

Letting Queries Actually Be Queries In Dynamo-Browse

Tue, 13 Sep 2022 22:44:18 +1000

I spent some more time working on dynamo-browse over the weekend (I have to now that I’ve got a user-base 😄).

No real changes to scripting yet. It’s still only me that’s using it at the moment, and I’m hoping to keep it this way until I’m happy enough with the API. I think we getting close though. I haven’t made the changes discussed in the previous post about including the builtin plugin object. I’m thinking that instead of a builtin object, I’ll use another module instead, maybe something like the following:

const ext = require("audax:ext");

ext.registerCommand("thing", () => console.log("Do the thing"));

The idea is that the ext module will provide access to the extension points that the script can implement, such as registering new commands, etc.

At this stage, I’m not sure if I want to add the concept of namespaces to the ext module name. Could help in making it explicit that the user is accessing the hooks of “this” extension, as opposed to others. It may leave some room for some nice abilities, like provide a way to send messages to other extensions:

// com.example.showTables
// This extension will handle events which will prompt to show tables. 
const ext = require("audax:ext/this");

ext.on("show-tables", () => {
    ui.showTablePrompt();
});

// com.example.useShowTables
// This extension will use the "show-tables" message "com.example.showTables"
const showTables = require("audax:ext/com.example.showTables");

showTables.post("show-tables");

Then again, this might be unnecessary given that the JavaScript module facilities are there.

Anyway, what I actually did was start work on making queries actually run as queries against the DynamoDB table. In the released version, running a query (i.e. pressing ? and entering a query expression) will actually perform a table scan. Nothing against scans: they do what they need to do. But hardly the quickest way to get rows from DynamoDB if you know the partition and sort key.

So that’s what I’m working on now. Running a query of the form pk = "something" and sk = "else" where pk and sk are the partition and sort keys of the table will now call the dynamodb.Query API. This also works with the “begins with” operator: pk = "something" and sk ^= "prefix". Since sk is the sort key, this will be executed as part of a query to DynamoDB.

This also works if you were to swap the keys around, as in sk = "else" and pk = "something". Surprisingly the Go SDK that I’m using does not support allow expressions like this: you must have the partition key before the sort key if you’re using KeyAnd. This touches on one of the design goals I have for queries: the user shouldn’t need to care how the expression actually produces the result. If it can do so by running an actual query against the table, then it will; if not, it will do a scan. Generally, the user shouldn’t care either way: just get me the results in the most efficient way you can think of!

That said, it might be necessary for the user to control this to an extent, such as requiring the use of a scan if the planner would normally go for a query. I may add some control to this in the expression language to support this. But these should be, as a rule, very rarely used.

Anyway, that’s the biggest change that’s happening. There is something else regarding expressions that I’m in the process of working on now. I’ll touch on that in another blog post.

Tue, 06 Sep 2022 13:28:55 +1000

Finished version 0.0.3 of Audax Toolset yesterday. The code has been ready since the weekend, but it took me Sunday morning and yesterday (Monday) evening to finish updating the website. All done now.

Now the question is whether to continue working on it, or do something different for a change. There are a few people using Dynamo-Browse at work now, so part of me feels like I should continue building features for it. But I also feel like switching to another project, at least for a little while.

I guess we’ll let any squeaky wheels make the decision for me.

Thu, 01 Sep 2022 13:39:54 +1000

I’ve been using Dynamo-Browse all morning and I think I’ll make some notes about how the experience went. In short: the command line needs some quality of life improvements. Changing the values of two attributes on two different items, while putting them to the DynamoDB table each time, currently results in too many keystrokes, especially given that I was simply going back and forth between two different values for these attributes.

So, in no particular order, here is what I think needs to be improved with the Dynamo-Browse command line:

It needs command line completion. Typing out a full attribute name is so annoying, especially considering that you need to be a little careful you got the attribute name right, lest you actually add a new one.
It needs command line history. Pressing up a few times is so much better than typing out the full command again and again. Such a history could be something that lives in the workspace, preserving it across restarts.
The set-attr and del-attr commands need a switch to take the value directly, rather than by prompting the user to supply it after entering the command (it can still do that, but have an option to take it as a switch as well). I think having a -set switch after the attribute names would suffice.

Finally, I think it might be time to consider adding more language features to the command line. At the moment the commands are just made up of tokens, coming from a split on whitespace characters (while supporting quotes). But I think it may be necessary to convert this into a proper language grammar, and add some basic control structures to it, such as entering multiple commands in a single line. It doesn’t need to be a super sophisticated language: something similar to the like TCL or shell would be enough at first.

It might be that writing a script would have solved this problem, and it would (to a degree at least). But not everything needs to be a script. I tried writing a script this morning to do the thing I was working on and it felt just so overkill, especially considering how short-lived this script would actually be. Having something that you can whip up in a few minutes can be a great help. It would have probably taken me 15-30 minutes to write the script (plus the whole item update thing hasn’t been fully implemented yet).

Anyway, we’ll see which of the above improvements I’ll get to soon. I’m kinda thinking of putting this project on hold for a little while, so I could work on something different. But if this becomes too annoying, I may get to one or two of these.

Thinking About Scripting In Dynamo-Browse

Wed, 31 Aug 2022 09:31:28 +1000

I’ve been using the scripting facilities of dynamo-browse for a little while now. So far they’ve been working reasonably well, but I think there’s room for improvement, especially in how scripts are structured.

At the moment, scripts look a bit like this:

const db = require("audax:dynamo-browse");
const exec = require("audax:x/exec");

db.session.registerCommand("cust", () => {
    db.ui.prompt("Enter UserID: ").then((userId) => {
        return exec.system("/Users/lmika/projects/accounts/lookup-customer-id.sh", userId);
    }).then((customerId) => {
        let userId = output.replace(/\s/g, "");        
        return db.session.query(`pk="CUSTOMER#${customerId}"`, {
            table: `account-service-dev`
        });
    }).then((custResultSet) => {
        if (custResultSet.rows.length == 0) {
            db.ui.print("No such user found");
            return;            
        }
        db.session.resultSet = custResultSet;
    });
});

This example script — modified from one that I’m actually using — will create a new cust command which will prompt the user for a user ID, run a shell script to return an associated customer ID for that user ID, run a query for that customer ID in the “account-service-dev”, and if there are any results, set that as the displayed result set.

This is all working fine, but there are a few things that I’m a little unhappy about. For example, I don’t really like the idea of scripts creating new commands off the session willy-nilly. Commands feel like they should be associated with the script that are implementing them, and this is sort of done internally but I’d like it to be explicit to the script writer as well.

At the same time, I’m not too keen of requiring the script writer to do things like define a manifest. That would dissuade “casual” script writers from writing scripts to perform one-off tasks.

So, I’m thinking of adding an plugin global object, which provides the hooks that the script writer can use to extend dynamo-browse. This will kinda be like the window global object in browser-based JavaScript environments.

Another thing that I’d like to do is split out the various services of dynamo-browse into separate pseudo packages. In the script above, calling require("audax:dynamo-browse") will return a single dynamo-browse proxy object, which provides access to the various services like running queries or displaying things to the user. This results in a lot of db.session.this or db.ui.that, which is a little unwieldily. Separating these into different packages will allow the script writer to associate them to different package aliases at the top-level.

With these changes, the script above will look a little like this:

const session = require("audax:dynamo-browse/session");
const ui = require("audax:dynamo-browse/ui");
const exec = require("audax:x/exec");

plugin.registerCommand("cust", () => {
    ui.prompt("Enter UserID: ").then((userId) => {
        return exec.system("/Users/lmika/projects/accounts/lookup-customer-id.sh", userId);
    }).then((customerId) => {
        let userId = output.replace(/\s/g, "");        
        return session.query(`pk="CUSTOMER#${customerId}"`, {
            table: `account-service-dev`
        });
    }).then((custResultSet) => {
        if (custResultSet.rows.length == 0) {
            ui.print("No such user found");
            return;            
        }
        session.resultSet = custResultSet;
    });
});

Is this better? Worse? I’ll give it a try and see how this goes.

Dynamo-Browse: Using The Back-Stack To Go Forward

Tue, 23 Aug 2022 23:00:11 +1000

More work on Audax toolset. Reengineered the entire back-stack in dynamo-browse to support going forward as well as backwards, much like going forward in a web-browser. Previously, when going back, the current “view snapshot” was being popped off the stack and lost forever (a “view snapshot” is information about the currently viewed table, including the current query and filter if any are set). But I’d need to maintain these view snapshot if I wanted to add going forward as well.

So this evening, I changed it this such that view snapshots were no longer being deleted. Instead, there is now a pointer to the current view snapshot that will go up or down the stack when the user wants to go backwards or forwards. This involved changing the back-stack from a single-linked list to a doubly-linked list, but that was pretty straightforward. The logic to pop snapshots from the stack hasn’t been removed: in fact, it has been augmented so that it can now remove the entire head up to the current view snapshot. This happens when the user goes backwards a few times, then runs a new query or sets a new filter, again much like a web-browser where going backwards a few times, then clicking a link, will remove any chance of going forward from that point.

Last thing I changed this evening was to restore the last viewed table, query or filter when relaunching dyanmo-browse using an existing workspace. This works quite nicely with the back-stack as it works now: it just needs to read the view snapshot at the head.

Adding Copy-To-Clipboard And Layout Changes In Dynamo-Browse

Sat, 20 Aug 2022 11:56:10 +1000

Spent some more time on dynamo-browse this morning, just a little bit.

Got one new feature built and merged to main, which is the ability to copy the displayed item to the pasteboard by pressing c. This is directly inspired by a feature in K9S, which allows you to copy the logs of a running pod to the pasteboard (I use this feature all the time). The package I’m using to access the pasteboard is github.com/golang-design/clipboard, which seems to offer a nice cross-platform approach to doing this.

The copied item is not meant for machine use. It’s exactly what’s displayed in the item viewport and is intended for tracking items that change over time. What I’ve found myself doing, when I need to track such an item, is to manually select it with Terminal and pasting it to an untitled textedit window. I do this again when the item changes until I’m ready to eye the changes manually. It’s not sophisticated, but it tends to work. And although this feature is not a full-blown item comparer, it will make this use case a little easier. Plus, it will copy the entire image, not just the lines displayed when a viewed item is too large for the viewport.

Speaking of which, the other thing I started working on today was on allowing the user to resize the item viewport to take up more of the screen. At the moment, the displayed item viewport takes up the bottom 14 lines of the screen, and is not resizable. Naturally, for large items, this is not large enough, and having the ability to resize this viewport would be a good thing to have.

I’ve been thinking about the best way to handle this, and I’ve decided that an approach similar to how Vim does this — where you need to press a key-chord followed by a number to move the vertical split between two views — would be a little to annoying here. Too much fine grain control for when you just need the viewport bigger. So, I’m opting for a more course-grain approach, where you simply press w and the viewport size will cycle amongst 5 different sizes:

14 rows at the bottom of the screen (the default)
Half the screen
All but the top 6 rows of the screen, so you can still see a bit of the table view
The entire screen (i.e. hiding the table view)
Completely hidden (i.e. the table view takes up the entire screen).

Doing this will not necessarily steal focus from the table view: even if shown the entire screen, you can still go through each item by pressing Up or Down. But you’ll be able to see more of the selected item.

Anyway, that’s the theory. We’ll see how well this works in practice.

Adding A Back-Stack To Dynamo-Browse

Sun, 14 Aug 2022 10:21:47 +1000

Spent some more time working on dynamo-browse, this time adding a back-stack. This can be used to go back to the previously viewed table, query or filter by pressing the Backspace key, kind of like how a browser back button works.

This is the first feature that makes use of a workspace, which is a concept that I’ve been thinking about since the start of the project. A workspace is basically a file storing various bits of state that could be recalled in future launches of the tool. A workspace is always created, even if one is not explicitly specified by the user. The workspace filename be set by using the -w switch on launch, or if one is not specified, then a new workspace filename within the temp directory is generated. The workspace file itself is a Bolt database, which is a very simple, embeddable key/value store that uses B-Trees and memory-mapped data access. I’m actually using StormDB to access this file, since it provides a nice interface for storing and querying Go structs without having to worry about marshalling or unmarshalling.

The back-stack consist of view-snapshots, which are essentially records of the currently view table, filter, and query. The view-snapshots are stored as a linked list, and the back-stack simply pushes new snapshots as the view changes (because this is stored on file, the backlinks are actually ID references and not physical pointers). I had a bit of trouble getting this right at first. The initial version actually pushed a snapshot of the previous view before it changed (i.e. a new table was selected or the filter was changed). But this resulted in a bit of messy implementation, with push calls copied and pasted across the various controllers with various special cases, etc.

So I went back and reimplemented it so that the top of the back-stack is actually a snapshot of the current view. Pressing the Backspace key will actually pop the current snapshot, then read the head of the stack and use that to update the view. This means that there will always be one item in the stack, which bothered me a little at first, but that did make the code easier to implement.

There is another benefit to this arrangement which just came to me while I was writing this. Since the current state of the view is always at the top of the stack, I could add the ability to restore the last view when launching dynamo-browse with an existing workspace, saving the user from selecting a table and running their last query. This will come a bit later. For the meantime, I need to get this back-stack working with user scripts, so that when a script changes the view, the user can still go back.

Finally, a shoutout to boltcli which is a useful CLI tool for browsing a Bolt database. This tool has come in handy for debugging this feature.

More On Scripting In Dynamo-Browse

Wed, 10 Aug 2022 14:45:01 +1000

Making some more progress on the adding scripting of dynamo-browse. It helps being motivated to be able to write some scripts for myself, just to be able to make my own working life easier. So far the following things can be done within user scripts:

Define new commands
Display messages to, and get input from, the user
Read the attributes of, and set the attributes of, the currently selected row (only string attributes at this stage)
Run queries and get result sets, which can optionally be displayed to the user

Here’s a modified example of a script I’ve been working on that does most of what is listed above:

const db = require("audax:dynamo-browse");

db.session.registerCommand("cust", () => {
    db.ui.prompt("Enter user ID: ").then((userId) => {
        return db.session.query(`userId="${userId}"`);
    }).then((resultSet) => {
        db.session.currentResultSet = resultSet;
    });
});

db.session.registerCommand("sub", () => {    
    let currentItem = db.session.selectedRow;

    let currentUserId = currentItem.item["userId"];
    if (!currentUserId) {
        db.ui.print("No 'userId' on current record");
        return;
    }

    db.session.query(`pk="ID#${currentUserId}"`, {
        table: "billing-subscriptions"
    }).then((rs) => {
        db.session.currentResultSet = rs;
    });
});

The process of loading this script is pretty basic at the moment. I’ve added a loadscript command, which reads and executes a particular JavaScript file taken as an argument. Scripts can be loaded on startup by putting these loadscript commands (along with any other commands) in a simple RC file that is read on launch.

I’m not sure how much I like this approach. I would like to put some structure around how these scripts are written and loaded, so that they’re not just running atop of one another. However, I don’t want too much structure in place either. To me, having the floor for getting a script written and executed too high is a little demotivating, and sort of goes against the ability to automate these one-off tasks you sometimes come across. That’s the downsides I see with IDEs like GoLand and, to a lesser degree, VS Code: they may have vibrant plugin ecosystems, but the minimum amount of effort to automate something is quite high, that it may not be worth the hassle.

A nice middleground I’m considering is to make it to have two levels of automation: one using JavaScript, and the other closer to something like Bash or TCL. The JavaScript would be akin to plugins, and would have a rich enough API and runtime to support a large number of sophisticated use-cases, with the cost being a fair degree of effort to get something usable. The Bash or TCL level would have a lower level of sophistication, but it would be quicker to do these one-off tasks that you can bang out and throw away. This would mean splitting the effort between the scripting API and command line language, which would fall on me to do, so I’m not sure if I’d go ahead with this (the command line “language” is little more than a glorified whitespace split at the moment).

In the meantime, I’ll focus on making sure the JavaScript environment is good enough to use. This will take some time and dog-fooding so they’ll be plenty to do.

I’ve also started sharing the tool with those that I work with, which is exciting. I’ve yet to get any feedback from them yet, but they did seem quite interested when they saw me using it. I said this before, but I’ll say it again: I’m just glad that I put a bit of polish into the first release and got the website up with a link I can share with them. I guess this is something to remember for next time: what you’re working on doesn’t need to be complete, but it helps that it’s in a state that is easy enough to share with others.

Scripting In Dynamo-Browse

Wed, 03 Aug 2022 21:34:37 +1000

Making some progress with adding scripting capabilities to dynamo-browse. I’ve selected a pretty full-featured JS interpreter called goja. It has full support for ECMAScript 5.1 and some pretty good coverage of ES6 as well. It also has an event loop, which means that features such as setTimeout and promises come out of the box, which is nice. Best of all, it’s written in Go, meaning a single memory space and shared garbage collector.

I’ve started working the extension interface. I came up with a bit of a draft during the weekend and now just implementing the various bindings. I don’t want to go too fast here. I’m just thinking about all the cautionary tales from those that were responsible for making APIs, and introducing mistakes that they had to live with. I imagine one way to avoid this is to play with the extension API for as long as it make sense, smoothing out any rough edges when I find them.

It’s still early days, but here are some of the test scripts I’ve been writing:

// Define a new "hello" command which will ask for your name and
// print it at the bottom.
const dynamobrowse = require("audax:dynamo-browse");

dynamobrowse.session.registerCommand("hello", () => {
    dynamobrowse.ui.prompt("What is your name? ").then((name) => {
       dynamobrowse.ui.alert("Hello, " + name);
    });
});

The prompt method here asks the user for a line of text, and will return the user’s input. It acts a bit like the prompt function in the browser, except that it uses promises. Unfortunately, Goja does not yet support async/await so that means that anonymous functions will need to do for the time being (at least Goja supports arrow functions).

Here’s another script, which prints out the name of the table, and set the value of the address field on the first row:

const dynamobrowse = require("audax:dynamo-browse");

dynamobrowse.session.registerCommand("bla", () => {
    let rs = dynamobrowse.session.currentResultSet;
    
    let tableName = rs.table.name;
    dynamobrowse.ui.alert("Current table name = " + tableName);
    
    rs.rows[0].item.address = "123 Fake St.";
});

The rows array, and item object, makes use of DynamicArray and DyanmicObject, which is a nice feature in Goja in which you do not need to set the elements in the array, or fields on the object explicitly. Instead, when a field or element is looked up, Goja will effectively call a method on a Go interface with the field name or element index as a parameter. This is quite handy, particularly as it cuts down on memory copies.

These scripts work but they do require some improvements. For example, some of these names could possibly be shortened, just to reduce the amount of typing involved. They also need a lot of testing: I have no idea if the updated address in the example above will be saved properly.

Even so, I’m quite happy about how quickly this is coming along. Hopefully soon I’ll be in a position to write the test script I’m planning for work.

Also, I got my first beta user. Someone at work saw me use dynamo-browse today and asked where he could get a copy. I’m just so glad I got the website finished in time.

Release Preparation & Next Steps

Sat, 30 Jul 2022 10:19:56 +1000

Finally finished the website for the Audax toolset and cut an initial release (v0.0.2). I’ve also managed to get binary releases for Linux and Windows made. I’ve started to work on binary releases for MacOS with the intention of distributing it via Homebrew. This is completely new to me so I’m not sure what progress I’ve made so far: probably nothing. For one thing, I’ll need a separate machine to test with, since I’ve just been installing them from source on the machine’s I’ve been using. Code signing is going to be another thing that will be fun to deal with.

I’ll slip that into a bit of a background task for the moment. For now, I’d like to start work on a feature for “dynamo-browse” that I’ve been thinking of for a while: the ability for users to extend it with scripts.

Some might argue that it may be a little early for this feature, but I’m motivated to do it now as I have a personal need for this, which is navigating amongst a bunch of DynamoDB tables that all store related data. This is all work related, so can’t say too much here. But if I were just to say that doing this now is a little annoying. It involves copying and pasting keys, changing tables, and running filters and queries manually: all possible, but very time consuming. Having the ability to do all this with a single command or keybinding would be so much better. And since this is all work related, I can’t simply modify the source code directly as it will give too much away. Perfect candidate for using an embedded scripting language.

So I think that’s what I’ll start work on next.

I also need to think about starting work on “sqs-browse”. This would be a complete rewrite of what is currently there: which is something that can be used for polling SQS queues and viewing the messages there. I’m hoping for something a little more sophisticated, involving a workspace for writing and pushing messages to queues, along with pulling, saving and routing them from queues. I haven’t got an immediate need for this yet, but I’ve come close on occasion so I can’t leave this too late.

A New Name for Audax Tools (nee AWS Tools)

Sun, 24 Jul 2022 16:08:58 +1000

I think I’ve settled on a name for the project I’ve been calling “awstools”. “Settled” is probably a good word for it: I came up with it about a week ago, and dismissed it at first as being pretty ordinary¹. But over that time, it’s been slowly growing on me. Also, I’ve yet to come up with any alternatives that are better.

Anyway, the name that I’ve settled on is the Audax Toolset.

It’s not a great name. Honestly, anyone with more creativity that me could probably come up with a bunch of names better than this. I’m going to play the name-is-the-thing-you-say-to-talk-about-the-thing here.

But the name was not completely random. “Audax” was chosen from the second part of the Latin name for a wedged-tail eagle: Aquila audax. Translated from Latin it means bold, and although I would definitely not describe this project as in any way “bold”, it is a TUI based application which, given the limited palette for building UIs, may have some instances of bold font. But what I’m trying to invoke with this name is the sense of an eagle flying in a cloudless sky. Cloudless → cloudy → the Cloud → AWS (yeah, like I said it’s not a great name).

Anyway, it’s good enough for the moment. And the domain name audax.tools was available, which was a good thing. Right now, the domain doesn’t point to anything but it will eventually point to the website and user guide.

Speaking of which, the user guide is still slowly coming along. I got a very rough draft of the actual guide itself finished. I’ll need to proofread it and flesh it out a little, and also add some screenshots. The reference, which will list all the commands and key bindings (there isn’t many of them), needs to be written as well, but that’s probably the easiest thing to write. I will admit that the guide itself makes for pretty dry reading, and I’m not entirely convinced it would be enough for someone coming in cold to download up the tools and start using them. I may also need to write some form of tutorial, or maybe even record a demo video. Something to consider once I’ve got the rest of the site finished, I guess.

I use the term “ordinary” here as a euphemism for “not very good”. ↩︎

AWS Tools: Documentation & The Website

Thu, 21 Jul 2022 22:23:41 +1000

Worked a little more on “awstools” (still haven’t thought of a good alternative name for it). I think the “dynamo-browse” tool is close to being in a releasable state. I’ve spent the last couple of days trying to clean up most of the inconsistencies, and making sure that it’s being packaged correctly.

Now it’s documentation writing time. I’m working my way through a very basic website and user guide. It’s been a little while since I’ve written any form of user-level documentation — most of the documents I write have been for other developers I work closely with — and I admit that it feels like a bit of a slog. It might be the tone of writing that I’ve adopted: a little dry and impersonal, trying to walk that fine line between being informative without swamping the reader with big blocks of words. I might need to work on that: no real reason why the documentation needs to be boring to the reader.

The website itself will be a statically generated site using Hugo and will most likely be served using GitHub pages. I’ve settled on the terminal theme, since “awstools” is a suite of terminal-based apps. That reasoning might be a little corny, but to be honest, I have grown to actually like the theme itself. I haven’t settled on a domain for it yet.

While working on the documentation, I ran into a useful website that contains a comprehensive list of HTML entities, complete with previews. Good reference for the arrows glyphs I need to use to represent key bindings in the document.

AWS Tools Dev Diary

Sat, 16 Jul 2022 11:54:42 +1000

A little more work on “awstools” today, mainly on a bit of a cleanup spree to make them suitable for others to use. This generally means fixing up any inconsistencies in how the commands work. An example of this is the put command, which now writes all modified items that are marked to the table (or if there are no marked items, all modified items) instead of just the selected one. This brings it closer to how the delete command works.

Also merged the set-s and set-n commands into a single set-attr command, which now has the ability to specify an optional attribute type along with the attribute name. This still only works with the currently selected item, and I think I’ll keep it like that for the moment. I do want something to modify attributes of all marked items (or even all items), but it might be better if that was a separate command, as it may allow for some potentially useful actions like adding a suffix to the value instead of simply changing it.

Some of these command names are a bit unwieldy, like set-attr, but I’m hopeful to replace many of these with simple keystrokes down the line. I’m trying not to reserve many generic names like “set” in the off chance of adding something like TCL or similar to make simple scripts (this is in addition to something closer to JavaScript for more feature-full extensions). Nothing settled here, but trying to keep that option open.

WWDC Videos In Broadtail

Mon, 16 May 2022 20:20:00 +1000

Some more work on Broadtail. This time, I added the ability to use it to download Apple WWDC videos.

The way it works is based on the existing RSS feed concept. In order to get the list of videos for a particular WWDC year, you “subscribe” to that by setting up a feed with the new “Apple Developer Videos” type. The external ID is taken from the URL slug of the web-site that Apple publishes the session videos. For example, for WWDC 2021, the external ID would be “wwdc2021”.

Downloading the videos is more or less the same.

There are a few differences between this feed type, and the YouTube RSS feed. For instance, it only makes use of what is available from the website, which means details like publishing date or duration are not really available. This is why the “Publishing” date is displayed as “unknown”. That’s also why the videos are arranged in alphabetical order and the feed itself is not automatically refreshed (although doing so manually by clicking “Refresh” within the feed page will work). These are actually properties that can now be applied to all feeds of one wishes, although the YouTube feeds are still arranged in reverse chronological order by default.

From a coding perspective, this involved a lot of refactoring. I was hoping to move to a more generic feed and video type, but this was the feature that eventually got me to do so. Thing is that if I wanted to add more feed and video types in the future, it should be easier to do so.

Feed Rules In Broadtail

Mon, 09 May 2022 20:17:00 +1000

Generally, when there’s a video that I’m interesting in watching, I take a look at Broadtail to see if it’s available. When it is, I go ahead and download it.

However, some videos take a long time to download — we’re talking 10 hours or so — and they’re usually published when I’m not looking, like during the night when I’m asleep (thank’s time-zones). So I’d thought it would be nice for Broadtail to kick off the download for me when the video shows up in the feed.

So I’ve added Feed Rules to do this.

Feed rules are very simple automations that happen when new items are found in during the RSS feed poll. When the video shows up in the feed, and matches the rule condition, Broadtail will perform the rule action for that video.

Feed Rules are added as a new sub-section in “Settings”, which itself is a new top-level section of the app (the “General” sub-section is empty at this stage).

Feed Rules consist of a name, whether the rule is active, a set of conditions, and a set of action. A feed item will need to match all the conditions of the rule in order for the actions to be performed.

The conditions of a feed rule touch upon the following properties of a feed item:

The feed in which it appears in. This can be set to “any” to apply the rule to all feed items.
Whether the title matches a given string. The match rules are similar to the searches in the feed item list views, which are appearance of each of the space separated tokens somewhere in the title (in any case) with phrases appearing as quoted strings.
Whether the description matches a given string.

If a feed item matches all the conditions, Broadtail can perform the following actions for the feed item:

Start a download of the video
Mark the feed item as a favourite

There might be more conditions or actions added in the future. So far this seems to be the bare minimum to make the feature usable.

Mon, 04 Apr 2022 20:12:00 +1000

Doing a small weekend/week-long project at the moment to track favourite moments in a few podcasts I’m listening to. This is something that I’ve been thinking about for a while, and I’m not entirely sure what compelled me to actually start work on it. Probably because the system I’ve been using so far — a set of timestamped Pocketcast links managed in Pinboard — has been growing quite recently and much of the limitations involved, such as the list being unordered and no skip back 30 seconds available on playback, is start to annoy me. It’s also a chance for a bit of novelty, at least for a few days or so.

It took roughly a day or so to get a small Buffalo web-app up and running which does most of what I want. It just needs some styling and a better way to play the episodes, which is what I’m working on now. I really don’t want to spend more than a week working on this — last thing I need are more projects. But a good thing about this one is that I think the scope is naturally quite small, so no real risk of it blowing out to become too large.

Fri, 01 Apr 2022 20:11:00 +1000

Two new awstool commands: one for browsing SSM parameters and one for simply viewing JSON log files. The SSM parameter one was especially handy, as I was dealing with parameter subtrees a lot and doing that in the AWS web console is always a pain. As for the JSON log viewer; well, let’s just say there were one too many log files from Kubernete pods I needed to look at this week.

The pattern for working with state seems to be working. I may need to be a little careful that the state management doesn’t get too unwieldily as I add features and more things that need to be tracked. But at the moment, it seems to be manageable.

Tue, 29 Mar 2022 20:11:00 +1000

I’ve been racking my brain trying to best work out how to organise the code for awstools. My goals are to make it possible to have view models composable, have state centralised but also localised, and keep controllers from having too much responsibility. I started another tool, which browses SSM parameters, to try and work this all out.

I think I’ve settled on the following architecture:

Providers and Services will remain stateless
State will be managed by controllers
Operations in controllers are only available through tea.Cmd implementations.
Updates from controllers will only be available through tea.Msg implementations.
View models (i.e. tea.Model) will only know enough state to be able to render themselves.
There will be one master model which will coordinate the communication between controllers and view models. This model will react to messages from the controllers and update the views. It will also react to messages from the views and launch operations on the controllers.

We’ll see how this goes and whether it will scale as additional features are added.

Sun, 27 Mar 2022 20:11:00 +1000

More work on the AWS tools, mainly rebuilding the UI framework. Need to rip out all the Operation type stuff, as BubbleTea already does this using messages and commands (see tutorial 2). Also taking some time building some UI models that I can reuse across the various commands, including a few that deal with layout changes.

Also tracked down what was causing that delay when trying to create a new list. It turns out during the call to list.New(), a bunch of adaptive styles are created, which includes a test to see if the terminal is in light or dark mode. This calls some terminal IO methods which were blocking for a significant amount of time, we’re talking in the order of 10’s of seconds.

The good thing is that this check is only made once, so what I did was move the check into the main function. Some preliminary tests indicating that this may work: the lists are consistently being created very quickly again. We’ll see if this lasts.

Sat, 26 Mar 2022 20:10:00 +1000

More work on Broadtail this morning. Managed to get the new favourite stuff finished. Favourites are now a dedicated entity, instead of being tied to a feed item. This means that any video can now be favourited, including ones not from a feed. The favourite list is now a top-level menu item as well.

Also found a useful CLI tool for browsing BoltDB files.

New AWS Tools Commands

Fri, 25 Mar 2022 20:05:00 +1000

For a while now, I’ve been wanting some tools which would help manage AWS resources that would also run in the terminal. I know in most circumstances the AWS console would work, but I know for myself, there’s a lot of benefit from doing this sort of administration from the command line.

I use the terminal a lot when I’m developing or investigating something. Much of the time while I’m in the weeds I’ve got a bunch of tabs with tools running and producing output, and I’m switching between them as I try to get something working across a bunch of systems.

This is in addition to cases when I need to manage an AWS mock running on the local machine. The AWS console will not work then.

At the start of the week, I was thinking of at least the following three tools that I would like to see exist:

A TUI browser/workspace for DynamoDB tables
A TUI workspace for monitoring SQS queues
Some tool for reading JSON style log files.

As of yesterday, I actually got around to building the first two.

The first is a tool for browsing DynamoDB tables, which I’m calling dynamo-browse (yes, the names are not great). This tool does a scan of a DynamoDB table, and shows the resulting items in a table. Each item can be inspected in full in the lower half of the window by moving the selection.

At the moment this tool only does a simple scan, and some very lightweight editing of items (duplicate and delete). But already it’s proven useful with the task I was working on, especially when I came to viewing the contents of test DynamoDB instances running on the local machine.

The second tool is used for monitoring SQS queues.

This will poll for SQS messages from a queue and display them in the table here. The message itself will be pretty printed in the lower half of the screen. Messages can also be pushed to another queue. That’s pretty much it so far.

There are a bunch of things I’d like to do in these tools. Here’s a list of them:

Queries in DynamoDB: There’s currently no way to run queries or filter the resulting items. I’m hoping to design a small query language to do this. I’m already using Participal to power a very simple expression language to duplicate items, so as long as I can design something that is expressive enough and knows how to use particular indices, I think this should work.
Putting brand new items in DynamoDB: At the moment you can create new items based on existing items in a way, but there’s currently no way to create brand new items or adjust the attributes of existing items. For this, I’d like to see an “edit item” mode, where you can select the attribute and edit the value, change the type, add or remove attributes, etc. This would require some UI rework, which is already a bit tentative at this stage (it was sort of rushed together, and although some architectural changes have been made to the M and the C in MVC, work on the V is still outstanding).
Preview item changes before putting them to DynamoDB: This sort of extends the point above, where you see the diff between the old item and new item before it’s put into DynamoDB.
Workspaces in SQS Browse: One idea I have for SQS browse is the notion of “workspace”. This is a persistent storage area located on the disk where all the messages from the queue would be saved to. Because SQS browse is currently pulling messages from the SQS queue, I don’t want the user to get into a state where they’ve lost their messages for good. The idea is that a workspace is always created when SQS browse is launch. The user can choose the workspace file explicitly, but if they don’t, then the workspace will be created in the temp directory. Also implicit in this is support for opening existing workspaces to continue work in them.
Multiple queues in SQS Browse: Something that I’d like to see in SQS Browse is the ability to deal with multiple queues. Say you’re pulling from one queue and you’d like to push it to another. You can use a command to add a queue to the workspace. Then, you can do things like poll the queue, push messages in the workspace to the queue, monitor it’s queue length, etc. Queues would be addressable by number or some other way, so you can simply run the command push 2 to push the current message to queue 2.

As for when I’ll actively work on these tools. It will probably be when I need to use them. But in the short term, I’m glad I got the opportunity to start working on them. They’ve already proven quite useful to me.

Tue, 22 Mar 2022 20:00:00 +1000

Even more work on Feed Journaler. Still trying to tune the title removal logic. Will probably require a lot of testing.

Broadtail 0.0.7

Tue, 15 Mar 2022 19:54:00 +1000

Released Broadtail 0.0.7 about a week ago. This included some restyling of the job list on the home page, which now includes a progress bar updated using web-sockets (no need for page refreshes anymore).

For the frontend, the Websocket APIs that come from the browser are used. There’s not much to it — it’s managed by a Stimulus controller which sets up the websocket and listen for updates. The updates are then pushed as custom events to the main window, which the Stimulus controllers used to update the progress bar are listening out for. This allows for a single Stimulus controller to manage the websocket connection and make use of the window as a message bus.

Working out the layers of the progress bar took me a bit of time, as I wanted to make sure the text in the progress bar itself was readable as the progress bar filled. I settled in a HTML tree that looked like the following:

<div class="progressbar">
  <!-- The filled in layer, at z-index: 10 -->
  <div class="complete">
    <span class="label">45% complete</span>
  </div>

  <!-- The unfilled layer -->
  <span class="label">45% complete</span>
</div>

As you can see, there’s a base layer and a filled in layer that overlaps it. Both of these layers have a progress label that contain the same status message. As the .complete layer fills in, it will hide the unfilled layer and label. The various CSS properties used to get this effect can be found here.

The backend was a little easier. There is a nice websocket library for Go which handles the connection upgrades and provides a nice API for posting JSON messages. Once the upgrade is complete, a goroutine servicing the connection will just start listening to status updates from the jobs manager and forward these messages as JSON text messages.

Although this works, it’s not perfect. One small issue is that updates will not reconnect if there is an error. I imagine that it’s just a matter of listening out for the relevant events and retrying, but I’ll need to learn more about how this actually works. Another thing is that the styling of the progress bar relies of fixed widths. If I get around to reskinning the style of the entire application, that might be the time to address this.

The second thing this release has is a simple integration with Plex. If this integration is configured, Broadtail will now send a request to Plex to rescan the library for new files, meaning that there’s no real need to wait for the schedule rescan to occur before the videos are available in the app. This simply uses Plex’s API, but it needs the Plex token, which can be found using this method.

Anyway, that’s it for this version. I’m working on re-engineering how favourites work for the next release. Since this is still in early development, I won’t be putting in any logic to migrate the existing favourites, so just be weary that you may loose that data. If that’s going to be a problem, feel free to let me know.

Some More Updates of Broadtail

Sat, 05 Mar 2022 19:49:00 +1000

I’ve made some more changes to Broadtail over the last couple of weeks.

The home page now shows a list of recently published videos below the currently running jobs.

Clicking through to “Show All” displays all the published videos. A simple filter can be applied to filter them down to videos with titles containing the keywords (note: nothing fancy with the filter, just tokenisation and an OR query).

Finally, items can now be favourited. This can be used to select videos that you may want to download in the future. I personally use this to keep the list of “new videos” in the Plex server these videos go to to a minimum.

PGBC Scoring Rules

Thu, 13 Jan 2022 22:55:00 +1000

I get a bit of a thrill when there’s a need to design a mini-language. I have one facing me now for a little project I’m responsible for, which is maintaining a scoring site for a bocce comp I’m involve in with friends.

How scoring works now is that the winner of a particular bocce match gets one point for the season. The winner for the season is the person with the most points. However, we recently discuss the idea of adding “final matches,” which will give the match winner 7 points, the runner up 2 points, and the person who came in third 1 point. At the same time I want to add the notion of “friendly matches” which won’t count to the season score.

It might have been that a simple solution was to encode these rules directly in the app, and have a flag indicating whether a match was normal, final or friendly. But this was suboptimal as there is another variant of the game we play which do not have the notion of finals, and if we did, we may eventually have different rule for it. So I opted for a design in which a new “match type” is added as a new database entity, which will have the scoring rules encoded as a PostgreSQL JSON column type. Using this as a mechanism of encoding free(ish) structured data when there’s no need to query it has worked for me in the past. There was no need to add the notion of seasons points as it was already present as an easy way to keep track of wins for a season.

For the scoring rules JSON structure, I’m considering the use of an array of conditions. When a player meets the conditions of a particular array element, they will be awarded the points associated with that condition. Each player will only be permitted to match one condition, and if they don’t match any, they won’t get any points. The fields of the condition that a player can be matched to can be made up of the following attributes:

rank: (int) the position the player has in the match just played in accordance with the scoring, with 1 being the player with the highest score, 2 being the player with the second highest score, and so on.
winner: (bool) whether the player is considered the winner of the match. The person with the highest score usually is, but this is treated as an independent field and so it should be possible to define rules accordingly.
draw: (bool) whether the player shares their rank with another player. When a draw occurs, both winning players will have a rank of 1, with the player of the second highest score having a rank of 2.

Using this structure, a possible scoring rules definition for a normal match may look like the following:

{ "season_score": [
  { "condition": { "winner": true }, "points": 1 }
]}

whereas a rules definition for the final match may look like the following:

{ "season_score": [
  { "condition": { "rank": 1 }, "points": 7 },
  { "condition": { "rank": 2 }, "points": 2 },
  { "condition": { "rank": 3 }, "points": 1 }
}]

Finally, for friendlies, the rules can simply look like the following:

{ "season_score": [] }

I think this provides a great deal of flexibility and extensibility without making the rules definition too complicated.

Alto Catalogue Update

Sat, 13 Nov 2021 21:55:00 +1000

I’ve really tied myself up in knots here. I’m spending some time working on Alto Catalogue, trying to streamline the process of uploading individual tracks into a new album. This is a workflow that is absolutely not user friendly at the moment, and the only way I’ve gotten tracks into the catalogue is to run a hacked-together tool to upload the tracks from the command line. The reason why I’m addressing this now is that it’s slightly embarrassing to have this open-source project without having a nice way of doing something that, by all accounts, is quite fundamental (a good hint for when you’re facing this is when it comes time to write the end-user documentation: if you can’t explain how to do something in a way that doesn’t include the word “hack”, “complicated”, or “unsupported”, then something is missing).

So I’m trying to close this feature gap, but it’s proving to be more complicated than I expected. The main issue relates ID3 tags and how media is arrange in the repository. Previous versions of the catalogue actually did have a way of uploading track media to the repository, which is essentially an S3 bucket. The way this work is that the catalogue will issue the browser a pre-signed Put URL, and the browser could upload the track media directly to S3. But in order to get a pre-signed URL, you need to know the object key, which is a bit like a file path. The old upload flow had the user enter the object key manually in the upload form.

This worked but I had some real issues with it. The first is that I’d like the objects within the S3 bucket to be organised in a nice way, for example “artist/album/tracknum-title.mp3”. I’m hoping that this S3 bucket will be my definitive music collection and I don’t want just some random IDs that are completely indecipherable when I browse the objects in the S3 bucket. That way, if I were ever to shutdown the catalogue down or loose all the metadata, I’d still be able to navigate my collection via the object keys alone.

The second was that this approach did not take into account the track metadata. Track metadata is managed in a PostgreSQL database and had to be entered in manually; yes, this included the track duration. The only reason I used the hacked together tool to upload tracks was that it was a tool I was already using to set ID3 tags on MP3 files, and it was trivial to add a HTTP client to do the upload from there. Obviously, asking users to run a separate tool to do their track uploads is not going to fly.

So I’m hoping to improve this. The ideal flow would be that the user will simply select an MP3 from their file system. When they click upload, the following things will happen:

The ID3 tags of the MP3 will be read.
That metadata will be used to determine the location of the object in S3.
A pre-signed URL will be generated and sent to the browser to upload the file.
The file is uploaded to S3.
A new track record is created with the same metadata.

The libraries I’m using to read the ID3 tags and track duration requires the track media to be available as a file on the local file system (I assume this is for random access). Simply uploading the track media to the local file system would be the easiest approach, since it would allow me to read the metadata, upload the media to the repository on the backend, and setup the track metadata all in a single transaction. But I have some reservations about allowing large uploads to the server, and most of the existing infrastructure already makes use of pre-signed URLs. So the first run at this feature involved uploading the file to S3 and then downloading it on the server backend to read the metadata.

But you see the problem here: in order to generate a pre-signed URL to upload the object to S3, I need to know the location of the media, which I want to derive from the track metadata. So if I don’t want uploads to go straight to the file system, I need the object to already be in S3 in order to work out the best location of where to put the object in S3.

So I’m wondering what the best ways to fix this would be. My current thing is this series of events:

Create a pre-signed URL to a temporary location in the S3 bucket.
Allow the user to Upload the media directly to that location in the S3 bucket.
On the server, download that media object to get the metadata and duration.
From that, derive the objects location and move the object within S3, something I’m guessing should be relatively easy if the objects are in the same bucket.
Create a new track record from the metadata.

The alternative is biting the bullet and allowing track uploads directly to the file system. That will simplify the crazy workflow above but means that I’ll need to configure the server for large uploads. This is not entirely without precedence though: there is a feature for uploading tracks in a zip file downloaded from a URL which uses the local file system. So there’s not a whole lot stopping me from doing this altogether.

The third approach might be looking for a JavaScript library to read the ID3 tags. This is not great as I’d need to get the location from the server anyway, as the metadata-derive object location is configured on a per repository basis. It also means I’ll be mixing up different ways to get metadata.

In any case, not a great set of options here.

Feeds In Broadtail

Sat, 06 Nov 2021 20:32:00 +1000

My quest to watch YouTube without using YouTube got a little closer recently with the addition of feeds in Broadtail. This uses the YouTube RSS feed endpoint to list videos recently added to a channel or playlist.

There are a bunch of channels that I watch regularly but I’m very hesitant to subscribe to them within YouTube itself (sorry YouTubers, but I choose not to smash that bell icon). I’m generally quite hesitant to give any signal to YouTube about my watching habits, feeding their machine learning models even more information about myself. But I do want to know when new videos are available, so that I can get them into Plex once they’re released. There is where feeds come in handy.

Also improved is the display of video metadata when selecting a feed item or entering a video ID in the quick look bar. Previously this would immediately start a download of the video, but I prefer knowing more about the video first. These downloads aren’t free, and they usually take many hours to get. Better to know more about them before committing to it.

Incidentally, I think this mode of watching has a slight benefit. There are days when I spend the whole evening binging YouTube, not so much following the algorithm but looking at the various channels I’m interested in for videos that I haven’t seen yet. Waiting several hours for a video download feels a little more measured, and less likely to send me down the YouTube rabbit hole. I’m sure there will still be evenings when I do nothing else other than watch TV, but hopefully that’s more of a choice rather than an accident.

I think this is enough on Broadtail for the time being. It’s more or less functional for what I want to do with it. Time to move onto something else.

Some Screenshots Of Broadtail

Sun, 31 Oct 2021 12:09:00 +1000

I spent some time this morning doing some styling work on Broadtail, my silly little YouTube video download manager I’m working on.

Now, I think it’s fair to say that I’m not a designer. And these designs look a little dated, but, surprisingly, this is sort of the design I’m going for: centered pages, borders, etc. A bit of a retro, tasteless style that may be ugly, but still usable(-ish).

It’s not quite finished — the colours need a bit of work — but it’s sort of the style I have in my head.

Sat, 23 Oct 2021 19:24:00 +1000

More work on the project I mentioned yesterday, codenamed Broadtail. Most of the work was around the management of download jobs. I’m using a job management library I’ve built for another project and integrated it here so that the video downloads could be observable from the web frontend. The library works quite well, but at the moment, the jobs are not kept on any sort of disk storage. They are kept in memory until they’re manually cleared, but I’m hoping to only keep the active jobs in memory, and store historical jobs onto disk. So most of today’s session was spent on making that possible, along with some screens to list and view job details.

Start of Yet Another Project Because I Can't Help Myself

Fri, 22 Oct 2021 19:16:00 +1000

One of the reasons why I stopped work on Lorikeet was that I was inspired by those on Micro.blog to setup a Plex server for my YouTube watching needs. A few years ago, I actually bought an old Intel Nuc for that reason, but I never got around to setting it up. I managed to do so last Wednesday and so far it’s working pretty well.

The next thing I’d like to do is setup RSS subscriptions for certain YouTube channels and automatically download the videos when they are publish. I plan to use “youtube-dl” for the actual video downloading part, but I’m hoping to build something that would poll the RSS feeds and trigger the download when new videos are published. I’m hoping that this service would have a web-based frontend so I don’t have to login via SSH to monitor progress, etc.

The download’s would need to be automatic as the requests made by youtube-dl seem to be throttled by YouTube and a longish video may take several hours to download. If this was a manual process, assuming that I would actually remember to start the download myself, the video won’t be ready for my evening viewing. I’m hoping that my timezone would work to my advantage here. The evenings on the US East Coast are my mornings, so if a video download starts at the beginning of the day, hopefully it would be finish when my evening rolls around. I guess we’ll see.

Anyway, that’s what my current coding project will be on: something that would setup RSS subscriptions for YouTube channels, and download new videos when they are published.

This is probably one of those things that already exist out there. That may be true, but there are certain things that I’m hoping to add down the line. One such thing might be adding the notion of an “interest level” to channels which would govern how long a video would be kept around. For example, a channel that is marked as very interested would have every video downloaded and stored into Plex straight away. Mildly interested channels would have videos download but kept in a holding place until I choose to watch it, in which case it would be moved to Plex. If that doesn’t happen in 7 days or so, the videos would be removed.

I’d like to also add some video lifecycle management into the mix as well, just to avoid the disk being completely used up. I can see instances where I’d like to mark videos as “keep for ever” and all the others will churn away after 14 days or so. It might be worth checking out what Plex offers for this, just to avoid doubling up on effort.

But that’s all for the future. For the moment, my immediate goal is to get the basics working.

Abandoning Project Lorikeet

Mon, 18 Oct 2021 19:58:00 +1000

I’ll admit it: the mini-project that I have been working on may not have been a good idea.

The project, which I gave the codename Lorikeet, was to provide a way to stream YouTube videos to a Chromecast without using the YouTube app. Using the YouTube app is becoming a real pain. Ads aside, they’ve completely replaced the Chromecast experience from a very basic viewing destination to something akin to a Google TV, complete with recommendations of “Breaking News” from news services that I have no interest in seeing.

So I spent some time trying to build something to avoid the YouTube app completely, using a mixture of youtube-dl, a Buffalo web-app, and a Flutter mobile app. I spent the last week on it (it’s not pretty so no screenshots), but at this stage I don’t see much point continuing to work on it.

For one, the experience is far from perfect. Video loading is slow and there are cases when the video pauses due to buffering. I’m sure there are ways around this, but I really don’t want to spend the time learning how to do this.

It was also expensive. I have a Linode server running in Sydney which acts as a bit of a hobby server (it’s also running Pagepark to serve this site); but in order to be closer to the YouTube CDNs that are closer to me, I had to rent a server that would run in Melbourne. And there are not many VPS hosting providers that offer hosting here.

So I went with Google Cloud.

Now, I’m sure there’s a lot to like about Google Cloud, but I found its VPS hosting to be quite sub-par. For just over $10 USD a month, I had a Linux virtual server with 512 MB of RAM, 8 GB of storage, and a CPU which I’d imagine is throttled all the way back as trying to do anything of significants slowed it to a crawl. I had immense issues installing OS updates, getting the Dokku based web-app deployed, and trying to avoid hitting the storage limit.

For the same amount of money, Linode offers me a virtual server with 2 GB of RAM, 50 GB of storage, and a real virtual CPU. This server is running 4 Dokku apps, 3 of them with dedicated PostgreSQL databases, and apart from occasionally needing to remove dangling Docker images, I’ve had zero issues with it. None! (The podcasters were right).

Where was I? Oh, yeah. So, that’s the reason why I’m abandoning this project and will need to re-evaluate my online video watching experience. I might give Plex a try, although before doing something like setting up a dedicated media server, I’ll probably just use the Mac Mini I’ve been using for a desktop in the short term.

So, yeah, that’s it. It’s hard to abandon a project you spent any amount of time on. I suppose the good thing is that I got to play around with Flutter and learnt how to connect to a Chromecast using Dart, so it’s not a complete waste.