Screenshots
- Audax toolset is now distributed via Homebrew. Check out the Downloads page for instructions.
- A new mark command to mark all, unmark all, or toggle marked rows. The
unmarkcommand is now an alias tomark none. - Query expressions involving the partition and sort key of the main table are now executed as a DynamoDB queries, instead of scans.
- The query expression language now supports conjunction, disjunction, and dot references.
- Fixed a bug which was not properly detecting whether MacOS was in light mode. This was making some highlighted colours hard to see while in dark mode.
- Fixed the table-selection filter, which was never properly working since the initial release.
- Fixed the back-stack service to prevent duplicate views from being pushed.
- Fixed some conditions which were causing seg. faults.
-
Or you can like, comment or subscribe on YouTube if that’s your thing 😛. ↩︎
-
Well, there might be a way using ANSI escape sequences, but that goes against the approach of the framework. ↩︎
- First determine whether the operation is either
=or^=(or whatever else) - If it’s
=and the field on the left is either a partition or sort key, this can be represented as a query - If it’s
^=, first determine whether the operand is a string, (if not, fail) and then determine whether the field on the left is a sort key. If so, then this can be query. - Otherwise, it will have to be a scan.
- The feed in which it appears in. This can be set to “any” to apply the rule to all feed items.
- Whether the title matches a given string. The match rules are similar to the searches in the feed item list views, which are appearance of each of the space separated tokens somewhere in the title (in any case) with phrases appearing as quoted strings.
- Whether the description matches a given string.
- Start a download of the video
- Mark the feed item as a favourite
- A TUI browser/workspace for DynamoDB tables
- A TUI workspace for monitoring SQS queues
- Some tool for reading JSON style log files.
- Queries in DynamoDB: There’s currently no way to run queries or filter the resulting items. I’m hoping to design a small query language to do this. I’m already using Participal to power a very simple expression language to duplicate items, so as long as I can design something that is expressive enough and knows how to use particular indices, I think this should work.
- Putting brand new items in DynamoDB: At the moment you can create new items based on existing items in a way, but there’s currently no way to create brand new items or adjust the attributes of existing items. For this, I’d like to see an “edit item” mode, where you can select the attribute and edit the value, change the type, add or remove attributes, etc. This would require some UI rework, which is already a bit tentative at this stage (it was sort of rushed together, and although some architectural changes have been made to the M and the C in MVC, work on the V is still outstanding).
- Preview item changes before putting them to DynamoDB: This sort of extends the point above, where you see the diff between the old item and new item before it’s put into DynamoDB.
- Workspaces in SQS Browse: One idea I have for SQS browse is the notion of “workspace”. This is a persistent storage area located on the disk where all the messages from the queue would be saved to. Because SQS browse is currently pulling messages from the SQS queue, I don’t want the user to get into a state where they’ve lost their messages for good. The idea is that a workspace is always created when SQS browse is launch. The user can choose the workspace file explicitly, but if they don’t, then the workspace will be created in the temp directory. Also implicit in this is support for opening existing workspaces to continue work in them.
- Multiple queues in SQS Browse: Something that I’d like to see in SQS Browse is the ability to deal with multiple queues. Say you’re pulling from one queue and you’d like to push it to another. You can use a command to add a queue to the workspace. Then, you can do things like poll the queue, push messages in the workspace to the queue, monitor it’s queue length, etc. Queues would be addressable by number or some other way, so you can simply run the command push 2 to push the current message to queue 2.
Audax Toolset Version 0.1.0
Audax Toolset version 0.1.0 is finally released and is available on GitHub. This version contains updates to Dynamo-Browse, which is still the only tool in the toolset so far.
Here are some of the headline features.
Adjusting The Displayed Columns
Consider a table full of items that look like the following:
pk S 00cae3cc-a9c0-4679-9e3a-032f75c2b506
sk S 00cae3cc-a9c0-4679-9e3a-032f75c2b506
address S 3473 Ville stad, Jersey , Mississippi 41540
city S Columbus
colors M (2 items)
door S MintCream
front S Tan
name S Creola Konopelski
officeOpened BOOL False
phone N 9974834360
ratings L (3 items)
0 N 4
1 N 3
2 N 4
web S http://www.investorgranular.net/proactive/integrate/open-source
Let’s say you’re interested in seeing the city, the door colour and the website in the main table which, by default, would look something like this:
There are a few reasons why the table is laid out this way. The partition and sort key are always the first two columns, followed by any declared fields that may be used for indices. This is followed by all the other top-level fields sorted in alphabetical order. Nested fields are not included as columns, and maps and list fields are summarised with the number of items they hold, e.g. (2 items). This makes it impossible to only view the columns you’re interested in.
Version 0.1.0 now allows you to adjust the columns of the table. This is done using the Fields Popup, which can be opened by pressing f.
While this popup is visible you can show columns, hide them, or move them left or right. You can also add new columns by entering a Query Expression, which can be used to reveal the value of nested fields within the main table. It’s now possible to change the columns of the table to be exactly what you’re interested in:
Read-only Mode And Result Limits
Version 0.1.0 also contains some niceties for reducing the impact of working on production tables. Dynamo-Browse can now be started in read-only mode using the -ro flag, which will disable all write operations — a useful feature if you’re paranoid about accidentally modifying data on production databases.
Another new flag is -default-limit which will change the default number of items returned from scans and queries from 1000 to whatever you want. This is useful to cut down on the number of read capacity units Dynamo-Browse will use on the initial scans of production tables.
These settings are also changeable from while Dynamo-Browse using the new set command:
Progress Indicators And Cancelation
Dynamo-Browse now indicates running operations, like scans or queries, with a spinner. This improves the user experience of prior versions of Dynamo-Browse, which gave no feedback of running operations whatsoever and would simply “pop-up” the result of such operations in a rather jarring way.
With this spinner visible in the status bar, it is also now possible to cancel an operation by pressing Ctrl-C. You have the option to view any partial results that were already retrieved at the time.
Other Changes
Here are some of the other bug-fix and improvements that are also included in this release:
Full details of the changes can be found on GitHub. Details about the various features can also be found in the user manual.
Finally, although it does not touch on any of the features described above, I recorded a introduction video on the basics of using Dynamo-Browse to view items of a DynamoDB table:
No promises, but I may record further videos touching on other aspects of the tool in the future. If that happens, I’ll make sure to mention them here.1
Working on recording that how-to video again this morning. Managed to get a take that I’m happy with. This is after maybe 35 recordings in total which didn’t work (wow, video is not easy). Now downloading the trial version of Final Cut Pro to try and edit the thing.
Overlay Composition Using Bubble Tea
Working on a new feature for Dynamo-Browse which will allow the user to modify the columns of the table: move them around, sort them, hide them, etc. I want the feature to be interactive instead of a whole lot of command incantations that are tedious to write. I also kind of want the table whose columns are being manipulated to be visible, just so that the affects of the change would be apparent to the user while they make them.
This is also an excuse to try something out using Bubble Tea — the TUI framework I’m using — which is to add the ability to display overlays. These are UI elements (the overlay) that appear above other UI elements (the background). The two are composed into a single view such that it looks like the overlay is obscuring parts of the background, similar to how windows work in MacOS.
Bubble Tea doesn’t have anything like this built in, and the way views are rendered in Bubble Tea doesn’t make this super easy. The best way to describe how views work is to think of them as “scanlines.” Each view produces a string with ANSI escape sequences to adjust the style. The string can contain newlines which can be used to move the cursor down while rendering. Thus, there’s no easy way position the cursor at an arbitrary position and render characters over the screen.1
So, I thought I’d tackle this myself.
Attempt 1
On the surface, the logic for this is simple. I’ll render the background layer up to the top most point of the overlay. Then for each scan line within the top and bottom border of the overlay, I’ll render the background up to the overlay’s left border, then the contents of the overlay itself, then the rest of the background from the right border of the overlay.
My first attempt was something like this:
line := backgroundScanner.Text()
if row >= c.overlayY && row < c.overlayY+c.overlayH {
// When scan line is within top & bottom of overlay
compositeOutput.WriteString(line[:c.foreX])
foregroundScanPos := row - c.overlayY
if foregroundScanPos < len(foregroundViewLines) {
displayLine := foregroundViewLines[foregroundScanPos]
compositeOutput.WriteString(lipgloss.PlaceHorizontal(
c.overlayW,
lipgloss.Left,
displayLine,
lipgloss.WithWhitespaceChars(" ")),
)
}
compositeOutput.WriteString(line[c.overlayX+c.overlayW:])
} else {
// Scan line is beyond overlay boundaries of overlay
compositeOutput.WriteString(line)
}
Here’s how that looked:
Yeah, not great.
Turns out I forgot two fundamental things. One is that the indices of Go strings works on the underlying byte array, not runes. This means that attempting to slice a string between multi-byte Unicode runes would produce junk. It’s a little difficult to find this in the Go Language Guide apart from this cryptic line:
A string’s bytes can be accessed by integer indices 0 through len(s)-1
But it’s relatively easy to test within the Go playground:
package main
import "fmt"
func main() {
fmt.Println("世界😀😚🙃😇🥸😎"[1:6]) // ��界
fmt.Println(string([]rune("世界😀😚🙃😇🥸😎")[1:6])) // 界😀😚🙃😇
}
The second issue is that I’m splitting half-way through an ANSI escape sequence. I don’t know how long the escape sequence is to style the header of the item view, but I’m betting that it’s longer than 5 bytes (the overlay is to be position at column 5). That would explain why there’s nothing showing up to the left of the overlay for most rows, and why the sequence 6;48;5;188m is there.
Attempt 2
I need to modify the logic so that zero-length escape sequences are preserved. Fortunately, one of Bubble Tea’s dependency is reflow, which offers a bunch of nice utilities for dealing with ANSI escape sequences. The function that looks promising is truncate.String, which will truncate a string at a given width.
So changing the logic slightly, the solution became this:
// When scan line is within top & bottom of overlay
compositeOutput.WriteString(truncate.String(line, uint(c.overlayX)))
foregroundScanPos := r - c.overlayY
if foregroundScanPos < len(foregroundViewLines) {
displayLine := foregroundViewLines[foregroundScanPos]
compositeOutput.WriteString(lipgloss.PlaceHorizontal(
c.overlayW,
lipgloss.Left,
displayLine,
lipgloss.WithWhitespaceChars(" "),
))
}
rightStr := truncate.String(line, uint(c.foreX+c.overlayW))
compositeOutput.WriteString(line[len(rightStr):])
The results are a little better. At least the left side of the overlay looked OK now:
But there are still problems here. Notice the [0m at the right side of the overlay on the selected row. I can assure you that’s not part of the partition key; take a look at the item view to see for yourself. And while you’re there, notice the header of the item view? That should be a solid grey bar, but instead it’s cut off at the overlay.
I suspect that rightStr does not have the proper ANSI escape sequences. I’ll admit that the calculation used to set rightStr is a bit of a hack. I’ll need to replace it with a proper way to detect the boundary of an ANSI escape sequence. But it’s more than just that. If an ANSI escape sequence starts off at the left side of the string, and continues on “underneath” the overlay, it should be preserved on the right side of the overlay as well. The styling of the selected row and the item view headers are examples of that.
Attempt 3
So here’s what I’m considering: we “render” the background underneath the overlay to a null buffer while recording any escape sequences that were previously set on the left, or were changed underneath the overlay. We also keep track of the number of visible characters that were seen. Once the scan line position reached the right border of the overlay, we replay all the ANSI escape sequences in the order that were found, and then render the right hand side of the scan line from the first visible character.
I was originally considering rendering these characters to a null reader, but what I actually did was simply count the length of visible characters in a loop. The function to do this looks like this:
func (c *Compositor) renderBackgroundUpTo(line string, x int) string {
ansiSequences := new(strings.Builder)
posX := 0
inAnsi := false
for i, c := range line {
if c == ansi.Marker {
ansiSequences.WriteRune(c)
inAnsi = true
} else if inAnsi {
ansiSequences.WriteRune(c)
if ansi.IsTerminator(c) {
inAnsi = false
}
} else {
if posX >= x {
return ansiSequences.String() + line[i:]
}
posX += runewidth.RuneWidth(c)
}
}
return ""
}
And the value set to rightStr is changed to simply used this function:
rightStr := c.renderBackgroundUpTo(line, c.foreX+c.overlayW)
Here is the result:
That looks a lot better. Gone are the artefacts from splitting in ANSI escape sequences, and the styling of the selected row and item view header are back.
I can probably work with this. I’m hoping to use this to provide a way to customise the columns with the table view. It’s most likely going to power other UI elements as well.
🎉
Intermediary Representation In Dynamo-Browse Expressions
One other thing I did in Dynamo-Browse is change how the query AST produced the actual DynamoDB call.
Previously, the AST produced the DynamoDB call directly. For example, if we were to use the expression pk = "something" and sk ^= "prefix", the generated AST may look something like the following:
The AST will then be traversed to determine whether this could be handled by either running a query or a scan. This is called “planning” and the results of this will determine which DynamoDB API endpoint will be called to produce the result. This expression may produce a call to DynamoDB that would look like this:
client.Query(&dynamodb.QueryInput{
TableName: "my-table-name",
KeyConditionExpression: "#0 = :0 and beings_with(#1, :1)",
ExpressionAttributeNames: map[string]string{
"#0": "pk",
"#1": "sk",
},
ExpressionAttributeValues: map[string]types.AttributeValue{
":0": &types.StringAttributeValue{ Value: "something" },
":1": &types.StringAttributeValue{ Value: "prefix" },
},
})
Now, instead of determining the various calls to DynamoDB itself, the AST will generate an intermediary representation, something similar to the following:
The planning traversal will now happen off this tree, much like it did over the AST.
For such a simple expression, the benefits of this extra step may not be so obvious. But there are some major advantages that I can see from doing this.
First, it simplifies the planning logic quite substantially. If you compare the first tree with the second, notice how the nodes below the “and” node are both of type “binOp”. This type of node represents a binary operation, which can either be = or ^=, plus all the other binary operators that may come along. Because so many operators are represented by this single node type, the logic of determining whether this part of the expression can be represented as a query will need to look something like the following:
This is mixing various stages of the compilation phase in a single traversal: determining what the operator is, determining whether the operands are valid (^= must have a string operand), and working out how we can run this as a query, if at all. You can imagine the code to do this being large and fiddly.
With the IR tree, the logic can be much simpler. The work surrounding the operand is done when the AST tree is traverse. This is trivial: if it’s = then produce a “fieldEq”; if it’s ^= then produce a “fieldBeginsWith”, and so on. Once we have the IR tree, we know that when we encounter a “fieldEq” node, this attribute can be represented as a query if the field name is either the partition or sort key. And when we encounter the “fieldBeginsWith” node, we know we can use a query if the field name is the sort key.
Second, it allows the AST to be richer and not tied to how the actual call is planned out. You won’t find the ^= operator in any of the expressions DynamoDB supports: this was added to Dynamo-Browse’s expression language to make it easier to write. But if we were to add the “official” function for this as well — begins_with() — and we weren’t using the IR, we would need to have the planning logic for this in two places. With an IR, we can simply have both operations produce a “fieldBeginsWith” node. Yes, there could be more code encapsulated by this IR node (there’s actually less) but it’s being leverage by two separate parts of the AST.
And since expressions are not directly mappable to DynamoDB expression types, we can theoretically do things like add arithmetic operations or a nice suite of utility functions. Provided that these produce a single result, these parts of the expression can be evaluated while the IR is being built, and the literal value returned that can be used directly.
It felt like a few other things went right with this decision. I was expecting this to take a few days, but I was actually able to get it built in a single evening. I’m also happy about how maintainable the code turned out to be. Although there are two separate tree-like types that need to be managed, both have logic which is much simpler than what we were dealing with before.
All in all, I’m quite happy with this decision.
Saw someone at work use Numi to show some maths so I’m giving it a try. Only just started using it but I already like it. Feels very similar to Tot and Boop: a small MacOS utility that fits nicely in that middle-ground between Calculator and a spreadsheet.
It could be that I’m just an old fogie that doesn’t like whimsy, but chalk me up as someone who doesn’t like the wiggling seek bar in Android’s playback notification.
You can tell I’m bored at work when I spend time building stupid little utilities for myself instead of actually doing the task I should be doing. Today’s stupid little utility: a TUI tool to list merge requests I’ve posted for review. Saves a trip to the browser.
Those “Accept Yours” & “Accept Theirs” buttons in GoLand’s conflicts dialog are a bit of a tease. Might be that I’m missing something (which is possible) but I though a file appearing here means that I manually need to review it. Would anyone blindly accept changes like this?
I’ve been seeing this message an awful lot recently when I’m trying to use mobile data. I originally thought it was Telstra, but now I suspect it’s Android, as restarting the phone seemed to have fixed it.
The names for GitHub Codespaces are randomly generated, but I kinda like the one chosen for this website repo. Seems fitting in a way.
WWDC Videos In Broadtail
Some more work on Broadtail. This time, I added the ability to use it to download Apple WWDC videos.
The way it works is based on the existing RSS feed concept. In order to get the list of videos for a particular WWDC year, you “subscribe” to that by setting up a feed with the new “Apple Developer Videos” type. The external ID is taken from the URL slug of the web-site that Apple publishes the session videos. For example, for WWDC 2021, the external ID would be “wwdc2021”.
Downloading the videos is more or less the same.
There are a few differences between this feed type, and the YouTube RSS feed. For instance, it only makes use of what is available from the website, which means details like publishing date or duration are not really available. This is why the “Publishing” date is displayed as “unknown”. That’s also why the videos are arranged in alphabetical order and the feed itself is not automatically refreshed (although doing so manually by clicking “Refresh” within the feed page will work). These are actually properties that can now be applied to all feeds of one wishes, although the YouTube feeds are still arranged in reverse chronological order by default.
From a coding perspective, this involved a lot of refactoring. I was hoping to move to a more generic feed and video type, but this was the feature that eventually got me to do so. Thing is that if I wanted to add more feed and video types in the future, it should be easier to do so.
Feed Rules In Broadtail
Generally, when there’s a video that I’m interesting in watching, I take a look at Broadtail to see if it’s available. When it is, I go ahead and download it.
However, some videos take a long time to download — we’re talking 10 hours or so — and they’re usually published when I’m not looking, like during the night when I’m asleep (thank’s time-zones). So I’d thought it would be nice for Broadtail to kick off the download for me when the video shows up in the feed.
So I’ve added Feed Rules to do this.
Feed rules are very simple automations that happen when new items are found in during the RSS feed poll. When the video shows up in the feed, and matches the rule condition, Broadtail will perform the rule action for that video.
Feed Rules are added as a new sub-section in “Settings”, which itself is a new top-level section of the app (the “General” sub-section is empty at this stage).
Feed Rules consist of a name, whether the rule is active, a set of conditions, and a set of action. A feed item will need to match all the conditions of the rule in order for the actions to be performed.
The conditions of a feed rule touch upon the following properties of a feed item:
If a feed item matches all the conditions, Broadtail can perform the following actions for the feed item:
There might be more conditions or actions added in the future. So far this seems to be the bare minimum to make the feature usable.
Oh, I hope not.
New AWS Tools Commands
For a while now, I’ve been wanting some tools which would help manage AWS resources that would also run in the terminal. I know in most circumstances the AWS console would work, but I know for myself, there’s a lot of benefit from doing this sort of administration from the command line.
I use the terminal a lot when I’m developing or investigating something. Much of the time while I’m in the weeds I’ve got a bunch of tabs with tools running and producing output, and I’m switching between them as I try to get something working across a bunch of systems.
This is in addition to cases when I need to manage an AWS mock running on the local machine. The AWS console will not work then.
At the start of the week, I was thinking of at least the following three tools that I would like to see exist:
As of yesterday, I actually got around to building the first two.
The first is a tool for browsing DynamoDB tables, which I’m calling dynamo-browse (yes, the names are not great). This tool does a scan of a DynamoDB table, and shows the resulting items in a table. Each item can be inspected in full in the lower half of the window by moving the selection.
At the moment this tool only does a simple scan, and some very lightweight editing of items (duplicate and delete). But already it’s proven useful with the task I was working on, especially when I came to viewing the contents of test DynamoDB instances running on the local machine.
The second tool is used for monitoring SQS queues.
This will poll for SQS messages from a queue and display them in the table here. The message itself will be pretty printed in the lower half of the screen. Messages can also be pushed to another queue. That’s pretty much it so far.
There are a bunch of things I’d like to do in these tools. Here’s a list of them:
As for when I’ll actively work on these tools. It will probably be when I need to use them. But in the short term, I’m glad I got the opportunity to start working on them. They’ve already proven quite useful to me.
This is by far the most useful quick action I’ve made in Automator. It generates a UUID, and places it in the pasteboard. I’ve got it bound to Ctrl+Opt+Cmd+U and I’ve been using it constantly over the last week (writing a lot of tests with test data).
Some More Updates of Broadtail
I’ve made some more changes to Broadtail over the last couple of weeks.
The home page now shows a list of recently published videos below the currently running jobs.
Clicking through to “Show All” displays all the published videos. A simple filter can be applied to filter them down to videos with titles containing the keywords (note: nothing fancy with the filter, just tokenisation and an OR query).
Finally, items can now be favourited. This can be used to select videos that you may want to download in the future. I personally use this to keep the list of “new videos” in the Plex server these videos go to to a minimum.
Feeds In Broadtail
My quest to watch YouTube without using YouTube got a little closer recently with the addition of feeds in Broadtail. This uses the YouTube RSS feed endpoint to list videos recently added to a channel or playlist.
There are a bunch of channels that I watch regularly but I’m very hesitant to subscribe to them within YouTube itself (sorry YouTubers, but I choose not to smash that bell icon). I’m generally quite hesitant to give any signal to YouTube about my watching habits, feeding their machine learning models even more information about myself. But I do want to know when new videos are available, so that I can get them into Plex once they’re released. There is where feeds come in handy.
Also improved is the display of video metadata when selecting a feed item or entering a video ID in the quick look bar. Previously this would immediately start a download of the video, but I prefer knowing more about the video first. These downloads aren’t free, and they usually take many hours to get. Better to know more about them before committing to it.
Incidentally, I think this mode of watching has a slight benefit. There are days when I spend the whole evening binging YouTube, not so much following the algorithm but looking at the various channels I’m interested in for videos that I haven’t seen yet. Waiting several hours for a video download feels a little more measured, and less likely to send me down the YouTube rabbit hole. I’m sure there will still be evenings when I do nothing else other than watch TV, but hopefully that’s more of a choice rather than an accident.
I think this is enough on Broadtail for the time being. It’s more or less functional for what I want to do with it. Time to move onto something else.
Some Screenshots Of Broadtail
I spent some time this morning doing some styling work on Broadtail, my silly little YouTube video download manager I’m working on.
Now, I think it’s fair to say that I’m not a designer. And these designs look a little dated, but, surprisingly, this is sort of the design I’m going for: centered pages, borders, etc. A bit of a retro, tasteless style that may be ugly, but still usable(-ish).
It’s not quite finished — the colours need a bit of work — but it’s sort of the style I have in my head.
On a bit of a writing streak: 50 consecutive days of at least one blog post or journal entry.
P.S. I wonder if writing an entry about this streak, just to keep the streak going, is a form of cheating.