Screenshots
- Audax toolset is now distributed via Homebrew. Check out the Downloads page for instructions.
- A new mark command to mark all, unmark all, or toggle marked rows. The
unmarkcommand is now an alias tomark none. - Query expressions involving the partition and sort key of the main table are now executed as a DynamoDB queries, instead of scans.
- The query expression language now supports conjunction, disjunction, and dot references.
- Fixed a bug which was not properly detecting whether MacOS was in light mode. This was making some highlighted colours hard to see while in dark mode.
- Fixed the table-selection filter, which was never properly working since the initial release.
- Fixed the back-stack service to prevent duplicate views from being pushed.
- Fixed some conditions which were causing seg. faults.
-
Or you can like, comment or subscribe on YouTube if that’s your thing ๐. ↩︎
-
Well, there might be a way using ANSI escape sequences, but that goes against the approach of the framework. ↩︎
- First determine whether the operation is either
=or^=(or whatever else) - If it’s
=and the field on the left is either a partition or sort key, this can be represented as a query - If it’s
^=, first determine whether the operand is a string, (if not, fail) and then determine whether the field on the left is a sort key. If so, then this can be query. - Otherwise, it will have to be a scan.
- The feed in which it appears in. This can be set to โanyโ to apply the rule to all feed items.
- Whether the title matches a given string. The match rules are similar to the searches in the feed item list views, which are appearance of each of the space separated tokens somewhere in the title (in any case) with phrases appearing as quoted strings.
- Whether the description matches a given string.
- Start a download of the video
- Mark the feed item as a favourite
Spent the day restyling the Dynamo-Browse website. The Terminal theme was fun, but over time I found the site to be difficult to navigate. And if you consider that Dynamo-Browse is not the most intuitive tool out there, an easy to navigate user manual was probably important. So I replaced that theme with Hugo-Book, which I think is a much cleaner layout. After making the change, and doing a few small style fixes, I found it to be a significant improvement.
I also tried my hand at designing a logo for Dynamo-Browse. The blue box that came with the Terminal theme was fine for a placeholder, but it felt like it was time for a proper logo now.
I wanted something which gave the indication of a tool that worked on DynamoDB tables while also leaning into it’s TUI characteristics. My first idea was a logo that looked like the DynamoDB icon in ASCII art. So after attempting to design something that looks like it in Affinity Designer, and passing it through an online tool which generated ASCII images from PNG, this was the result:
I tried adjusting the colours of final image, and doing a few things in Acorn to thicken the ASCII characters themselves, but there was no getting around the fact that the logo just didn’t look good. The ASCII characters were too thin and too much of the background was bleeding through.
So after a break, I went back to the drawing board. I remembered that there were actually Unicode block characters which could produce filled-in rectangles of various heights, and I wondered if using them would be a nice play on the DynamoDB logo. Also, since the Dynamo-Browse screen consists of three panels, with only the top one having the accent colour, I thought having a similar colour banding would make a nice reference. So I came up with this design:
And I must say, I like it. It does look a little closer to low-res pixel art than ASCII art, but what it’s trying to allude to is clear. It looks good in both light mode and dark mode, and it also makes for a nice favicon.
That’s all the updates for the moment. I didn’t get around to updating the screenshots, which are in dark-mode to blend nicely with the dark Terminal theme. They actually look okay on a light background, so I can probably hold-off on this until the UI is changed in some way.
Ok, thank you for the insistent reminders, Patreon. I know my subscription for CGP Grey is coming up for renewal. You only need to tell me once. ๐คฆโโ๏ธ
Project Exploration: A Check-in App
I’m in a bit of a exploratory phase at the moment. I’ve set aside Dynamo-Browse for now and looking to start something new. Usually I need to start two or three things before I find something that grabs me: it’s very rare that I find myself something to work on that’s exciting before I actually start working on it. And even if I start something, there’s a good chance that I won’t actually finish it.
So it might seem weird that I’d write about these explorations at all. Mainly I do it for prosperity reasons: just something to record so that when look back and think “what was that idea I had?”, I have something to reflect on.
With that out of the way, I do have a few ideas that are worth considering. Yesterday, I’ve started a new Flutter project in order to explore an idea for an app that’s been rattling around in my head for about a month or two.
Basically the idea is this: I keep a very simple check-in blog where I post check-ins for locations that seem useful to recall down the line. At the moment I post these check-ins manually, and usually a few hours after I leave in order to maintain some privacy of my movements. I’ve been entertaining the idea of an app that would do this: I can post a check-in when I arrive but it won’t be published several hours after I leave. The check-ins themselves follow a consistent format so the actual act of filling in the details can be handled by the app by presenting these properties like a regular form. At the same time, I wanted to try something with the Micropub format and possibly the IndieAuth protocol.
So I started playing around with a Flutter app that could do this. I got to the point where I can post new check-ins and they will show up in a test blog. Here are some screenshots:
It’s still very early stages at this moment. For one thing, check-ins are published as soon as you press “Checkin”. Eventually they should be stored on the device and then published at a later time. The form for entering checkins should also be slightly different based on the checkin type. Restaurants and cafes should offer a field to enter a star rating, and airport checkins should provide a way to describe your route.
I’m not sure if I will pursue this any further. We’ll see how I feel. But it was good to get something up and running in a couple of hours.
Flight home from the US is next week. I’m hoping for an uneventful trip. The news is not cooperating though. ๐
Dynamo-Browse Running With iSH
Bit of a fun one today. After thinking about how one could go about setting up a small dev environment on the iPad, I remembered that I actually had iSH installed. I’ve had for a while but I’ve never really used it since I never installed tools that would be particularly useful. Thinking about what tools I could install, I was curious as to whether Dynamo-Browse could run on it. I guess if Dynamo-Browse was a simple CLI tool that does something and produces some output, it wouldn’t be to difficult to achieve this. But I don’t think I’d be exaggerating if I said Dynamo-Browse is a bit more sophisticated than your run-of-the-mill CLI tool. Part of this is finding out not only whether building it was possible, but whether it will run well.
Answering the first question involves determining which build settings to use to actually produce a binary that worked. Dynamo-Browse is a Go app, and Go has some pretty decent cross-compiling facilities so I had no doubt that such settings existed. My first thought was a Darwin ARM binary since that’s the OS and architecture of the iPad. But one of the features of iSH is that it actually converts the binary through a JIT before it runs it. And it turns out, after poking around a few of the included binaries using file, that iSH expects binaries to be ELF 32-bit Linux binaries.
Thus, setting GOOS to linux and GOARCH to 386 will produced a binary that would run in iSH:
GOOS=linux GOARCH=386 go build -o dynamo-browse-linux ./cmd/dynamo-browse/
After uploading it to the iPad using Airdrop and launching in in iSH, success!
So, is runs; but does is run well? Well, sadly no. Loading and scanning the table worked okay, but doing anything in the UI was an exercise in frustration. It takes about two seconds for the key press to be recognised and to move the selected row up or down. I’m not sure what the cause of this is but I suspect it’s the screen redrawing logic. There’s a lot of string manipulation involved which is not the most memory efficient. I’m wondering if building the app using TinyGo would improve things.
But even so, I’m reasonably impress that this worked at all. Whether it will mean that I’ll be using iSH more often for random tools I build remains to be seen, but at least the possibility is there.
Update: While watching a re:Invent session on how a company moved from Intel to ARM, they mentioned a massive hit to performance around a function that calculates the rune length of a Unicode character. This is something that this application is constantly doing in order to layout the elements on the screen. So I’m wondering if this utility function could be the cause of the slowdown.
Update 2: Ok, after thinking about this more, I think the last update makes no sense. For one thing, the binary iSH is running is an Intel one, and although iSH is interpreting it, I canโt imagine the slowdowns here are a result of the compiled binary. For another, both Intel and ARM (M1) builds of Dynamo-Browse work perfectly well on desktops and laptops (at least on MacOS systems). So the reason for the slowdown must be something else.
Google Photos came up with this amusing suggestion today.
Just to be clear: it was the birds that were rotated, not the camera. ๐
Audax Toolset Version 0.1.0
Audax Toolset version 0.1.0 is finally released and is available on GitHub. This version contains updates to Dynamo-Browse, which is still the only tool in the toolset so far.
Here are some of the headline features.
Adjusting The Displayed Columns
Consider a table full of items that look like the following:
pk S 00cae3cc-a9c0-4679-9e3a-032f75c2b506
sk S 00cae3cc-a9c0-4679-9e3a-032f75c2b506
address S 3473 Ville stad, Jersey , Mississippi 41540
city S Columbus
colors M (2 items)
door S MintCream
front S Tan
name S Creola Konopelski
officeOpened BOOL False
phone N 9974834360
ratings L (3 items)
0 N 4
1 N 3
2 N 4
web S http://www.investorgranular.net/proactive/integrate/open-source
Let’s say you’re interested in seeing the city, the door colour and the website in the main table which, by default, would look something like this:
There are a few reasons why the table is laid out this way. The partition and sort key are always the first two columns, followed by any declared fields that may be used for indices. This is followed by all the other top-level fields sorted in alphabetical order. Nested fields are not included as columns, and maps and list fields are summarised with the number of items they hold, e.g. (2 items). This makes it impossible to only view the columns you’re interested in.
Version 0.1.0 now allows you to adjust the columns of the table. This is done using the Fields Popup, which can be opened by pressing f.
While this popup is visible you can show columns, hide them, or move them left or right. You can also add new columns by entering a Query Expression, which can be used to reveal the value of nested fields within the main table. It’s now possible to change the columns of the table to be exactly what you’re interested in:
Read-only Mode And Result Limits
Version 0.1.0 also contains some niceties for reducing the impact of working on production tables. Dynamo-Browse can now be started in read-only mode using the -ro flag, which will disable all write operations โ a useful feature if you’re paranoid about accidentally modifying data on production databases.
Another new flag is -default-limit which will change the default number of items returned from scans and queries from 1000 to whatever you want. This is useful to cut down on the number of read capacity units Dynamo-Browse will use on the initial scans of production tables.
These settings are also changeable from while Dynamo-Browse using the new set command:
Progress Indicators And Cancelation
Dynamo-Browse now indicates running operations, like scans or queries, with a spinner. This improves the user experience of prior versions of Dynamo-Browse, which gave no feedback of running operations whatsoever and would simply “pop-up” the result of such operations in a rather jarring way.
With this spinner visible in the status bar, it is also now possible to cancel an operation by pressing Ctrl-C. You have the option to view any partial results that were already retrieved at the time.
Other Changes
Here are some of the other bug-fix and improvements that are also included in this release:
Full details of the changes can be found on GitHub. Details about the various features can also be found in the user manual.
Finally, although it does not touch on any of the features described above, I recorded a introduction video on the basics of using Dynamo-Browse to view items of a DynamoDB table:
No promises, but I may record further videos touching on other aspects of the tool in the future. If that happens, I’ll make sure to mention them here.1
Working on recording that how-to video again this morning. Managed to get a take that I’m happy with. This is after maybe 35 recordings in total which didn’t work (wow, video is not easy). Now downloading the trial version of Final Cut Pro to try and edit the thing.
Overlay Composition Using Bubble Tea
Working on a new feature for Dynamo-Browse which will allow the user to modify the columns of the table: move them around, sort them, hide them, etc. I want the feature to be interactive instead of a whole lot of command incantations that are tedious to write. I also kind of want the table whose columns are being manipulated to be visible, just so that the affects of the change would be apparent to the user while they make them.
This is also an excuse to try something out using Bubble Tea โ the TUI framework I’m using โ which is to add the ability to display overlays. These are UI elements (the overlay) that appear above other UI elements (the background). The two are composed into a single view such that it looks like the overlay is obscuring parts of the background, similar to how windows work in MacOS.
Bubble Tea doesn’t have anything like this built in, and the way views are rendered in Bubble Tea doesn’t make this super easy. The best way to describe how views work is to think of them as “scanlines.” Each view produces a string with ANSI escape sequences to adjust the style. The string can contain newlines which can be used to move the cursor down while rendering. Thus, there’s no easy way position the cursor at an arbitrary position and render characters over the screen.1
So, I thought I’d tackle this myself.
Attempt 1
On the surface, the logic for this is simple. I’ll render the background layer up to the top most point of the overlay. Then for each scan line within the top and bottom border of the overlay, I’ll render the background up to the overlay’s left border, then the contents of the overlay itself, then the rest of the background from the right border of the overlay.
My first attempt was something like this:
line := backgroundScanner.Text()
if row >= c.overlayY && row < c.overlayY+c.overlayH {
// When scan line is within top & bottom of overlay
compositeOutput.WriteString(line[:c.foreX])
foregroundScanPos := row - c.overlayY
if foregroundScanPos < len(foregroundViewLines) {
displayLine := foregroundViewLines[foregroundScanPos]
compositeOutput.WriteString(lipgloss.PlaceHorizontal(
c.overlayW,
lipgloss.Left,
displayLine,
lipgloss.WithWhitespaceChars(" ")),
)
}
compositeOutput.WriteString(line[c.overlayX+c.overlayW:])
} else {
// Scan line is beyond overlay boundaries of overlay
compositeOutput.WriteString(line)
}
Here’s how that looked:
Yeah, not great.
Turns out I forgot two fundamental things. One is that the indices of Go strings works on the underlying byte array, not runes. This means that attempting to slice a string between multi-byte Unicode runes would produce junk. It’s a little difficult to find this in the Go Language Guide apart from this cryptic line:
A string’s bytes can be accessed by integer indices 0 through len(s)-1
But it’s relatively easy to test within the Go playground:
package main
import "fmt"
func main() {
fmt.Println("ไธ็๐๐๐๐๐ฅธ๐"[1:6]) // ๏ฟฝ๏ฟฝ็
fmt.Println(string([]rune("ไธ็๐๐๐๐๐ฅธ๐")[1:6])) // ็๐๐๐๐
}
The second issue is that I’m splitting half-way through an ANSI escape sequence. I don’t know how long the escape sequence is to style the header of the item view, but I’m betting that it’s longer than 5 bytes (the overlay is to be position at column 5). That would explain why there’s nothing showing up to the left of the overlay for most rows, and why the sequence 6;48;5;188m is there.
Attempt 2
I need to modify the logic so that zero-length escape sequences are preserved. Fortunately, one of Bubble Tea’s dependency is reflow, which offers a bunch of nice utilities for dealing with ANSI escape sequences. The function that looks promising is truncate.String, which will truncate a string at a given width.
So changing the logic slightly, the solution became this:
// When scan line is within top & bottom of overlay
compositeOutput.WriteString(truncate.String(line, uint(c.overlayX)))
foregroundScanPos := r - c.overlayY
if foregroundScanPos < len(foregroundViewLines) {
displayLine := foregroundViewLines[foregroundScanPos]
compositeOutput.WriteString(lipgloss.PlaceHorizontal(
c.overlayW,
lipgloss.Left,
displayLine,
lipgloss.WithWhitespaceChars(" "),
))
}
rightStr := truncate.String(line, uint(c.foreX+c.overlayW))
compositeOutput.WriteString(line[len(rightStr):])
The results are a little better. At least the left side of the overlay looked OK now:
But there are still problems here. Notice the [0m at the right side of the overlay on the selected row. I can assure you that’s not part of the partition key; take a look at the item view to see for yourself. And while you’re there, notice the header of the item view? That should be a solid grey bar, but instead it’s cut off at the overlay.
I suspect that rightStr does not have the proper ANSI escape sequences. I’ll admit that the calculation used to set rightStr is a bit of a hack. I’ll need to replace it with a proper way to detect the boundary of an ANSI escape sequence. But it’s more than just that. If an ANSI escape sequence starts off at the left side of the string, and continues on “underneath” the overlay, it should be preserved on the right side of the overlay as well. The styling of the selected row and the item view headers are examples of that.
Attempt 3
So here’s what I’m considering: we “render” the background underneath the overlay to a null buffer while recording any escape sequences that were previously set on the left, or were changed underneath the overlay. We also keep track of the number of visible characters that were seen. Once the scan line position reached the right border of the overlay, we replay all the ANSI escape sequences in the order that were found, and then render the right hand side of the scan line from the first visible character.
I was originally considering rendering these characters to a null reader, but what I actually did was simply count the length of visible characters in a loop. The function to do this looks like this:
func (c *Compositor) renderBackgroundUpTo(line string, x int) string {
ansiSequences := new(strings.Builder)
posX := 0
inAnsi := false
for i, c := range line {
if c == ansi.Marker {
ansiSequences.WriteRune(c)
inAnsi = true
} else if inAnsi {
ansiSequences.WriteRune(c)
if ansi.IsTerminator(c) {
inAnsi = false
}
} else {
if posX >= x {
return ansiSequences.String() + line[i:]
}
posX += runewidth.RuneWidth(c)
}
}
return ""
}
And the value set to rightStr is changed to simply used this function:
rightStr := c.renderBackgroundUpTo(line, c.foreX+c.overlayW)
Here is the result:
That looks a lot better. Gone are the artefacts from splitting in ANSI escape sequences, and the styling of the selected row and item view header are back.
I can probably work with this. I’m hoping to use this to provide a way to customise the columns with the table view. It’s most likely going to power other UI elements as well.
๐
Intermediary Representation In Dynamo-Browse Expressions
One other thing I did in Dynamo-Browse is change how the query AST produced the actual DynamoDB call.
Previously, the AST produced the DynamoDB call directly. For example, if we were to use the expression pk = "something" and sk ^= "prefix", the generated AST may look something like the following:
The AST will then be traversed to determine whether this could be handled by either running a query or a scan. This is called “planning” and the results of this will determine which DynamoDB API endpoint will be called to produce the result. This expression may produce a call to DynamoDB that would look like this:
client.Query(&dynamodb.QueryInput{
TableName: "my-table-name",
KeyConditionExpression: "#0 = :0 and beings_with(#1, :1)",
ExpressionAttributeNames: map[string]string{
"#0": "pk",
"#1": "sk",
},
ExpressionAttributeValues: map[string]types.AttributeValue{
":0": &types.StringAttributeValue{ Value: "something" },
":1": &types.StringAttributeValue{ Value: "prefix" },
},
})
Now, instead of determining the various calls to DynamoDB itself, the AST will generate an intermediary representation, something similar to the following:
The planning traversal will now happen off this tree, much like it did over the AST.
For such a simple expression, the benefits of this extra step may not be so obvious. But there are some major advantages that I can see from doing this.
First, it simplifies the planning logic quite substantially. If you compare the first tree with the second, notice how the nodes below the “and” node are both of type “binOp”. This type of node represents a binary operation, which can either be = or ^=, plus all the other binary operators that may come along. Because so many operators are represented by this single node type, the logic of determining whether this part of the expression can be represented as a query will need to look something like the following:
This is mixing various stages of the compilation phase in a single traversal: determining what the operator is, determining whether the operands are valid (^= must have a string operand), and working out how we can run this as a query, if at all. You can imagine the code to do this being large and fiddly.
With the IR tree, the logic can be much simpler. The work surrounding the operand is done when the AST tree is traverse. This is trivial: if it’s = then produce a “fieldEq”; if it’s ^= then produce a “fieldBeginsWith”, and so on. Once we have the IR tree, we know that when we encounter a “fieldEq” node, this attribute can be represented as a query if the field name is either the partition or sort key. And when we encounter the “fieldBeginsWith” node, we know we can use a query if the field name is the sort key.
Second, it allows the AST to be richer and not tied to how the actual call is planned out. You won’t find the ^= operator in any of the expressions DynamoDB supports: this was added to Dynamo-Browse’s expression language to make it easier to write. But if we were to add the “official” function for this as well โ begins_with() โ and we weren’t using the IR, we would need to have the planning logic for this in two places. With an IR, we can simply have both operations produce a “fieldBeginsWith” node. Yes, there could be more code encapsulated by this IR node (there’s actually less) but it’s being leverage by two separate parts of the AST.
And since expressions are not directly mappable to DynamoDB expression types, we can theoretically do things like add arithmetic operations or a nice suite of utility functions. Provided that these produce a single result, these parts of the expression can be evaluated while the IR is being built, and the literal value returned that can be used directly.
It felt like a few other things went right with this decision. I was expecting this to take a few days, but I was actually able to get it built in a single evening. I’m also happy about how maintainable the code turned out to be. Although there are two separate tree-like types that need to be managed, both have logic which is much simpler than what we were dealing with before.
All in all, I’m quite happy with this decision.
Saw someone at work use Numi to show some maths so I’m giving it a try. Only just started using it but I already like it. Feels very similar to Tot and Boop: a small MacOS utility that fits nicely in that middle-ground between Calculator and a spreadsheet.
It could be that I’m just an old fogie that doesn’t like whimsy, but chalk me up as someone who doesn’t like the wiggling seek bar in Android’s playback notification.
You can tell I’m bored at work when I spend time building stupid little utilities for myself instead of actually doing the task I should be doing. Today’s stupid little utility: a TUI tool to list merge requests I’ve posted for review. Saves a trip to the browser.
Those “Accept Yours” & “Accept Theirs” buttons in GoLand’s conflicts dialog are a bit of a tease. Might be that I’m missing something (which is possible) but I though a file appearing here means that I manually need to review it. Would anyone blindly accept changes like this?
I’ve been seeing this message an awful lot recently when I’m trying to use mobile data. I originally thought it was Telstra, but now I suspect it’s Android, as restarting the phone seemed to have fixed it.
The names for GitHub Codespaces are randomly generated, but I kinda like the one chosen for this website repo. Seems fitting in a way.
WWDC Videos In Broadtail
Some more work on Broadtail. This time, I added the ability to use it to download Apple WWDC videos.
The way it works is based on the existing RSS feed concept. In order to get the list of videos for a particular WWDC year, you โsubscribeโ to that by setting up a feed with the new โApple Developer Videosโ type. The external ID is taken from the URL slug of the web-site that Apple publishes the session videos. For example, for WWDC 2021, the external ID would be โwwdc2021โ.
Downloading the videos is more or less the same.
There are a few differences between this feed type, and the YouTube RSS feed. For instance, it only makes use of what is available from the website, which means details like publishing date or duration are not really available. This is why the โPublishingโ date is displayed as โunknownโ. Thatโs also why the videos are arranged in alphabetical order and the feed itself is not automatically refreshed (although doing so manually by clicking โRefreshโ within the feed page will work). These are actually properties that can now be applied to all feeds of one wishes, although the YouTube feeds are still arranged in reverse chronological order by default.
From a coding perspective, this involved a lot of refactoring. I was hoping to move to a more generic feed and video type, but this was the feature that eventually got me to do so. Thing is that if I wanted to add more feed and video types in the future, it should be easier to do so.
Feed Rules In Broadtail
Generally, when thereโs a video that Iโm interesting in watching, I take a look at Broadtail to see if itโs available. When it is, I go ahead and download it.
However, some videos take a long time to download โ weโre talking 10 hours or so โ and theyโre usually published when Iโm not looking, like during the night when Iโm asleep (thankโs time-zones). So Iโd thought it would be nice for Broadtail to kick off the download for me when the video shows up in the feed.
So Iโve added Feed Rules to do this.
Feed rules are very simple automations that happen when new items are found in during the RSS feed poll. When the video shows up in the feed, and matches the rule condition, Broadtail will perform the rule action for that video.
Feed Rules are added as a new sub-section in โSettingsโ, which itself is a new top-level section of the app (the โGeneralโ sub-section is empty at this stage).
Feed Rules consist of a name, whether the rule is active, a set of conditions, and a set of action. A feed item will need to match all the conditions of the rule in order for the actions to be performed.
The conditions of a feed rule touch upon the following properties of a feed item:
If a feed item matches all the conditions, Broadtail can perform the following actions for the feed item:
There might be more conditions or actions added in the future. So far this seems to be the bare minimum to make the feature usable.
Oh, I hope not.
