Workpad
-
Dynamo-Browse was actually the second TUI tool I made as part of what was called βawstoolsβ. The first was actually an SQS browser. ↩︎
- Audax toolset is now distributed via Homebrew. Check out the Downloads page for instructions.
- A new mark command to mark all, unmark all, or toggle marked rows. The
unmarkcommand is now an alias tomark none. - Query expressions involving the partition and sort key of the main table are now executed as a DynamoDB queries, instead of scans.
- The query expression language now supports conjunction, disjunction, and dot references.
- Fixed a bug which was not properly detecting whether MacOS was in light mode. This was making some highlighted colours hard to see while in dark mode.
- Fixed the table-selection filter, which was never properly working since the initial release.
- Fixed the back-stack service to prevent duplicate views from being pushed.
- Fixed some conditions which were causing seg. faults.
-
Or you can like, comment or subscribe on YouTube if that’s your thing π. ↩︎
-
Someone once said that an insane person is one that does the same thing twice and expects different results. That someone has never worked in technology. ↩︎
-
Well, there might be a way using ANSI escape sequences, but that goes against the approach of the framework. ↩︎
- First determine whether the operation is either
=or^=(or whatever else) - If it’s
=and the field on the left is either a partition or sort key, this can be represented as a query - If it’s
^=, first determine whether the operand is a string, (if not, fail) and then determine whether the field on the left is a sort key. If so, then this can be query. - Otherwise, it will have to be a scan.
- It needs command line completion. Typing out a full attribute name is so annoying, especially considering that you need to be a little careful you got the attribute name right, lest you actually add a new one.
- It needs command line history. Pressing up a few times is so much better than typing out the full command again and again. Such a history could be something that lives in the workspace, preserving it across restarts.
- The
set-attranddel-attrcommands need a switch to take the value directly, rather than by prompting the user to supply it after entering the command (it can still do that, but have an option to take it as a switch as well). I think having a-setswitch after the attribute names would suffice. - 14 rows at the bottom of the screen (the default)
- Half the screen
- All but the top 6 rows of the screen, so you can still see a bit of the table view
- The entire screen (i.e. hiding the table view)
- Completely hidden (i.e. the table view takes up the entire screen).
- Define new commands
- Display messages to, and get input from, the user
- Read the attributes of, and set the attributes of, the currently selected row (only string attributes at this stage)
- Run queries and get result sets, which can optionally be displayed to the user
-
I use the term “ordinary” here as a euphemism for “not very good”. ↩︎
Dynamo-Browse Running With iSH
Bit of a fun one today. After thinking about how one could go about setting up a small dev environment on the iPad, I remembered that I actually had iSH installed. I’ve had for a while but I’ve never really used it since I never installed tools that would be particularly useful. Thinking about what tools I could install, I was curious as to whether Dynamo-Browse could run on it. I guess if Dynamo-Browse was a simple CLI tool that does something and produces some output, it wouldn’t be to difficult to achieve this. But I don’t think I’d be exaggerating if I said Dynamo-Browse is a bit more sophisticated than your run-of-the-mill CLI tool. Part of this is finding out not only whether building it was possible, but whether it will run well.
Answering the first question involves determining which build settings to use to actually produce a binary that worked. Dynamo-Browse is a Go app, and Go has some pretty decent cross-compiling facilities so I had no doubt that such settings existed. My first thought was a Darwin ARM binary since that’s the OS and architecture of the iPad. But one of the features of iSH is that it actually converts the binary through a JIT before it runs it. And it turns out, after poking around a few of the included binaries using file, that iSH expects binaries to be ELF 32-bit Linux binaries.
Thus, setting GOOS to linux and GOARCH to 386 will produced a binary that would run in iSH:
GOOS=linux GOARCH=386 go build -o dynamo-browse-linux ./cmd/dynamo-browse/
After uploading it to the iPad using Airdrop and launching in in iSH, success!
So, is runs; but does is run well? Well, sadly no. Loading and scanning the table worked okay, but doing anything in the UI was an exercise in frustration. It takes about two seconds for the key press to be recognised and to move the selected row up or down. I’m not sure what the cause of this is but I suspect it’s the screen redrawing logic. There’s a lot of string manipulation involved which is not the most memory efficient. I’m wondering if building the app using TinyGo would improve things.
But even so, I’m reasonably impress that this worked at all. Whether it will mean that I’ll be using iSH more often for random tools I build remains to be seen, but at least the possibility is there.
Update: While watching a re:Invent session on how a company moved from Intel to ARM, they mentioned a massive hit to performance around a function that calculates the rune length of a Unicode character. This is something that this application is constantly doing in order to layout the elements on the screen. So I’m wondering if this utility function could be the cause of the slowdown.
Update 2: Ok, after thinking about this more, I think the last update makes no sense. For one thing, the binary iSH is running is an Intel one, and although iSH is interpreting it, I canβt imagine the slowdowns here are a result of the compiled binary. For another, both Intel and ARM (M1) builds of Dynamo-Browse work perfectly well on desktops and laptops (at least on MacOS systems). So the reason for the slowdown must be something else.
Nee Audax Toolset
I’ve decided to retire the Audax Toolset name, at least for the moment. It was too confusing to explain what it actually was, and with only a single tool implemented, this complexity was unnecessary.
The project is now named after the sole tool that is available: Dynamo-Browse. The site is now at dynamobrowse.app, although the old domain will still work. I haven’t renamed the repository just yet so there will still be references to “audax”, particularly the downloads section. I’m hoping to do this soon.
I think I’m happy with this decision. It really now puts focus on the project, and removes the feeling that another tool had to be built just to satisfy the name.
Most of whatβs going on with Audax and Dynamo-Browse is βclosing the gapβ between the possible queries and scans that can be performed over a DynamoDB table, and how they’re represented in Dynamo-Browse query expression language. Most of the constructs of DynamoDBβs conditions expression language can now be represented. The last thing to add is the size() function, and that is proving to be a huge pain.
The reason is that the IR representation is using the expression builder package to actually put the expression together. These builders uses Goβs type system to enforce which constructs work with each other one. But this clashes with how I built the IR representation types, which are essentially structs implement a common interface. Without having an overarching type to represent an expression builder, Iβm left with either using a very broad type like any, or completely ditching this package and doing something else to build the expression.
It feels pretty annoying reaching the brick wall just when I was finishing this off. But I guess themβs the breaks.
One other thing Iβm still considering is spinning out Dynamo-Browse into a separate project. It currently sits under the βAudaxβ umbrella, with the intention of releasing other tools as part of the tool set. These tools actually exist1 but I havenβt been working on them and theyβre not in a fit enough state to release them. So the whole Audax concept is confusing and difficult to explain with only one tool available at the moment.
I suppose if I wanted to work on the other tools, this will work out in the end. But Iβm not sure that I do, at least not now. And even if I do, I’m now beginning to wonder if building them as TUI tools would be the best way to go.
So maybe the best course of action is to make Dynamo-Browse a project in it’s own right. I think it’s something I can resurrect later should I get to releasing a second tool.
Edit at 9:48: I managed to get support for the size function working. I did it by adding a new interface type with a function that returns a expression.OperandBuilder. The existing IR types representing names and values were modified to inherit this interface, which gave me a common type I could use for the equality and comparison expression builder functions.
This meant that the IR nodes that required a name and literal value operand β which are the only constructs allowed for key expressions β had to be split out into separate types from the “generic” ones that only worked on any OperandBuilder node. But this was not as large a change as I was expecting, and actually made the code a little neater.
Audax Toolset Version 0.1.0
Audax Toolset version 0.1.0 is finally released and is available on GitHub. This version contains updates to Dynamo-Browse, which is still the only tool in the toolset so far.
Here are some of the headline features.
Adjusting The Displayed Columns
Consider a table full of items that look like the following:
pk S 00cae3cc-a9c0-4679-9e3a-032f75c2b506
sk S 00cae3cc-a9c0-4679-9e3a-032f75c2b506
address S 3473 Ville stad, Jersey , Mississippi 41540
city S Columbus
colors M (2 items)
door S MintCream
front S Tan
name S Creola Konopelski
officeOpened BOOL False
phone N 9974834360
ratings L (3 items)
0 N 4
1 N 3
2 N 4
web S http://www.investorgranular.net/proactive/integrate/open-source
Let’s say you’re interested in seeing the city, the door colour and the website in the main table which, by default, would look something like this:
There are a few reasons why the table is laid out this way. The partition and sort key are always the first two columns, followed by any declared fields that may be used for indices. This is followed by all the other top-level fields sorted in alphabetical order. Nested fields are not included as columns, and maps and list fields are summarised with the number of items they hold, e.g. (2 items). This makes it impossible to only view the columns you’re interested in.
Version 0.1.0 now allows you to adjust the columns of the table. This is done using the Fields Popup, which can be opened by pressing f.
While this popup is visible you can show columns, hide them, or move them left or right. You can also add new columns by entering a Query Expression, which can be used to reveal the value of nested fields within the main table. It’s now possible to change the columns of the table to be exactly what you’re interested in:
Read-only Mode And Result Limits
Version 0.1.0 also contains some niceties for reducing the impact of working on production tables. Dynamo-Browse can now be started in read-only mode using the -ro flag, which will disable all write operations β a useful feature if you’re paranoid about accidentally modifying data on production databases.
Another new flag is -default-limit which will change the default number of items returned from scans and queries from 1000 to whatever you want. This is useful to cut down on the number of read capacity units Dynamo-Browse will use on the initial scans of production tables.
These settings are also changeable from while Dynamo-Browse using the new set command:
Progress Indicators And Cancelation
Dynamo-Browse now indicates running operations, like scans or queries, with a spinner. This improves the user experience of prior versions of Dynamo-Browse, which gave no feedback of running operations whatsoever and would simply “pop-up” the result of such operations in a rather jarring way.
With this spinner visible in the status bar, it is also now possible to cancel an operation by pressing Ctrl-C. You have the option to view any partial results that were already retrieved at the time.
Other Changes
Here are some of the other bug-fix and improvements that are also included in this release:
Full details of the changes can be found on GitHub. Details about the various features can also be found in the user manual.
Finally, although it does not touch on any of the features described above, I recorded a introduction video on the basics of using Dynamo-Browse to view items of a DynamoDB table:
No promises, but I may record further videos touching on other aspects of the tool in the future. If that happens, I’ll make sure to mention them here.1
Putting the final touches on the website for the upcoming release of Audax Toolset v0.1.0, and I’m finding myself a bit unhappy with it. Given that Dynamo-Browse is the only tool in this “suite”, it feels weird putting together a landing page with a whole lot of prose about this supposed collection of tools. There’s no great place to talk more about Dynamo-Browse right there on the landing page.
Part of me is wondering whether it would be better focusing the site solely on Dynamo-Browse, and leave all this Audax Toolset stuff on the back-burner, at least until (or unless) another tool is made available through this collection. I’m wondering if I’ll need to rearrange the codebase to do this, and spin out the other commands currently in development into separate repositories.
Bridging The Confidence Gap
I had to do some production work with DynamoDB this morning. It wasn’t particularly complicated work: run a query, get a couple of rows, change two attributes on each one. I could have used Dynamo-Browse to do this. But I didn’t. Despite building a tool designed for doing these sorts of things, and using it constantly for all sorts of non-prod stuff, I couldn’t bring myself to use it on a production database.
I’m wondering why this is. It’s not like I have any issues with using Dynamo-Browse for the non-prod stuff. Sure there a bugs, and things that the tool can’t do yet, but I haven’t (yet) encountered a situation where the tool actually corrupted data. I also make sure that the code that touches the database is tested against a “real” instance of DynamoDB (it’s actually a mock server running in Docker).
Yet I still don’t have the confidence to use it on a production database. And okay, part of me understands this to a degree. I’ve only been using this tool for a handful of months, and when it comes to dealing with production data, having a sense of caution, bordering on a small sense of paranoia, is probably healthy. But if I can’t be confident that Dynamo-Browse will work properly when it matters the most, how can I expect others to have that confident?
So I’d like to get to the bottom of this. After a bit of thought, I think there are three primary causes of my lack of confidence here.
The first is simple: there are features missing in the tool that are needed to do my job. Things such as running a query over an index, or the ability to impose a conditional check on updates. This is a legitimate concern in my opinion: if you need to do something, and Dynamo-Browse doesn’t support it, then you can’t use Dynamo-Browse to do that thing, plain and simple. It’s also the easiest concern to address: either add the feature, or say that feature is unsupported and don’t expect people to use Dynamo-Browse if they need it.
I think the second cause is a lack of insight into what the tool is actually doing. Even though I built the tool and tested it during development, there’s always this feeling of uncertainty in the back of my head while I’m using it when the stakes are high. That feeling of “when I do this, although I think the tool will behave this way, how will the tool actually behave?” Part of this, I think, comes from a decade and a half of being burned by technology as part of my job, and growing a healthy (?) sense of distrust for it.1
I’m sure some of this will resolve itself as I continue to use Dynamo-Browse, but the only approach I know of gaining assurance that the tool is working as intended is if the tool tells me what it’s doing. Logging will help here, but I think some features that provide the user the ability to check what’s going to happen prior to actually doing it would be useful as well.
The third cause is probably a lack of controlled end-to-end testing. There does exists a suite of unit tests with a decent (maybe not good, but decent) level of coverage, but it does not extend to the UI layer, which is a little problematic. It might be that more testing of the application as a whole would help here.
This means more manual testing, but it might also be possible to setup some automated testing of the entire tool end-to-end here as well. What’s going for me is the fact that Dynamo-Browse runs in the terminal, and is responsible for rendering the UI and handling the event loop instead of palming this off to the OS. Looking at some of the features that Bubble Tea offers, it might be possible to run the application headless, and simulate use by pumping in keyboard events. Verifying that the UI is such may be a little difficult, but what I can test is what is actually read from and written to the database, which is what I’m mostly interested in.
I’m still unsure as to what I’ll do to address these three causes of concern. This is still a hobby project that I do in my spare time, and some of the mundane tasks, like more manual testing, sound unappealing. On the other hand, I do want to use this tool for my job, and I want others to use it as well. So I can’t really complain that others choose not to use it if I cannot feel comfortable using it when it matters most. So I probably should do some of each. And who knows, I may actually get some enjoyment out of doing it. I certainly would get some enjoyment from knowing that others can rely on it.
Things Dynamo-Browse Need
I’m working with dynamo-browse a lot this morning and I’m coming up with a bunch of usability shortcomings. I’m listing them here so I don’t forget.
The query language needs an in operator, such as pk in ("abc", "123"). This works like in in all the other database services out there, in that the expression effectively becomes `pk = “abc” or pk = “123”. Is this operator supported natively in DynamoDB? π€ Need to check that.
Likewise, the query language needs something to determine whether an attribute contains a substring. I believe DynamoDB expressions support this natively, so it probably makes sense to use that.
Longer term, it would be nice to include the results of the current result set in the expression. For example, assuming the current result set has these records:
pk sk thing place category
11 aa rock home geology
22 bb paper home art
33 cc scissors home utensils
and you want to effectively query for the rows where pk is equal to the set of pk in the current result set, having a way to do that in the expression language would save a lot of copy-and-pasting. An example might be pk in @pk or something similar, which could produce a result set of the form:
pk sk thing place category
11 aa rock home geology
11 ab sand beach ~
22 bb paper home art
22 bc cardboard shops ~
33 cc scissors home utensils
33 cd spoon cafe ~
Another way to do this might be to add support for filters, much like the expressions in JQ. For example, the second result set could be retrieved just by using a query expression of the form place = "home" | fanout pk, which could effectively do the following pesudo-code:
firstResultSet := query("place = 'home')
secondResultSet := newResultSet()
for pk in resultSet['pk'] {
secondResultSet += query("pk = ?", pk)
}
For dealing with workspaces, a quick way to open up the last workspace that you had opened would be nice, just in case you accidentally close Dynamo-Browse and you want to restore the last session you had opened. Something like screen -r. I think in this case having workspace files stored in the temporary directory might be a problem. Maybe having some way to set where workspace files are stored by default would work? π€
For dealing with marking rows, commands to quickly mark or unmark rows in bulk. There’s already the unmark command, but something similar for marking all rows would be good. This could be extended to only mark (or unmark) certain rows, such as those that match a particular substring (then again, using filters would work here as well).
It might be nice for refresh to keep your current row and column position, instead of going to the top-left. And it would also be nice to have a way to refresh credentials (could possiably be handled using the scripting framework).
And finally, if I have any doubt that the feature to hide or rearrange columns would not be useful, there was an instance today where I wish this feature was around. So, keep working on that.
I’m sure there will be more ideas. If so, I’ll post them here.
Overlay Composition Using Bubble Tea
Working on a new feature for Dynamo-Browse which will allow the user to modify the columns of the table: move them around, sort them, hide them, etc. I want the feature to be interactive instead of a whole lot of command incantations that are tedious to write. I also kind of want the table whose columns are being manipulated to be visible, just so that the affects of the change would be apparent to the user while they make them.
This is also an excuse to try something out using Bubble Tea β the TUI framework I’m using β which is to add the ability to display overlays. These are UI elements (the overlay) that appear above other UI elements (the background). The two are composed into a single view such that it looks like the overlay is obscuring parts of the background, similar to how windows work in MacOS.
Bubble Tea doesn’t have anything like this built in, and the way views are rendered in Bubble Tea doesn’t make this super easy. The best way to describe how views work is to think of them as “scanlines.” Each view produces a string with ANSI escape sequences to adjust the style. The string can contain newlines which can be used to move the cursor down while rendering. Thus, there’s no easy way position the cursor at an arbitrary position and render characters over the screen.1
So, I thought I’d tackle this myself.
Attempt 1
On the surface, the logic for this is simple. I’ll render the background layer up to the top most point of the overlay. Then for each scan line within the top and bottom border of the overlay, I’ll render the background up to the overlay’s left border, then the contents of the overlay itself, then the rest of the background from the right border of the overlay.
My first attempt was something like this:
line := backgroundScanner.Text()
if row >= c.overlayY && row < c.overlayY+c.overlayH {
// When scan line is within top & bottom of overlay
compositeOutput.WriteString(line[:c.foreX])
foregroundScanPos := row - c.overlayY
if foregroundScanPos < len(foregroundViewLines) {
displayLine := foregroundViewLines[foregroundScanPos]
compositeOutput.WriteString(lipgloss.PlaceHorizontal(
c.overlayW,
lipgloss.Left,
displayLine,
lipgloss.WithWhitespaceChars(" ")),
)
}
compositeOutput.WriteString(line[c.overlayX+c.overlayW:])
} else {
// Scan line is beyond overlay boundaries of overlay
compositeOutput.WriteString(line)
}
Here’s how that looked:
Yeah, not great.
Turns out I forgot two fundamental things. One is that the indices of Go strings works on the underlying byte array, not runes. This means that attempting to slice a string between multi-byte Unicode runes would produce junk. It’s a little difficult to find this in the Go Language Guide apart from this cryptic line:
A string’s bytes can be accessed by integer indices 0 through len(s)-1
But it’s relatively easy to test within the Go playground:
package main
import "fmt"
func main() {
fmt.Println("δΈηπππππ₯Έπ"[1:6]) // οΏ½οΏ½η
fmt.Println(string([]rune("δΈηπππππ₯Έπ")[1:6])) // ηππππ
}
The second issue is that I’m splitting half-way through an ANSI escape sequence. I don’t know how long the escape sequence is to style the header of the item view, but I’m betting that it’s longer than 5 bytes (the overlay is to be position at column 5). That would explain why there’s nothing showing up to the left of the overlay for most rows, and why the sequence 6;48;5;188m is there.
Attempt 2
I need to modify the logic so that zero-length escape sequences are preserved. Fortunately, one of Bubble Tea’s dependency is reflow, which offers a bunch of nice utilities for dealing with ANSI escape sequences. The function that looks promising is truncate.String, which will truncate a string at a given width.
So changing the logic slightly, the solution became this:
// When scan line is within top & bottom of overlay
compositeOutput.WriteString(truncate.String(line, uint(c.overlayX)))
foregroundScanPos := r - c.overlayY
if foregroundScanPos < len(foregroundViewLines) {
displayLine := foregroundViewLines[foregroundScanPos]
compositeOutput.WriteString(lipgloss.PlaceHorizontal(
c.overlayW,
lipgloss.Left,
displayLine,
lipgloss.WithWhitespaceChars(" "),
))
}
rightStr := truncate.String(line, uint(c.foreX+c.overlayW))
compositeOutput.WriteString(line[len(rightStr):])
The results are a little better. At least the left side of the overlay looked OK now:
But there are still problems here. Notice the [0m at the right side of the overlay on the selected row. I can assure you that’s not part of the partition key; take a look at the item view to see for yourself. And while you’re there, notice the header of the item view? That should be a solid grey bar, but instead it’s cut off at the overlay.
I suspect that rightStr does not have the proper ANSI escape sequences. I’ll admit that the calculation used to set rightStr is a bit of a hack. I’ll need to replace it with a proper way to detect the boundary of an ANSI escape sequence. But it’s more than just that. If an ANSI escape sequence starts off at the left side of the string, and continues on “underneath” the overlay, it should be preserved on the right side of the overlay as well. The styling of the selected row and the item view headers are examples of that.
Attempt 3
So here’s what I’m considering: we “render” the background underneath the overlay to a null buffer while recording any escape sequences that were previously set on the left, or were changed underneath the overlay. We also keep track of the number of visible characters that were seen. Once the scan line position reached the right border of the overlay, we replay all the ANSI escape sequences in the order that were found, and then render the right hand side of the scan line from the first visible character.
I was originally considering rendering these characters to a null reader, but what I actually did was simply count the length of visible characters in a loop. The function to do this looks like this:
func (c *Compositor) renderBackgroundUpTo(line string, x int) string {
ansiSequences := new(strings.Builder)
posX := 0
inAnsi := false
for i, c := range line {
if c == ansi.Marker {
ansiSequences.WriteRune(c)
inAnsi = true
} else if inAnsi {
ansiSequences.WriteRune(c)
if ansi.IsTerminator(c) {
inAnsi = false
}
} else {
if posX >= x {
return ansiSequences.String() + line[i:]
}
posX += runewidth.RuneWidth(c)
}
}
return ""
}
And the value set to rightStr is changed to simply used this function:
rightStr := c.renderBackgroundUpTo(line, c.foreX+c.overlayW)
Here is the result:
That looks a lot better. Gone are the artefacts from splitting in ANSI escape sequences, and the styling of the selected row and item view header are back.
I can probably work with this. I’m hoping to use this to provide a way to customise the columns with the table view. It’s most likely going to power other UI elements as well.
Intermediary Representation In Dynamo-Browse Expressions
One other thing I did in Dynamo-Browse is change how the query AST produced the actual DynamoDB call.
Previously, the AST produced the DynamoDB call directly. For example, if we were to use the expression pk = "something" and sk ^= "prefix", the generated AST may look something like the following:
The AST will then be traversed to determine whether this could be handled by either running a query or a scan. This is called “planning” and the results of this will determine which DynamoDB API endpoint will be called to produce the result. This expression may produce a call to DynamoDB that would look like this:
client.Query(&dynamodb.QueryInput{
TableName: "my-table-name",
KeyConditionExpression: "#0 = :0 and beings_with(#1, :1)",
ExpressionAttributeNames: map[string]string{
"#0": "pk",
"#1": "sk",
},
ExpressionAttributeValues: map[string]types.AttributeValue{
":0": &types.StringAttributeValue{ Value: "something" },
":1": &types.StringAttributeValue{ Value: "prefix" },
},
})
Now, instead of determining the various calls to DynamoDB itself, the AST will generate an intermediary representation, something similar to the following:
The planning traversal will now happen off this tree, much like it did over the AST.
For such a simple expression, the benefits of this extra step may not be so obvious. But there are some major advantages that I can see from doing this.
First, it simplifies the planning logic quite substantially. If you compare the first tree with the second, notice how the nodes below the “and” node are both of type “binOp”. This type of node represents a binary operation, which can either be = or ^=, plus all the other binary operators that may come along. Because so many operators are represented by this single node type, the logic of determining whether this part of the expression can be represented as a query will need to look something like the following:
This is mixing various stages of the compilation phase in a single traversal: determining what the operator is, determining whether the operands are valid (^= must have a string operand), and working out how we can run this as a query, if at all. You can imagine the code to do this being large and fiddly.
With the IR tree, the logic can be much simpler. The work surrounding the operand is done when the AST tree is traverse. This is trivial: if it’s = then produce a “fieldEq”; if it’s ^= then produce a “fieldBeginsWith”, and so on. Once we have the IR tree, we know that when we encounter a “fieldEq” node, this attribute can be represented as a query if the field name is either the partition or sort key. And when we encounter the “fieldBeginsWith” node, we know we can use a query if the field name is the sort key.
Second, it allows the AST to be richer and not tied to how the actual call is planned out. You won’t find the ^= operator in any of the expressions DynamoDB supports: this was added to Dynamo-Browse’s expression language to make it easier to write. But if we were to add the “official” function for this as well β begins_with() β and we weren’t using the IR, we would need to have the planning logic for this in two places. With an IR, we can simply have both operations produce a “fieldBeginsWith” node. Yes, there could be more code encapsulated by this IR node (there’s actually less) but it’s being leverage by two separate parts of the AST.
And since expressions are not directly mappable to DynamoDB expression types, we can theoretically do things like add arithmetic operations or a nice suite of utility functions. Provided that these produce a single result, these parts of the expression can be evaluated while the IR is being built, and the literal value returned that can be used directly.
It felt like a few other things went right with this decision. I was expecting this to take a few days, but I was actually able to get it built in a single evening. I’m also happy about how maintainable the code turned out to be. Although there are two separate tree-like types that need to be managed, both have logic which is much simpler than what we were dealing with before.
All in all, I’m quite happy with this decision.
Letting Queries Actually Be Queries In Dynamo-Browse
I spent some more time working on dynamo-browse over the weekend (I have to now that I’ve got a user-base π).
No real changes to scripting yet. It’s still only me that’s using it at the moment, and I’m hoping to keep it this way until I’m happy enough with the API. I think we getting close though. I haven’t made the changes discussed in the previous post about including the builtin plugin object. I’m thinking that instead of a builtin object, I’ll use another module instead, maybe something like the following:
const ext = require("audax:ext");
ext.registerCommand("thing", () => console.log("Do the thing"));
The idea is that the ext module will provide access to the extension points that the script can implement, such as registering new commands, etc.
At this stage, I’m not sure if I want to add the concept of namespaces to the ext module name. Could help in making it explicit that the user is accessing the hooks of “this” extension, as opposed to others. It may leave some room for some nice abilities, like provide a way to send messages to other extensions:
// com.example.showTables
// This extension will handle events which will prompt to show tables.
const ext = require("audax:ext/this");
ext.on("show-tables", () => {
ui.showTablePrompt();
});
// com.example.useShowTables
// This extension will use the "show-tables" message "com.example.showTables"
const showTables = require("audax:ext/com.example.showTables");
showTables.post("show-tables");
Then again, this might be unnecessary given that the JavaScript module facilities are there.
Anyway, what I actually did was start work on making queries actually run as queries against the DynamoDB table. In the released version, running a query (i.e. pressing ? and entering a query expression) will actually perform a table scan. Nothing against scans: they do what they need to do. But hardly the quickest way to get rows from DynamoDB if you know the partition and sort key.
So that’s what I’m working on now. Running a query of the form pk = "something" and sk = "else" where pk and sk are the partition and sort keys of the table will now call the dynamodb.Query API.
This also works with the “begins with” operator: pk = "something" and sk ^= "prefix". Since sk is the sort key, this will be executed as part of a query to DynamoDB.
This also works if you were to swap the keys around, as in sk = "else" and pk = "something". Surprisingly the Go SDK that I’m using does not support allow expressions like this: you must have the partition key before the sort key if you’re using KeyAnd. This touches on one of the design goals I have for queries: the user shouldn’t need to care how the expression actually produces the result. If it can do so by running an actual query against the table, then it will; if not, it will do a scan. Generally, the user shouldn’t care either way: just get me the results in the most efficient way you can think of!
That said, it might be necessary for the user to control this to an extent, such as requiring the use of a scan if the planner would normally go for a query. I may add some control to this in the expression language to support this. But these should be, as a rule, very rarely used.
Anyway, that’s the biggest change that’s happening. There is something else regarding expressions that I’m in the process of working on now. I’ll touch on that in another blog post.
Finished version 0.0.3 of Audax Toolset yesterday. The code has been ready since the weekend, but it took me Sunday morning and yesterday (Monday) evening to finish updating the website. All done now.
Now the question is whether to continue working on it, or do something different for a change. There are a few people using Dynamo-Browse at work now, so part of me feels like I should continue building features for it. But I also feel like switching to another project, at least for a little while.
I guess we’ll let any squeaky wheels make the decision for me.
I’ve been using Dynamo-Browse all morning and I think I’ll make some notes about how the experience went. In short: the command line needs some quality of life improvements. Changing the values of two attributes on two different items, while putting them to the DynamoDB table each time, currently results in too many keystrokes, especially given that I was simply going back and forth between two different values for these attributes.
So, in no particular order, here is what I think needs to be improved with the Dynamo-Browse command line:
Finally, I think it might be time to consider adding more language features to the command line. At the moment the commands are just made up of tokens, coming from a split on whitespace characters (while supporting quotes). But I think it may be necessary to convert this into a proper language grammar, and add some basic control structures to it, such as entering multiple commands in a single line. It doesn’t need to be a super sophisticated language: something similar to the like TCL or shell would be enough at first.
It might be that writing a script would have solved this problem, and it would (to a degree at least). But not everything needs to be a script. I tried writing a script this morning to do the thing I was working on and it felt just so overkill, especially considering how short-lived this script would actually be. Having something that you can whip up in a few minutes can be a great help. It would have probably taken me 15-30 minutes to write the script (plus the whole item update thing hasn’t been fully implemented yet).
Anyway, we’ll see which of the above improvements I’ll get to soon. I’m kinda thinking of putting this project on hold for a little while, so I could work on something different. But if this becomes too annoying, I may get to one or two of these.
Thinking About Scripting In Dynamo-Browse
I’ve been using the scripting facilities of dynamo-browse for a little while now. So far they’ve been working reasonably well, but I think there’s room for improvement, especially in how scripts are structured.
At the moment, scripts look a bit like this:
const db = require("audax:dynamo-browse");
const exec = require("audax:x/exec");
db.session.registerCommand("cust", () => {
db.ui.prompt("Enter UserID: ").then((userId) => {
return exec.system("/Users/lmika/projects/accounts/lookup-customer-id.sh", userId);
}).then((customerId) => {
let userId = output.replace(/\s/g, "");
return db.session.query(`pk="CUSTOMER#${customerId}"`, {
table: `account-service-dev`
});
}).then((custResultSet) => {
if (custResultSet.rows.length == 0) {
db.ui.print("No such user found");
return;
}
db.session.resultSet = custResultSet;
});
});
This example script β modified from one that I’m actually using β will create a new cust command which will prompt the user for a user ID, run a shell script to return an associated customer ID for that user ID, run a query for that customer ID in the “account-service-dev”, and if there are any results, set that as the displayed result set.
This is all working fine, but there are a few things that I’m a little unhappy about. For example, I don’t really like the idea of scripts creating new commands off the session willy-nilly. Commands feel like they should be associated with the script that are implementing them, and this is sort of done internally but I’d like it to be explicit to the script writer as well.
At the same time, I’m not too keen of requiring the script writer to do things like define a manifest. That would dissuade “casual” script writers from writing scripts to perform one-off tasks.
So, I’m thinking of adding an plugin global object, which provides the hooks that the script writer can use to extend dynamo-browse. This will kinda be like the window global object in browser-based JavaScript environments.
Another thing that I’d like to do is split out the various services of dynamo-browse into separate pseudo packages. In the script above, calling require("audax:dynamo-browse") will return a single dynamo-browse proxy object, which provides access to the various services like running queries or displaying things to the user. This results in a lot of db.session.this or db.ui.that, which is a little unwieldily. Separating these into different packages will allow the script writer to associate them to different package aliases at the top-level.
With these changes, the script above will look a little like this:
const session = require("audax:dynamo-browse/session");
const ui = require("audax:dynamo-browse/ui");
const exec = require("audax:x/exec");
plugin.registerCommand("cust", () => {
ui.prompt("Enter UserID: ").then((userId) => {
return exec.system("/Users/lmika/projects/accounts/lookup-customer-id.sh", userId);
}).then((customerId) => {
let userId = output.replace(/\s/g, "");
return session.query(`pk="CUSTOMER#${customerId}"`, {
table: `account-service-dev`
});
}).then((custResultSet) => {
if (custResultSet.rows.length == 0) {
ui.print("No such user found");
return;
}
session.resultSet = custResultSet;
});
});
Is this better? Worse? I’ll give it a try and see how this goes.
Dynamo-Browse: Using The Back-Stack To Go Forward
More work on Audax toolset. Reengineered the entire back-stack in dynamo-browse to support going forward as well as backwards, much like going forward in a web-browser. Previously, when going back, the current “view snapshot” was being popped off the stack and lost forever (a “view snapshot” is information about the currently viewed table, including the current query and filter if any are set). But I’d need to maintain these view snapshot if I wanted to add going forward as well.
So this evening, I changed it this such that view snapshots were no longer being deleted. Instead, there is now a pointer to the current view snapshot that will go up or down the stack when the user wants to go backwards or forwards. This involved changing the back-stack from a single-linked list to a doubly-linked list, but that was pretty straightforward. The logic to pop snapshots from the stack hasn’t been removed: in fact, it has been augmented so that it can now remove the entire head up to the current view snapshot. This happens when the user goes backwards a few times, then runs a new query or sets a new filter, again much like a web-browser where going backwards a few times, then clicking a link, will remove any chance of going forward from that point.
Last thing I changed this evening was to restore the last viewed table, query or filter when relaunching dyanmo-browse using an existing workspace. This works quite nicely with the back-stack as it works now: it just needs to read the view snapshot at the head.
Adding Copy-To-Clipboard And Layout Changes In Dynamo-Browse
Spent some more time on dynamo-browse this morning, just a little bit.
Got one new feature built and merged to main, which is the ability to copy the displayed item to the pasteboard by pressing c. This is directly inspired by a feature in K9S, which allows you to copy the logs of a running pod to the pasteboard (I use this feature all the time). The package I’m using to access the pasteboard is github.com/golang-design/clipboard, which seems to offer a nice cross-platform approach to doing this.
The copied item is not meant for machine use. It’s exactly what’s displayed in the item viewport and is intended for tracking items that change over time. What I’ve found myself doing, when I need to track such an item, is to manually select it with Terminal and pasting it to an untitled textedit window. I do this again when the item changes until I’m ready to eye the changes manually. It’s not sophisticated, but it tends to work. And although this feature is not a full-blown item comparer, it will make this use case a little easier. Plus, it will copy the entire image, not just the lines displayed when a viewed item is too large for the viewport.
Speaking of which, the other thing I started working on today was on allowing the user to resize the item viewport to take up more of the screen. At the moment, the displayed item viewport takes up the bottom 14 lines of the screen, and is not resizable. Naturally, for large items, this is not large enough, and having the ability to resize this viewport would be a good thing to have.
I’ve been thinking about the best way to handle this, and I’ve decided that an approach similar to how Vim does this β where you need to press a key-chord followed by a number to move the vertical split between two views β would be a little to annoying here. Too much fine grain control for when you just need the viewport bigger. So, I’m opting for a more course-grain approach, where you simply press w and the viewport size will cycle amongst 5 different sizes:
Doing this will not necessarily steal focus from the table view: even if shown the entire screen, you can still go through each item by pressing Up or Down. But you’ll be able to see more of the selected item.
Anyway, that’s the theory. We’ll see how well this works in practice.
Adding A Back-Stack To Dynamo-Browse
Spent some more time working on dynamo-browse, this time adding a back-stack. This can be used to go back to the previously viewed table, query or filter by pressing the Backspace key, kind of like how a browser back button works.
This is the first feature that makes use of a workspace, which is a concept that I’ve been thinking about since the start of the project. A workspace is basically a file storing various bits of state that could be recalled in future launches of the tool. A workspace is always created, even if one is not explicitly specified by the user. The workspace filename be set by using the -w switch on launch, or if one is not specified, then a new workspace filename within the temp directory is generated. The workspace file itself is a Bolt database, which is a very simple, embeddable key/value store that uses B-Trees and memory-mapped data access. I’m actually using StormDB to access this file, since it provides a nice interface for storing and querying Go structs without having to worry about marshalling or unmarshalling.
The back-stack consist of view-snapshots, which are essentially records of the currently view table, filter, and query. The view-snapshots are stored as a linked list, and the back-stack simply pushes new snapshots as the view changes (because this is stored on file, the backlinks are actually ID references and not physical pointers). I had a bit of trouble getting this right at first. The initial version actually pushed a snapshot of the previous view before it changed (i.e. a new table was selected or the filter was changed). But this resulted in a bit of messy implementation, with push calls copied and pasted across the various controllers with various special cases, etc.
So I went back and reimplemented it so that the top of the back-stack is actually a snapshot of the current view. Pressing the Backspace key will actually pop the current snapshot, then read the head of the stack and use that to update the view. This means that there will always be one item in the stack, which bothered me a little at first, but that did make the code easier to implement.
There is another benefit to this arrangement which just came to me while I was writing this. Since the current state of the view is always at the top of the stack, I could add the ability to restore the last view when launching dynamo-browse with an existing workspace, saving the user from selecting a table and running their last query. This will come a bit later. For the meantime, I need to get this back-stack working with user scripts, so that when a script changes the view, the user can still go back.
Finally, a shoutout to boltcli which is a useful CLI tool for browsing a Bolt database. This tool has come in handy for debugging this feature.
More On Scripting In Dynamo-Browse
Making some more progress on the adding scripting of dynamo-browse. It helps being motivated to be able to write some scripts for myself, just to be able to make my own working life easier. So far the following things can be done within user scripts:
Here’s a modified example of a script I’ve been working on that does most of what is listed above:
const db = require("audax:dynamo-browse");
db.session.registerCommand("cust", () => {
db.ui.prompt("Enter user ID: ").then((userId) => {
return db.session.query(`userId="${userId}"`);
}).then((resultSet) => {
db.session.currentResultSet = resultSet;
});
});
db.session.registerCommand("sub", () => {
let currentItem = db.session.selectedRow;
let currentUserId = currentItem.item["userId"];
if (!currentUserId) {
db.ui.print("No 'userId' on current record");
return;
}
db.session.query(`pk="ID#${currentUserId}"`, {
table: "billing-subscriptions"
}).then((rs) => {
db.session.currentResultSet = rs;
});
});
The process of loading this script is pretty basic at the moment. I’ve added a loadscript command, which reads and executes a particular JavaScript file taken as an argument. Scripts can be loaded on startup by putting these loadscript commands (along with any other commands) in a simple RC file that is read on launch.
I’m not sure how much I like this approach. I would like to put some structure around how these scripts are written and loaded, so that they’re not just running atop of one another. However, I don’t want too much structure in place either. To me, having the floor for getting a script written and executed too high is a little demotivating, and sort of goes against the ability to automate these one-off tasks you sometimes come across. That’s the downsides I see with IDEs like GoLand and, to a lesser degree, VS Code: they may have vibrant plugin ecosystems, but the minimum amount of effort to automate something is quite high, that it may not be worth the hassle.
A nice middleground I’m considering is to make it to have two levels of automation: one using JavaScript, and the other closer to something like Bash or TCL. The JavaScript would be akin to plugins, and would have a rich enough API and runtime to support a large number of sophisticated use-cases, with the cost being a fair degree of effort to get something usable. The Bash or TCL level would have a lower level of sophistication, but it would be quicker to do these one-off tasks that you can bang out and throw away. This would mean splitting the effort between the scripting API and command line language, which would fall on me to do, so I’m not sure if I’d go ahead with this (the command line “language” is little more than a glorified whitespace split at the moment).
In the meantime, I’ll focus on making sure the JavaScript environment is good enough to use. This will take some time and dog-fooding so they’ll be plenty to do.
I’ve also started sharing the tool with those that I work with, which is exciting. I’ve yet to get any feedback from them yet, but they did seem quite interested when they saw me using it. I said this before, but I’ll say it again: I’m just glad that I put a bit of polish into the first release and got the website up with a link I can share with them. I guess this is something to remember for next time: what you’re working on doesn’t need to be complete, but it helps that it’s in a state that is easy enough to share with others.
Scripting In Dynamo-Browse
Making some progress with adding scripting capabilities to dynamo-browse. I’ve selected a pretty full-featured JS interpreter called goja. It has full support for ECMAScript 5.1 and some pretty good coverage of ES6 as well. It also has an event loop, which means that features such as setTimeout and promises come out of the box, which is nice. Best of all, it’s written in Go, meaning a single memory space and shared garbage collector.
I’ve started working the extension interface. I came up with a bit of a draft during the weekend and now just implementing the various bindings. I don’t want to go too fast here. I’m just thinking about all the cautionary tales from those that were responsible for making APIs, and introducing mistakes that they had to live with. I imagine one way to avoid this is to play with the extension API for as long as it make sense, smoothing out any rough edges when I find them.
It’s still early days, but here are some of the test scripts I’ve been writing:
// Define a new "hello" command which will ask for your name and
// print it at the bottom.
const dynamobrowse = require("audax:dynamo-browse");
dynamobrowse.session.registerCommand("hello", () => {
dynamobrowse.ui.prompt("What is your name? ").then((name) => {
dynamobrowse.ui.alert("Hello, " + name);
});
});
The prompt method here asks the user for a line of text, and will return the user’s input. It acts a bit like the prompt function in the browser, except that it uses promises. Unfortunately, Goja does not yet support async/await so that means that anonymous functions will need to do for the time being (at least Goja supports arrow functions).
Here’s another script, which prints out the name of the table, and set the value of the address field on the first row:
const dynamobrowse = require("audax:dynamo-browse");
dynamobrowse.session.registerCommand("bla", () => {
let rs = dynamobrowse.session.currentResultSet;
let tableName = rs.table.name;
dynamobrowse.ui.alert("Current table name = " + tableName);
rs.rows[0].item.address = "123 Fake St.";
});
The rows array, and item object, makes use of DynamicArray and DyanmicObject, which is a nice feature in Goja in which you do not need to set the elements in the array, or fields on the object explicitly. Instead, when a field or element is looked up, Goja will effectively call a method on a Go interface with the field name or element index as a parameter. This is quite handy, particularly as it cuts down on memory copies.
These scripts work but they do require some improvements. For example, some of these names could possibly be shortened, just to reduce the amount of typing involved. They also need a lot of testing: I have no idea if the updated address in the example above will be saved properly.
Even so, I’m quite happy about how quickly this is coming along. Hopefully soon I’ll be in a position to write the test script I’m planning for work.
Also, I got my first beta user. Someone at work saw me use dynamo-browse today and asked where he could get a copy. I’m just so glad I got the website finished in time.
Release Preparation & Next Steps
Finally finished the website for the Audax toolset and cut an initial release (v0.0.2). I’ve also managed to get binary releases for Linux and Windows made. I’ve started to work on binary releases for MacOS with the intention of distributing it via Homebrew. This is completely new to me so I’m not sure what progress I’ve made so far: probably nothing. For one thing, I’ll need a separate machine to test with, since I’ve just been installing them from source on the machine’s I’ve been using. Code signing is going to be another thing that will be fun to deal with.
I’ll slip that into a bit of a background task for the moment. For now, I’d like to start work on a feature for “dynamo-browse” that I’ve been thinking of for a while: the ability for users to extend it with scripts.
Some might argue that it may be a little early for this feature, but I’m motivated to do it now as I have a personal need for this, which is navigating amongst a bunch of DynamoDB tables that all store related data. This is all work related, so can’t say too much here. But if I were just to say that doing this now is a little annoying. It involves copying and pasting keys, changing tables, and running filters and queries manually: all possible, but very time consuming. Having the ability to do all this with a single command or keybinding would be so much better. And since this is all work related, I can’t simply modify the source code directly as it will give too much away. Perfect candidate for using an embedded scripting language.
So I think that’s what I’ll start work on next.
I also need to think about starting work on “sqs-browse”. This would be a complete rewrite of what is currently there: which is something that can be used for polling SQS queues and viewing the messages there. I’m hoping for something a little more sophisticated, involving a workspace for writing and pushing messages to queues, along with pulling, saving and routing them from queues. I haven’t got an immediate need for this yet, but I’ve come close on occasion so I can’t leave this too late.
A New Name for Audax Tools (nee AWS Tools)
I think I’ve settled on a name for the project I’ve been calling “awstools”. “Settled” is probably a good word for it: I came up with it about a week ago, and dismissed it at first as being pretty ordinary1. But over that time, it’s been slowly growing on me. Also, I’ve yet to come up with any alternatives that are better.
Anyway, the name that I’ve settled on is the Audax Toolset.
It’s not a great name. Honestly, anyone with more creativity that me could probably come up with a bunch of names better than this. I’m going to play the name-is-the-thing-you-say-to-talk-about-the-thing here.
But the name was not completely random. “Audax” was chosen from the second part of the Latin name for a wedged-tail eagle: Aquila audax. Translated from Latin it means bold, and although I would definitely not describe this project as in any way “bold”, it is a TUI based application which, given the limited palette for building UIs, may have some instances of bold font. But what I’m trying to invoke with this name is the sense of an eagle flying in a cloudless sky. Cloudless β cloudy β the Cloud β AWS (yeah, like I said it’s not a great name).
Anyway, it’s good enough for the moment. And the domain name audax.tools was available, which was a good thing. Right now, the domain doesn’t point to anything but it will eventually point to the website and user guide.
Speaking of which, the user guide is still slowly coming along. I got a very rough draft of the actual guide itself finished. I’ll need to proofread it and flesh it out a little, and also add some screenshots. The reference, which will list all the commands and key bindings (there isn’t many of them), needs to be written as well, but that’s probably the easiest thing to write. I will admit that the guide itself makes for pretty dry reading, and I’m not entirely convinced it would be enough for someone coming in cold to download up the tools and start using them. I may also need to write some form of tutorial, or maybe even record a demo video. Something to consider once I’ve got the rest of the site finished, I guess.