Working on a new feature for Dynamo-Browse which will allow the user to modify the columns of the table: move them around, sort them, hide them, etc. I want the feature to be interactive instead of a whole lot of command incantations that are tedious to write. I also kind of want the table whose columns are being manipulated to be visible, just so that the affects of the change would be apparent to the user while they make them.

This is also an excuse to try something out using Bubble Tea — the TUI framework I’m using — which is to add the ability to display overlays. These are UI elements (the overlay) that appear above other UI elements (the background). The two are composed into a single view such that it looks like the overlay is obscuring parts of the background, similar to how windows work in MacOS.

Bubble Tea doesn’t have anything like this built in, and the way views are rendered in Bubble Tea doesn’t make this super easy. The best way to describe how views work is to think of them as “scanlines.” Each view produces a string with ANSI escape sequences to adjust the style. The string can contain newlines which can be used to move the cursor down while rendering. Thus, there’s no easy way position the cursor at an arbitrary position and render characters over the screen.¹

So, I thought I’d tackle this myself.

Attempt 1

On the surface, the logic for this is simple. I’ll render the background layer up to the top most point of the overlay. Then for each scan line within the top and bottom border of the overlay, I’ll render the background up to the overlay’s left border, then the contents of the overlay itself, then the rest of the background from the right border of the overlay.

My first attempt was something like this:

line := backgroundScanner.Text()
if row >= c.overlayY && row < c.overlayY+c.overlayH {
    // When scan line is within top & bottom of overlay
    compositeOutput.WriteString(line[:c.foreX])
    
    foregroundScanPos := row - c.overlayY
    if foregroundScanPos < len(foregroundViewLines) {
        displayLine := foregroundViewLines[foregroundScanPos]
        compositeOutput.WriteString(lipgloss.PlaceHorizontal(
            c.overlayW,
            lipgloss.Left,
            displayLine,
            lipgloss.WithWhitespaceChars(" ")),
        )
    }
    
    compositeOutput.WriteString(line[c.overlayX+c.overlayW:])
} else {
    // Scan line is beyond overlay boundaries of overlay
    compositeOutput.WriteString(line)
}

Here’s how that looked:

Yeah, not great.

Turns out I forgot two fundamental things. One is that the indices of Go strings works on the underlying byte array, not runes. This means that attempting to slice a string between multi-byte Unicode runes would produce junk. It’s a little difficult to find this in the Go Language Guide apart from this cryptic line:

A string’s bytes can be accessed by integer indices 0 through len(s)-1

But it’s relatively easy to test within the Go playground:

package main

import "fmt"

func main() {
    fmt.Println("世界😀😚🙃😇🥸😎"[1:6])                   // ��界
    fmt.Println(string([]rune("世界😀😚🙃😇🥸😎")[1:6]))   // 界😀😚🙃😇
}

The second issue is that I’m splitting half-way through an ANSI escape sequence. I don’t know how long the escape sequence is to style the header of the item view, but I’m betting that it’s longer than 5 bytes (the overlay is to be position at column 5). That would explain why there’s nothing showing up to the left of the overlay for most rows, and why the sequence 6;48;5;188m is there.

Attempt 2

I need to modify the logic so that zero-length escape sequences are preserved. Fortunately, one of Bubble Tea’s dependency is reflow, which offers a bunch of nice utilities for dealing with ANSI escape sequences. The function that looks promising is truncate.String, which will truncate a string at a given width.

So changing the logic slightly, the solution became this:

// When scan line is within top & bottom of overlay
compositeOutput.WriteString(truncate.String(line, uint(c.overlayX)))

foregroundScanPos := r - c.overlayY
if foregroundScanPos < len(foregroundViewLines) {
    displayLine := foregroundViewLines[foregroundScanPos]
    compositeOutput.WriteString(lipgloss.PlaceHorizontal(
        c.overlayW,
        lipgloss.Left,
        displayLine,
        lipgloss.WithWhitespaceChars(" "),
    ))
}

rightStr := truncate.String(line, uint(c.foreX+c.overlayW))
compositeOutput.WriteString(line[len(rightStr):])

The results are a little better. At least the left side of the overlay looked OK now:

But there are still problems here. Notice the [0m at the right side of the overlay on the selected row. I can assure you that’s not part of the partition key; take a look at the item view to see for yourself. And while you’re there, notice the header of the item view? That should be a solid grey bar, but instead it’s cut off at the overlay.

I suspect that rightStr does not have the proper ANSI escape sequences. I’ll admit that the calculation used to set rightStr is a bit of a hack. I’ll need to replace it with a proper way to detect the boundary of an ANSI escape sequence. But it’s more than just that. If an ANSI escape sequence starts off at the left side of the string, and continues on “underneath” the overlay, it should be preserved on the right side of the overlay as well. The styling of the selected row and the item view headers are examples of that.

Attempt 3

So here’s what I’m considering: we “render” the background underneath the overlay to a null buffer while recording any escape sequences that were previously set on the left, or were changed underneath the overlay. We also keep track of the number of visible characters that were seen. Once the scan line position reached the right border of the overlay, we replay all the ANSI escape sequences in the order that were found, and then render the right hand side of the scan line from the first visible character.

I was originally considering rendering these characters to a null reader, but what I actually did was simply count the length of visible characters in a loop. The function to do this looks like this:

func (c *Compositor) renderBackgroundUpTo(line string, x int) string {
    ansiSequences := new(strings.Builder)
    posX := 0
    inAnsi := false

    for i, c := range line {
        if c == ansi.Marker {
            ansiSequences.WriteRune(c)
            inAnsi = true
        } else if inAnsi {
            ansiSequences.WriteRune(c)
            if ansi.IsTerminator(c) {
                inAnsi = false
            }
        } else {
            if posX >= x {
                return ansiSequences.String() + line[i:]
            }
            posX += runewidth.RuneWidth(c)
        }
    }
    return ""
}

And the value set to rightStr is changed to simply used this function:

rightStr := c.renderBackgroundUpTo(line, c.foreX+c.overlayW)

Here is the result:

That looks a lot better. Gone are the artefacts from splitting in ANSI escape sequences, and the styling of the selected row and item view header are back.

I can probably work with this. I’m hoping to use this to provide a way to customise the columns with the table view. It’s most likely going to power other UI elements as well.

One other thing I did in Dynamo-Browse is change how the query AST produced the actual DynamoDB call.

Previously, the AST produced the DynamoDB call directly. For example, if we were to use the expression pk = "something" and sk ^= "prefix", the generated AST may look something like the following:

The AST will then be traversed to determine whether this could be handled by either running a query or a scan. This is called “planning” and the results of this will determine which DynamoDB API endpoint will be called to produce the result. This expression may produce a call to DynamoDB that would look like this:

client.Query(&dynamodb.QueryInput{
    TableName: "my-table-name",
    KeyConditionExpression: "#0 = :0 and beings_with(#1, :1)",
    ExpressionAttributeNames: map[string]string{
        "#0": "pk",
        "#1": "sk",
    },
    ExpressionAttributeValues: map[string]types.AttributeValue{
         ":0": &types.StringAttributeValue{ Value: "something" },
         ":1": &types.StringAttributeValue{ Value: "prefix" },
    },
})

Now, instead of determining the various calls to DynamoDB itself, the AST will generate an intermediary representation, something similar to the following:

The planning traversal will now happen off this tree, much like it did over the AST.

For such a simple expression, the benefits of this extra step may not be so obvious. But there are some major advantages that I can see from doing this.

First, it simplifies the planning logic quite substantially. If you compare the first tree with the second, notice how the nodes below the “and” node are both of type “binOp”. This type of node represents a binary operation, which can either be = or ^=, plus all the other binary operators that may come along. Because so many operators are represented by this single node type, the logic of determining whether this part of the expression can be represented as a query will need to look something like the following:

• First determine whether the operation is either = or ^= (or whatever else)
• If it’s = and the field on the left is either a partition or sort key, this can be represented as a query
• If it’s ^=, first determine whether the operand is a string, (if not, fail) and then determine whether the field on the left is a sort key. If so, then this can be query.
• Otherwise, it will have to be a scan.

This is mixing various stages of the compilation phase in a single traversal: determining what the operator is, determining whether the operands are valid (^= must have a string operand), and working out how we can run this as a query, if at all. You can imagine the code to do this being large and fiddly.

With the IR tree, the logic can be much simpler. The work surrounding the operand is done when the AST tree is traverse. This is trivial: if it’s = then produce a “fieldEq”; if it’s ^= then produce a “fieldBeginsWith”, and so on. Once we have the IR tree, we know that when we encounter a “fieldEq” node, this attribute can be represented as a query if the field name is either the partition or sort key. And when we encounter the “fieldBeginsWith” node, we know we can use a query if the field name is the sort key.

Second, it allows the AST to be richer and not tied to how the actual call is planned out. You won’t find the ^= operator in any of the expressions DynamoDB supports: this was added to Dynamo-Browse’s expression language to make it easier to write. But if we were to add the “official” function for this as well — begins_with() — and we weren’t using the IR, we would need to have the planning logic for this in two places. With an IR, we can simply have both operations produce a “fieldBeginsWith” node. Yes, there could be more code encapsulated by this IR node (there’s actually less) but it’s being leverage by two separate parts of the AST.

And since expressions are not directly mappable to DynamoDB expression types, we can theoretically do things like add arithmetic operations or a nice suite of utility functions. Provided that these produce a single result, these parts of the expression can be evaluated while the IR is being built, and the literal value returned that can be used directly.

It felt like a few other things went right with this decision. I was expecting this to take a few days, but I was actually able to get it built in a single evening. I’m also happy about how maintainable the code turned out to be. Although there are two separate tree-like types that need to be managed, both have logic which is much simpler than what we were dealing with before.

All in all, I’m quite happy with this decision.