UCL on Leon Mika

Wed, 04 Feb 2026 07:51:56 +1100

Over the weekend, curious to know how well it’ll do, I asked ChatGPT to generate some code in UCL, the toy language I sometimes write about here. I asked it to product a script that would print out the Fibonacci sequence up to a given value. It didn’t do too well, producing a script which looked like a strange hybrid between TCL and shell.

This was somewhat expected. What wasn’t expected was to see UCL pop up as a topic within Pulse the next day:

Very amusing.

Mon, 26 Jan 2026 09:36:01 +1100

Was not expecting to spend yesterday morning working on Dequoter, my Boop clone. Opened it up to do some light work and when I looked up, a couple of hours have passed. Added a few more processors to deal with lines, such as splitting and joining on commas. Added a status bar for processes that return information rather than filter text, such as returning a line count.

Also integrated UCL, because of course I did. Added two commands: a UCL: Evaluate which executes the input as a UCL script and displays the response in the status bar, and a UCL: Replace which replaces the UCL script with it’s output. This makes it possible to generate text from scripts, like a bunch of lines to test out the line count processor:

map (seq 20) { |n| "Line $n" } | strs:join "\n"

It’ll also make for a useful scratchpad for testing out some UCL commands.

Devlog: Dynamo Browse - Item View Annotations and Asynchronous Tasks

Tue, 04 Nov 2025 21:42:40 +1100

The laundry list of things to do in Dynamo Browse has grown over the last week, as I find myself wanting using it and wanting more from it. I’ve knocked off many of the small ones: fixing bugs, making it easier to get the first item from a result set. The time has come to tackle some of the larger ones.

The first is the ability to annotate fields in the item view, as in add additional information to the right of the value. I would personally find that useful for describing the value of an other table using it’s ID. But the point is that it can be anything. I’d like users to add these annotations via extensions, using UCL. The way I’m thinking of doing this is by “installing” a field annotator, something along the lines as follows:

ui:add-item-annotator { |rs attr_path| 
    return ""
}

This will add an annotator which will take a result set, and a path to an attribute. It will return the annotated value. We’ll see how well this will work, but first I’ll need to build the Go types. Rendering the item view is done using an ItemRenderer which basically walks through the attributes of an item and renders it as a table. I think I’ll change this to take an item annotation. I also realised I need a type to represent the attribute path. I think a simple linked list would work here. Here’s what I got:

type AttrPathNode struct {  
    Name    string  
    Index   int  
    IsIndex bool  
    Parent  *AttrPathNode  
}

The trouble is representing both string attribute names and numerical indicies in the same type. Go doesn’t have a great way of doing this, so I’ve gone with something simple and added boolean which would be true if the node is actually an integer index to a list or set.

Actually, that may not be necessary. Poking around the renderer code, I found this type used to represent the sub items of an item:

type SubItem struct {  
    Key   string  
    Value Renderer  
}

So I can simply use strings for the key. Good to know. Okay, added a test annotator and gave it a quick test: Not bad, although found some glaring issues. It would be nice if the annotation was closer to the value. And laying out the annotations in a dedicated column will cause the entire column to shift as I move through the items. So instead of making it a separate column, I’ll try simply concatenating it to the end of the value. I’ll also apply the meta-info styling to dim the text a little: That’s much better. I’m banking on only a few attributes having annotations, so I’m hoping it won’t be as busy as it looks here.

Now to consider the UCL integration. Hmm, how am I going to connect the two? As expected, injecting the annotation directly into ItemRenderer would introduce a dependency loop. What I could do instead is add a setter allowing the caller of the ItemRenderer to change the annotation on the fly, and inject the ItemRenderer as a dependency of the command controller.

Implemented this, and it was actually easier than I thought. Now to test this. I settled on the following UCL command:

ui:set-item-annotator { |rs item path| 
    "annotation"
}

where:

rs is the result set
item is the item being rendered in the view
path is the attribute path, represented as a list.

The attribute path is a new proxy, just to reduce the amount of memory copying between Go and UCL. I kept the same list index semantics, where > 0 starts from the left, and < 0 starts from the right, but it was a little mind bending to reverse this for a linked list.

A simple annotator which returns the currently displayed item value as an annotation can be implemented as follows:

ui:set-item-annotator { |rs item path|
    $item.($path.(-1))
}

Here it is in action: Brilliant, it’s working! Naturally as commands are simple UCL statements, this could be entered on the command line.

One thing to consider is that unlike normal commands, the UCL block rendering the annotation is running on the UI thread. Not sure I like this, but the amount of effort required to change this would be significant. I guess I’ll just tell everyone to keep annotation rendering fast. And as it’s currently implement, it is reasonably fast. Or more accurately, it’s not noticeably slow, which is good enough.

The next feature to add to Dynamo Browse is a way to asynchronously schedule blocks. I think it may be worth tapping into the existing command looper in some way.

Commands in Dynamo Browse are invoked on a dedicated goroutine. Calling execute() will attempt to send a command via a channel. If that fails, execute() will return an error indicating that a command is currently running. This keeps running UCL code off the UI thread, and it also makes it possible to implement commands like ui:prompt synchronously from the UCL code’s perspective, when in reality it pauses this goroutine and sends a message to the UI with a callback to resume the thread:

func (m *uiModule) uiPrompt(
    ctx context.Context,
    args ucl.CallArgs,
) (any, error) {  
    var prompt string  
    if err := args.Bind(&prompt); err != nil {  
        return nil, err  
    }  
  
    resChan := make(chan string)  
    cancelChan := make(chan struct{})  
    go func() {  
        commandctrl.PostMsg(ctx, events.PromptForInputMsg{  
            Prompt: prompt,  
            OnDone: func(value string) tea.Msg {
                resChan <- value  
                return nil  
            },  
            OnCancel: func() tea.Msg {  
                cancelChan <- struct{}{}  
                return nil  
            },
        })  
    }()  
  
    select {  
    case value := <-resChan:  
        return value, nil  
    case <-cancelChan:  
        return nil, nil  
    case <-ctx.Done():  
        return nil, ctx.Err()  
    }  
}

So tapping into this event loop would be nice. But how does one do so?

I think, probably the simplest way to do so is with buffered channels. These are bounded, meaning that there will be an upper limit to the number of pending tasks, but maybe that’s not a bad thing. Having too many pending tasks crowding out the user’s ability to run commands will probably make for a poor user experience. So let’s set the limit to something quite generous, say 50, and integrate it into the event loop:

for {  
    select {  
    case cmdChan := <-c.cmdChan:  
        res, err := c.ExecuteAndWait(ctx, cmdChan.cmd)  
        if err != nil {  
            c.postMessage(events.Error(err))  
        } else if res != nil {  
            c.postMessage(events.StatusMsg(fmt.Sprint(res)))  
        }  
        if execCtx.requestRefresh {  
            c.postMessage(events.ResultSetUpdated{})  
        }  
        // New code here
    case task := <-c.pendingTaskChan:  
        if err := task.task(ctx); err != nil {  
            c.postMessage(events.Error(err))  
        }  
    }  
}

Now for the UCL API. I have an idea of adding a new async package for this, which will provide commands for running things asynchronously. It’s simplest command would be async:do, which will schedule a block when the command loop is free. So invoking the following:

ui:command testasync {  
    echo "This"
    async:do {
        echo "Other"
        async:do { echo "Sierra" }
    }
    async:do {
        echo "Romeo"
    }
    echo "That"
}

Should display:

This
That
Other
Romeo
Sierra

In the logs. Trying it out and here’s how it looks: Okay, that’s pretty good. Now to build on this. The next thing to add is a command that will schedule tasks in the future, similar to window.setTimeout(). For this, I plan to use the gocron package. Lots there, but for my purpose, I plan to use the OneTimeJob type. It does feel like bringing in a crane to lift an empty wood pallet, but the alternative is building my own scheduler (or getting AI to vibe-code one) using a heap. Maybe something for later, but I think this is fine for now.

To make use of this, I’ll add async:in, which takes a timeout in seconds, and a block to run. To test this, I’ll do the classic “count down” tests:

proc countdown { |from|
    if (le $from 0) {
        echo "Blast off!!"
        return
    }
    echo "$from"
    async:in 1 { countdown (sub $from 1) }
}

ui:testasync {
    countdown 10
}

Here it is in action:

One last thing to add to async, the ability to run a query in the background and execute a block once the results are available, keeping long running queries off the UCL goroutine. I think for this I will add another thread-pool which will execute up to two queries in the background, then schedule a task to run once the results are ready.

The API for running queries is pretty well established, to the point that I’ve actually got a helper functions which deal with the UCL side of things, so it’s just a matter of implementing an asynchronous version of this. I’m thinking of an interface along the following lines:

async:query <query> { |rs|
    # do something with the result-set $rs
}

It should also support keyword arguments too. Unfortunately, UCL requires keyword arguments to be placed after the positional arguments, so they would need to be placed after the block, which is a little yucky.

Implemented the code, now for the test. This one’s a little contrived. What it does is asynchronously run a query on startup, counting the number of opened and closed offices in the business-addresses table. Once those results are available, it will install a new column annotation, which will display the count to the right of the officeOpened fields.

proc _prep_officeCount {
    openedOffices = 0
    closedOffices = 0

    async:query 'officeOpened=true' { |rs|
        openedOffices = len $rs
        async:query 'officeOpened=false' { |rs|
            closedOffices = len $rs

            ui:set-item-annotator { |rs item path|
                if (eq $path.(-1) "officeOpened") {
                    if $item.officeOpened {
                        "Count = ${openedOffices}"
                    } else {
                        "Count = ${closedOffices}"
                    }
                } else {
                    ""
                }
            }
        } -table business-addresses
    } -table business-addresses
}

_prep_officeCount

It is contrived, but I’m hoping to use this in a real setting where I annotate ID fields with a value I retrieve from another table, so a use case like this is pretty close to why I built this feature. Because the queries are running in the background, the goal is to avoid blocking the UI thread, but since this runs on startup before the user selects a table, the table needs to be specified as part of the call. I may have to have a think of how I am to fix that.

But here it is working. Note the Count = 244 next to the officeOpened field:

Excellent. The next thing to do is try this out in a real setting. I finished off making a proper goroutine pool for the task that actually runs the query — which I’m calling the “aux task pool” — and a few other changes, like making the meta information a little easier to see.

Devlog: UCL - Adding Some Missing Library Functions

Mon, 27 Oct 2025 22:26:14 +1100

Working on UCL, adding some missing builtins I’ve been finding myself wanting. Nothing too interesting. Just functions like strs:has-prefix, strs:trim-suffix, and other functions involving strings and lists that are missing. That sort of thing.

I am facing some decisions around strs:substr. See, the usual way these builtins are validating arguments is that if there are more arguments than are necessary, they are generally just ignored. One could describe this as the “JavaScript” approach to argument validation: open Node and evaluate "hello".substr(1,2,3,4,5,6,7,8,9) and one will get the same result as if they simply typed in "hello".substr(1,2). But now I’m wondering if it’s better to assert that the arguments are either one or two positions, along with the string. What if I wanted to add more positional arguments in the future?

Although in practice, why would I want to do that? Well, okay, I do have an idea of for taking multiple substrings in a single call. So could be useful? Ah, probably doesn’t matter: it’ll just be me using this for now.

Anyway, strs:substr. One deviation from JavaScript I think I would like to do is that if a single number is used, that would be treated as the end position of the substring; as oppose to JavaScript which treats it as the start position. So running strings:sub "hello, world" 5 will return "hello", rather than ", world". To start from the right, a negative position can be used: strings:sub "hello, world" -5 will return "world". Two numbers will be treated as a start (inclusive) and end (exclusive) positions, like most other languages; although both can be negative which will set the position from the right side.

strs:substr "hello, world" 5   
--> "hello"

strs:substr "hello, world" -5  
--> "world"

strs:substr "hello, world" 3 10
--> "lo, wor"

Okay, that’s working for strings. Need to do the same thing for lists. I could almost use the same function.

Oh, I keep forgetting that I need to worry about nils. How should sublists work if the passed in list is nil? Maybe it’s better to just return a nil. Less hassle than requiring the user to deal with nil values. I’ll start with that approach, but I may change that the future. Should I do the same for strs:substr? Hmm, no. I think I’ll keep the default binding rules for nil to strings, which is basically convert it into an empty string. That saves me from changing the binding logic. Maybe I should lock that decision in with a test.

Hmm, also, many of the other list builtins support iterators. If I were going to add iterator support to sublist , then how would I handle the negative numbers? Iterators, by definition, can be boundless. Maybe leave iterator support out for now.

lists:sublist [a b c] 1
--> [a]

lists:sublist [a b c] -1
--> [c]

It also looks like the argument binder has made my mind up about nils for me. I added the test which testing for nils, and I got an expected listable error.

lists:sublist () 1
--> error: expected listable

I think that’s fine. Might be better to fail on nils actually.

Getting back to the argument question raised earlier, I did run into an issue where I was using strs:split without a second argument. When one does that, UCL will do something similar to Go, and return a list of each individual rune. I think that’s a useful feature, but I think I’d prefer it if that option was explicit, and if a second argument were not set, raise an error.

Okay, a few more items on my todo list. I need a string replace function. I generally prefer such a function to replace all the instances of a string, rather than just the first; but that might be pushing my preference a little too far. So I’ll settle with replacing the first instance of a substring and add an -all flag to replace all instances.

Actually, no. Go has support for specifying a count, so I’ll use that by adding an -n COUNT option. If N is unset, then all the substrings are replaced.

strs:replace "hello hello hello" "hello" "world"
--> world world world

strs:replace "hello hello hello" "hello" "world" -n 1
--> world hello hello

Also, given that this is using Go’s builtin strings.Replace function, setting “match” to the empty string will cause replace to match between each rune, plus the start and end of the string.

strs:replace "each one" "" "|"
--> |e|a|c|h| |o|n|e|

Could be useful, so I’ll keep this feature in.

Nearing the end now: adding an -in switch to os:exec to set stdin for an command. The original idea was to name the switch -stdin, and I also considered -input. But when writing the tests, I naturally settled on -in.

os:exec tr '[a-z]' '[A-Z]' -in "hello"
--> HELLO

It would’ve been nice to accept stdin from a pipe. Sadly, that’s not possible with how pipes actually work.

And the last one: adding a nil? builtin, which will return true if the given item is nil. I guess in the grand scheme of things, it’s probably not strictly necessary, as anything that’s not zero, including nil, will be true. And one could always use eq $thing () to test if $thing is nil. So I may leave this one on the backlog.

Okay, finished. I wanted to add some stuff to Dynamo Browse, but I ran out of time. I will update the version of UCL used by Dynamo Browse, but all that other stuff will need to wait for tomorrow.

Fri, 24 Oct 2025 11:19:41 +1100

It’s funny how I approach certain features in the tools I make, such as adding UCL to Dynamo Browse. It’s been several months since I’ve done this, and I haven’t really used it for anything substantial until today. I guess because I get the sense that it’s half-finished (mainly due to the fact that it is half-finished) I tend to approach such features gingerly: in a careful way so as to avoid any problems. That’s probably not the best way to approach these features though. They need to be taken through the ringer, and just used, lest I never find their limitations or bugs that need to be fixed.

Anyway, this is a long winded way of saying that I’m glad I actually replaced the old scripting engine in Dynamo Browse with UCL. It was added to be used, and it actually came in useful today.

Devlog: UCL — Comparing UCL To Some Early Ideas

Fri, 22 Aug 2025 12:14:02 +1100

I was browsing through some very old notes I had when I came across one that contain an idea for a hypothetical shell-like command language, sort of what UCL was designed for. It was designed for a project called Nuget, which was a CLI torrent downloader (now defunct). Much like UCL, it was REPL based with a simple token-based command language, and I was thinking of ways to extend this to including scripting. Here’s the note in full:

The nuget command language will be something very similar to shell. It will be used to configure nuget (in an RC file), and configure extensions.

Example - check that the VPN is running
let piaCtlState = !piactl get connectionstate

if eval 'trim(${piaCtlState}) ne "Connected"' {
    notice "Warn: VPN not connected.  Use 'piactl' to connect"
}

on newjob {
    on done { echo "Job done: ${job}" }
}

I never implemented this language, opting to simply use TCL. But it’s interesting to see the desire of having an embeddable language for both REPLs and scripting alive and well back in February 2021 (the note’s datestamp).

Assuming that UCL existed at the time, I image the script would’ve looked a little less TCL-ly and a little closer to how Go templates work. Maybe something likes the following, assuming that the built-ins were implemented:

piaCtlState = os:shell "piactl get connectionstate"

if (ne (strs:trim $piaCtlState) "Connected") {
    notice "Warn: VPN not connected.  Use 'piactl' to connect"
}

# I'm guessing 'on' in the original idea would have job as a global variable.
# UCL can simply pass it in as an argument.
on newjob { |job|
  on done { echo "Job done: $job" }
}

Devlog: UCL — More About The Set Operator

Sat, 14 Jun 2025 12:35:00 +1100

I made a decision around the set operator in UCL this morning.

When I added the set operator, I made it such that when setting variables, you had to include the leading dollar sign:

$a = 123

The reason for this was that the set operator was also to be used for setting pseudo-variables, which had a different prefix character.

@ans = "this"

I needed the user to include the @ prefix to distinguish the two, and since one variable type required a prefix, it made sense to require it for the other.

I’ve been trying this for a while, and I’ve deceided I didn’t like it. It felt strange to me. It shouldn’t, really, as it’s similar to how variable assignments work in Go’s templating language, which I consider an inspiration for UCL. On the other hand, TCL and Bash scripts, which are also inspirations, require the variable name to be written without the leading dollar sign in assignments. Heck, UCL itself still had constructs where referencing a name for a variable is done so without a leading dollar sign, such as block inputs. And I had no interest in changing that:

proc foo { |x|
    echo $x
}

for [1 2 3] { |v| foo $v }

So I made the decision to remove the need for the dollar sign prefix in the set operator. Now, when setting a variable, only the variable name can be used:

msg = "Hello"
echo $msg

In fact, if one were to use the leading dollar sign, the program will fail with an error.

This does have some tradeoffs. The first is that I still need to use the @ prefix for setting pseudo variables, and this change will violate the likeness of how the two look in assignments:

@ans = 123
bla = 234

The second is that this breaks the likeness of how a sub-index looks when reading it, verses how it looks when it’s being modified:

a = [1 2 3]
a.(1) = 4
$a
--> [1 4 3]
$a.(1)
--> 4

(One could argue that the dollar sign prefix makes sense here as the evaluator is dereferencing the list in order to modify the specific index. That’s a good argument, but it feels a little bit too esoteric to justify the confusion it would add).

This sucks, but I think they’re tradeoffs worth making. UCL is more of a command language than a templating language, so when asked to imagine similar languages, I like to think one will respond with TCL or shell-scripts, rather than Go templates.

And honestly, I think I just prefer it this way. I feel that I’m more likely to set regular variables rather than pseudo-variables and indicies. So why not go with the approach that seems nicer if you’re likely to encounter more often.

Finally, I did try support both prefixed and non-prefixed variables in the set operator, but this just felt like I was shying away from making a decision. So it wasn’t long before I scrapped that.

Devlog: Dynamo-Browse Now Scanning For UCL Extensions

Sun, 25 May 2025 14:55:49 +1100

Almost finished integrating UCL with Dynamo-Browse. As stated earlier, the goal is to use UCL for both the command and scripting language. Today, the goal was closer, with a development version of Dynamo-Browse no longer using old extension scripting language, and now scanning and evaluating UCL-based extensions on launch time.

This required some changes in how UCL evaluated script files. There’s a new ucl.WithSubEnv() option that can be passed into inst.Eval() to instruct the evaluator to fork the environment prior to evaluating the script:

f, _ := os.ReadFile("my-file.ucl")

inst := ucl.New()
inst.Eval(ctx, bytes.NewReader(r), ucl.WithSubEnv())

Under the hood, this forks the top-level environment frame and marks it as the frame to store all procs and global variables for the evaluated file. This effectively makes it a separate namespace, separating it from all other scripts and preventing cross-contamination when scripts share the same symbol.

I’m hoping that this will eventually be the basis for UCL modules, but for now, this is use within Dynamo-Browse to isolate each UCL extension. The downside of this is that it’s no longer possible to add new commands simply using the proc keyword. I think this may end up being a good thing though, as extensions will probably not be interested in exposing internal functions defined just for their own use.

To compensate for this, there is now a new command — ui:command — which allows the script author to expose a function as a command. In fact, what it does is save the command in the root environment frame, effectively making it work the old way proc did, and acting like a poor-mans extension export.

Now with all the extension loading in place, it’s time for the inaugural UCL extension. Here it is:

# Gets the partition key of the currently selected row, and adds it
# to the pasteboard. Bound to the 'U' key.

ui:command copy-pk {
    # Get the partition key value from the currently selected item
    $pkKey = @table.Keys.PK
    $pkValue = @item.($pkKey)

    # Place it in the pasteboard
    pb:put $pkValue

    echo "$pkKey copied to clipboard"
}

ui:bind "U" { copy-pk }

All the features are not quite implemented yet, but I think it’s good enough to start using it as part of my normal workflow. That’s usually the best way to find the rough edges.

Sun, 18 May 2025 15:07:39 +1100

All the recent changes to UCL is in service of unifying the scripting within Dynamo Browse. Right now there are two scripting languages: one for the commands entered after pressing :, and one for extensions. I want to replace both of them with UCL, which will power both interactive commands, and extensions.

Most of the commands used within the in-app REPL loop has been implemented in UCL. I’m now in the process of building out the UCL extension support, start with functions for working with result sets, and pseudo-variables for modifying elements of the UI.

Here’s a demo of what I’ve got so far. This shows the user’s ability to control the current result-set, and the selected item programatically. Even after these early changes, I’m already seeing much better support for doing such things than what was there before.

Devlog: UCL — Assignment

Sun, 18 May 2025 10:35:57 +1100

Toying with changing how assignment works in UCL. Up to now, assigning a variable to a value involved calling the set command, which took a variable name, and the value to assign:

set a "hello"
$a
--> "hello"

This has been fine, but I’m now running into few limitations with this approach. The first is that it doesn’t support setting subscript values within lists or values. Because set is just a regular command, the parser will evaluate any dot or sub-pipe expressions prior to invoking set. One way around this is to quoted the variable to assign as a string:

set list [1 2 3]
set "$list.(1)" 4

But this looks ugly, and will involve another pass of the parser every time set is called.

Another approach is converting set to a macro, which receives the arguments as the parse tree. This gives set more control over how to interpret the arguments. But it doesn’t resolve the second issue, which is an inconsistency introduced with a new feature I’m planning.

I want to add the notion of pseudo variables, which act like regular variables except that getting or setting them will invoke some hander logic from the embedding system, rather than set a value in memory. Think of them similar to how document and window work in a browser’s JavaScript runtime. My plan for them is to embed this Dynamo Browse to allow the user to access or modify the result-set or selected item, which will result in the UI changing. I’m sure I’ll have more to say about that in the future.

To distinguish them from normal variables, they’ll have an at-sign instead of a dollar sign:

@value
--> something

But as far as the user is concerned, they should act just like normal variables. And just like normal variables, they should be modifiable using the set command. I don’t want the variable and pseudo-variable to share the same namespace (since they look different, there’s no reason for them to share one), so I needed a way to distinguish between setting a pseudo-variable from a regular variable.

This first means they needed to be quoted but I hacked a quick version of set on this feature branch to support setting them:

set "@value" "new value"

But compare this with the first example. Notice that when we were setting a, we didn’t need to include the dollar sign. Here we have a case where one sort of variable requires the prefix symbol, and the other doesn’t. This form of inconsistency was not appealing to me.

Since the various modes of assignments has outgrown the ability of one command to do it all, I’ve decided to remove set and add basic assignments to the language. These look like typical assignments you see in most other languages:

$a = "hello"
$a
--> "hello"

But they’ll be able to fix the issues from using set, such as assign the value of subsets and showing consistent representation of what you’re actually trying to modify:

@value = "new value"

$list = [1 2 3]
$list.(1) = 4
$list
--> [1 4 3]

It also leads to some improvements of the common case. One small issue with using set is that it always required one to wrap the result of a sub-expression in parenthesis.

set h (strs:to-upper "Hello")

But since this is a grammar change, these parenthesis are no longer necessary.

$h = strs:to-upper "Hello"

$h = "Hello" | strs:to-upper

There may also be room for different forms of assignment, such as ensuring the value you’re trying to set is not nil. There was a variant of set, called set!, which threw an error when attempting to assign a variable to nil. This could be expressed as a different assignment form:

$h = "this is fine"

$h =! ()
--> error: trying to set $h to a nil value

I will acknowledge that doing this will mean loosing out on some theoretical benefits that came from using set. For one thing, it will no longer be possible to indirectly set a value. You can’t, for example, do this:

set b "a"
set $b "hello"
$a
--> "hello"

I’ve never really needed to do this, but I could see this being potentially useful. One way to support this might be to do something similar to what Lua does, and expose the environment as a hash. Pseudo-variables could be useful here:

$b = "a"
@_G.($b) = "hello"
$a
--> "hello"

I’ll hold off from adding something like this until I absolutely need to.

So that’s the current idea. I spent around an hour on this so far, just trying it out and seeing how it feels, and I think it’s got promise. I’ll keep it on the feature branch for now, but I suspect this will eventually become the new way to do assignment in UCL going forward.

Fri, 21 Mar 2025 12:36:03 +1100

One of the tools I built for work is starting to get more users, so I probably should remove UCL and replace it with a “real” command language. That’s the risk of building something for yourself: if it’s useful, others will want to use it.

I will miss using UCL, if I do have to remove it. Integrating another command language like TCL or Lisp is not easy, mainly because it’s difficult to map my domain to what the language supports. Other languages, like Lua or Python, map more nicely, but they’re awful to use as a command language. Sure, they may have REPLs, but dealing with the syntax is not fun when you’re just trying to get something done. That’s why I built UCL: to be useable in a REPL, yet rich enough to operate over structured data in a not-crappy way (it may not be glamorous, but it should be doing), while easy to integrate within a Go application.

Of course, if I want to continue to use it, it needs some effort put into it, such as documentation. So which one do I want more?

Thu, 20 Feb 2025 21:48:41 +1100

Have technically secured user no. 2 of UCL today, after sharing one of the tools that’s using the language with them at work. It’s just a shame that the docs are so far behind (read: not existent). All I really have are these blog posts about building it. Good thing he’s a reader 😛. (Hi, KK).

Sat, 08 Feb 2025 12:29:07 +1100

Moving all my project posts onto a separate blog… again. I tried writing them here, but I still feel like they belong elsewhere, where I have a bit more control over the layout and the design. Spent the morning configuring the theme, which was going to be orange but I had to change it to red as the orange didn’t provide a nice contrast for reading (you had to darken it almost to brown). I’m not sure if I’ll move the old posts over to it yet. Maybe the one’s on UCL.

Update on 14/2: No, changed my mind again. See this post.

UCL: Some Updates

Sat, 08 Feb 2025 10:32:00 +1100

Made a few minor changes to UCL. Well, actually, I made one large change. I’ve renamed the foreach builtin to for.

I was originally planning to have a for loop that worked much like other languages: you have a variable, a start value, and an end value, and you’d just iterate over the loop until you reach the end. I don’t know how this would’ve looked, but I imagined something like this:

for x 0 10 {
    echo $x
}
# numbers 0..9 would be printed.

But this became redundant after adding the seq builtin:

foreach (seq 10) { |x|
    echo $x
}

This was in addition to all the other useful things you could do with the foreach loop¹, such as loop over lists and hashes, and consume values from iterators. It’s already a pretty versatile loop. So I elected to go the Python way and just made it so that the for loop is the loop to use to iterate over collections.

This left an opening for a loop that dealt with guards, so I also added the while loop. Again, much like most languages, this loop would iterate over a block until the guard becomes false:

set x 0
while (lt $x 5) {
    echo $x
    set x (add $x 1)
}
echo "done"

Unlike the for loop, this is unusable in a pipeline (well, unless it’s the first component). I was considering having the loop return the result of the guard when it terminates, but I realised that would be either false, nil, or anything else that was “falsy.” So I just have the loop return nil. That said, you can break from this loop, and if the break call had a value, that would be used as the result of the loop:

set x 0
while (lt $x 5) {
    set x (add $x 1)
    if (ge $x 3) {
        break "Ahh"
    }
} | echo " was the break"

The guard is optional, and if left out, the while loop will iterate for ever.

The Set! Builtin

Many of these changes come from using of UCL for my job, and one thing I found myself doing recently is writing a bunch of migration scripts. This needed to get data from a database, which may or may not be present. If it’s not, I want the script to fail immediately so I can check my assumptions. This usually results in constructs like the following:

set planID (ls-plans | first { |p| eq $p "Plan Name" } | index ID)
if (not $planID) {
    error "cannot find plan"
}

And yeah, adding the if block is fine — I do it all the time when writing Go — but it would be nice to assert this when you’re trying to set the variable, for no reason other than the fact that you’re thinking about nullability while writing the expression to fetch the data.

So one other change I made was add the set! builtin. This will basically set the variable only if the expression is not nil. Otherwise, it will raise an error.

set! planID (ls-plans | first { |p| eq $p "Missing Plan" } | index ID)
# refusing to set! `planID` to nil value

This does mean that ! and ? are now valid characters to appear in identifiers, just like Ruby. I haven’t decided whether I want to start following the Ruby convention of question marks indicating a predicate or bangs indicating a mutation. Not sure that’s going to work out now, given that the bang is being used here to assert non-nullability. In either case, could be useful in the future.

And the seq builtin ↩︎

Idea for UCL: Methods

Sun, 02 Feb 2025 15:46:00 +1100

I’m toying with the idea of adding methods to UCL. This will be similar to the methods that exist in Lua, in that they’re essentially functions that pass in the receiver as the first argument, although methods would only be definable by the native layer for the first version.

Much like Lua though, methods would be invokable using the : “pair” operator.

strs:to-upper "Hello"
--> HELLO

The idea is to make some of these methods on the types themselves, allowing their use on literals and the result of pipelines, as well as variables:

set h "Hello"
$hello:to-upper
--> HELLO

"Hello":to-upper
--> HELLO

(cat "Hello " "world"):to-upper
--> HELLO WORLD

The use of methods would be purely conveience. One could conceive of a type, like a CSV table, where there’s a need to perform a series of operations over it.

The potential downside of using : is that it may make defining dictionaries more ambiguous. When the parser sees something that could either be a list or a dictionary, it does a scan to search for any pairs that may exist. If so, it treats the literal as a dictionary. But would this work with using : as the method separator?

["Hello":to-upper]
--> Is this a dictionary {"Hello":"to-upper"}, or the list ["HELLO"]?

So that would require something. Might be that I need to require method invocations to occur within parenthesis for list literals.

Ambiguities like this aside, I’m planning to keep it simple for now. Methods will not be user definable within UCL out of the gate, but would be effectively another interface available to native types. For the builtin ones that exist now, it’ll most likely be little more than syntactic sugar over the standard library functions:

$hello:to-upper

# equivalent to

strs:to-upper $hello

Speaking of the standard library, the use of : as a module-function separator may need some changes. A the moment it’s a bit of a hack: when a module is loaded, any procs defined within that module is stored in the environment with the operator: strs:to-upper for example. The parser is sophisticated enough to recognised : as a token, but when it parses two or more identifiers separated with :, it just joins them up into a single identifier.

What’s needed is something else, maybe using something based on methods. The current idea is to define modules as being some special form of object, where the “methods” are simply the names of the exported symbol.

I was curious to know whether the language was capable of doing something similar now. It’s conceivable that a similar concept could be drafted with procs returning dictionaries of procs, effectively acting as a namespace for a collection of functions. So a bit of time in the playground resulted in this experiment:

proc fns {
  return [
    upper: (proc { |x| strs:to-upper $x })
    lower: (proc { |x| strs:to-lower $x })
  ]
}

(fns).upper "Hello"
--> HELLO

(fns).lower "Hello"
--> hello

A good first start. This is theoretically possible in the language that exists at the moment. It’s not perfect thought. For one thing, the call to fns needs to be enclosed in parenthesis in order to invoke it. Leaving them out results in an error:

fns.upper "Hello"
--> error: command is not invokable

The same is true when using variables instead of procs. I tried this experiment again using variables and it came to the same limitations:

set fns [
  upper: (proc { |x| strs:to-upper $x })
  lower: (proc { |x| strs:to-lower $x })
]

($fns).upper "Hello"
--> HELLO

$fns.upper "Hello"
--> error: command is not invokable

Obviously the parser needs to be changed to add additional support for the : operator, but I also need to fix how strong . binds to values too. But I think this may have legs.

UCL: Iterators

Fri, 31 Jan 2025 09:41:44 +1100

Still working on UCL in my spare time, mainly filling out the standard library a little, like adding utility functions for lists and CSV files. Largest change made recently was the adding iterators to the mix of core types. These worked a lot like the streams of old, where you had a potentially unbounded source of values that could only be consumed one at a time. The difference with streams is that there is not magic to this: iterators work like any other type, so they could be stored in variables, passed around methods, etc (streams could only be consumed via pipes).

I augmented the existing high-level functions like map and filter to consume and produce iterators, but it was fun discovering other functions which also became useful. For example, there exists a head function which returned the first value of a list. But I discovered that the semantics also worked as a way to consume the next element from an iterator. So that’s what this function now does. This, mixed with the fact that iterators are truthy if they’re got at least one pending value, means that some of the builtins could now be implemented in UCL itself. Like the example below, which could potentially be used to reimplement itrs:to-list (this is a contrived example, as foreach would probably work better here).

proc to-list { |itr lst|
   if $itr {
      lists:add $lst (head $itr)
      return (to-list $itr $lst)
   }
   return $lst
}

to-list (itrs:from [1 2 3 4 5]) []

But the biggest advantage that comes from iterators is querying large data-stores with millions of rows. Being able to write a UCL script which sets up a pipeline of maps and filters and just let it churn through all the data in it’s own time is the dream.

list-customers | filter { |x| $x.HasPlan } | map { |x| $x.PlanID } | foreach echo

I’ve got a need for this in the internal backend tool that spurred the development of UCL, and I’m looking forward to using iterators to help here.

Sun, 19 Jan 2025 10:24:18 +1100

Started filling out the UCL website, mainly by documenting the core modules. It might be a little unnecessary to have a full website for this, given that the only person who’ll get any use from it right now will be myself. But who knows how useful it could be in the future? If nothing else, it’s a showcase on what I’ve been working on for this project.

Wed, 15 Jan 2025 22:40:42 +1100

I’ve been using UCL a lot recently, which is driving additional development on it. Spent a fair bit of time this evening fixing bugs and adding small features like string interpolation. Fix a number of grammar bugs too, that only popped up when I started writing multi-line scripts with it.

Thu, 12 Dec 2024 08:48:00 +1100

I plan to integrate UCL into another tool at work, so I spent last night improving it’s use as a REPL. Added support for onboard help and setting up custom type printing, which is useful for displaying tables of data. I started working on the tool today and it’s already feeling great.

Weekly Update - 3 Nov 2024

Sun, 03 Nov 2024 21:39:41 +1100

I probably should stop calling these “weekly updates,” seeing that they come up a lot less frequently than once a week. Maybe I should switch to something like “Sunday updates,” or maybe something closer to what this is, which is an excuse to procrastinate by writing about what I’ve been working on, rather than just working on it.

But I’m sure you’re not interested in my willowing about the frequency of these updates, so let’s just get straight to the meat of it.

Cyber Burger

All the logic, graphics, and sound-effects for power-up/power-downs are now finished. They now spawn in randomly, with a frequency and specific types dictated by the current stage. I also added the notion of a “milkshake bonus” which awards the player a bonus multiplier for a short amount of time.

I’ve also made a few balancing changes around demerits. Based on my own testing, I was pretty blasé about loosing demerits, as you could recover a demerit every time you finish a burger. I wanted to discourage that, so I changed things around a little. You still loose demerits if you screw up the burger your trying to build — such as making it too high or not catching the correct item — but you no longer recover demerits for every burger you complete. Instead, you recover one demerit for every $50 you’re awarded. This is now every three to four burgers, depending on how sophisticated they are, which I hope would make loosing demerits something the player would want to avoid.

There’s still the difficulty curve stuff left to do, but I think I’ll start working on the meta elements, like the title screen, main menu and high score tables. I can probably leave out the addition of stages and a difficulty curve if I’m honest, but I would like to have a decent title and menu screen.

The other thing to do is write the manual. I made a start the other day, but there’s much left to do on this front. Part of me wonders whether it make sense adding “on-board documentation.” But part of the fun of using Pico-8 on this project is to imagine a time where this game came out during the late 70’s and early 80’s and the 8-bit era of home consoles. And those games didn’t have on-board documentation. That said, I might add a quick start guide for those that didn’t RTFM.

UCL

I’ve been using that tool I’ve written for work quite often so there was a need to add some additional features to UCL. The biggest one was adding exceptions, but there’ve been a lot of little things like rounding out the standard library.

All the code written in a rush is starting to weigh this project down though, and I do think I’ll need to do some refactoring to tidy things up a little. I may need to work on documentation too, just for my own sake more than anything else. I doubt this would be anything more than the toy language it currently is, but it does have it’s uses, and whenever I need to reference a built-in, I’m always going to the source code. Which is fine, but I think I can do better on this front.

Other Projects

A few other things I worked on during the last fortnight:

I spent some time last weekend playing with Htmgo by making a simple world clock that would update every second. I ended up remaking this as a static web page backed by some WASM code. Knowing the time in UTC and some American cities could come in handy for my job. At least, that’s the theory: I haven’t had a need for it yet.
I also made some changes to Nano Journal to add support for uploading multiple attachments at once.

So that’s the update for this past fortnight.

Try-Catch In UCL - Some Notes

Mon, 21 Oct 2024 16:49:09 +1100

Stared working on a try command to UCL, which can be used to trap errors that occur within a block. This is very much inspired by try-blocks in Java and Python, where the main block will run, and if any error occurs, it will fall through to the catch block:

try {
  echo "Something bad can happen here"
} catch {
  echo "It's all right. I'll run next"
}

This is all I’ve got working at the moment, but I want to quickly write some notes on how I’d like this to work, lest I forget it later.

First, much like everything in UCL, these blocks should return a value. So it should be possible to do something like this:

set myResults (try {
  result-of-something-that-can-fail
} catch {
  "My default"
})
--> (result of the thing)

This is kind of like using or in Lua to fallback to a default, just that if the result fails with an error, the default value can be returned from the catch block. In might even be possible to simply this further, and have catch just return a value in cases where an actual block of code is unnecessary:

set myResults (try { result-of-something-that-can-fail } catch "My default")

One other thing to consider is how to represent the error. Errors are just treated out-of-band at the moment, and are represented as regular Go error types. It might be necessary to add a new error type to UCL, so that it can be passed through to the catch block for logging or switching:

try {
  do-something
} catch { |e|
  echo (cat "The error is " $e)
}

This could also be used as the return value if there is no catch block:

set myResult (try { error "my error" })
--> error: my error

Another idea I have is successive catch blocks, that would cascade one after the other if the one before it fails:

try {
  do-something
} catch {
  this-may-fail-also
} catch {
  echo "Always passes"
}

Unlike JavaScript or Python, I don’t think the idea of having catch blocks switching based on the error type would be suitable here. UCL is dynamic in nature, and having this static type checking feels a little wrong here. The catch blocks will only act as isolated blocks of execution, where an error would be caught and handled.

Finally, there’s finally, which would run regardless of which try or catch block was executed. I think, unlike the other two blocks, that the return value of a finally block will always be swallowed. I think this will work as the finally block should mainly be used for clean-up, and it’s the result of the try or catch blocks that are more important.

set res (try {
  "try"
} catch {
  "catch"
} finally {  
  "finally"
})
--> "try"

Anyway, this is the idea I have right now.

Update — I just realised that the idea of the last successful try block return an error, rather than letting it climb up the stack defeats the purpose of exceptions. So having something like the following:

try { error "this will fail" }

Should just unroll the stack and not return an error value. Although if there is a need to have an error value returned, then the following should work:

try { error "this will fail" } catch { |err| $err }
--> error: this will fail

Project Updates

Sun, 08 Sep 2024 18:14:51 +1100

Well, it’s been three weeks since my last post here, and as hard as it was to write this update, not writing it would’ve been harder. So let’s just skip the preamble and go straight to the update.

Cyber Burger (That Pico-8 Game)

I’m terrible at being coy, I’ll just spill the beens. That game I’ve been working on is call Cyber Burger. It’s based on a DOS game I saw on YouTube, and it seemed like a fun project to try and work on, with some tweaks to the gameplay that I think would make it more forgiving.

In the weeks since my last update, I finished with the prototypes and started working on the game itself. The initial set of art is done and a very simple “Game A” mode has been implemented. In this mode, you are shown a burger you’ll need to make in your basket. You do so by shooting down the items flying across the screen and catching them in order. When you do, you get a “tip”, which basically amounts to points. If you make a mistake, you’re given a demerit. There are five rounds in total, and once the round is complete, you move on to the next one, with maybe slightly different items, or different item speeds, etc.

An example play session, with the new graphics.

I managed to make a HTML version of this which plays through round 1. I gave it to someone at work to play test and the results were… well, they weren’t bad but it didn’t set the world on fire.

I think I’m okay with that, but I do need to keep working on it. I think one thing that would help is adding sound. And I think it might help me deliver this earlier if I abandoned Mode A and start working on Mode B, which is closer to an arcade style of game that would allow for continuous play. These two things, I’ll aim to work on this next week.

Oh, and I’ll need to fix the item spawner. Waiting ages for an item you need is no good.

If you’re interested in giving it a try, you can do so by following this link (it runs in the browser). Feel free to send any feedback you may have.

UCL

The other thing I’ve been spending some time on over the last week or so was UCL. I’ve been using that work tool which has this language quite often recently and I’ve been running against a number of bugs and areas where quality of life changes could be made. Just small things, such as allowing the foreach command to be called with a proc name instead of requiring a block, much like the map function. Those have been addressed.

But the biggest thing I’ve been worked on was building out the core library. I added the following functions:

The `seq` function, used for generating a sequence of integers. I like how I built this: it’s effectively a virtual list — that can be indexed, iterated over, or calculated the length of — but does not take up linear space.
Comparator functions, like `eq`, `ne`, `gt`, etc. plus settling on a type system much like Python, where values are strongly type (can’t compare ints to strings) but are also dynamic.
Arithmetic functions, like `add`, `sub`, etc. which operate on integers, and the `cat` function use to concatenate strings (these functions do try to cohere values to the correct type)
Logical functions, like `and`, `or`, and `not`.

Along with this, I’ve started working through the strings package, which would add the various string functions you see, like trimming whitespace, splitting, joining, etc. I’ve got trimming and converting to upper and lower case, and my goal for next week is to add splitting to and joining from string lists. Once that’s done I’ll probably put this on the back-burner again so I can finish off Cyber Burger or work on something else.

Just a reminder that there’s also a playground for this too, although I apologise for the lack of documentation. I’ll need to get onto that as well.

Indexing In UCL

Fri, 10 May 2024 12:09:46 +1100

I’ve been thinking a little about how to support indexing in UCL, as in getting elements from a list or keyed values from a map. There already exists an index builtin that does this, but I’m wondering if this can be, or even should be, supported in the language itself.

I’ve reserved . for this, and it’ll be relatively easy to make use of it to get map fields. But I do have some concerns with supporting list element dereferencing using square brackets. The big one being that if I were to use square brackets the same way that many other languages do, I suspect (although I haven’t confirmed) that it could lead to the parser treating them as two separate list literals. This is because the scanner ignores whitespace, and there’s no other syntactic indicators to separate arguments to proc calls, like commas:

echo $x[4]      --> echo $x [4]
echo [1 2 3][2] --> echo [1 2 3] [2]

So I’m not sure what to do here. I’d like to add support for . for map fields but it feels strange doing that just that and having nothing for list elements.

I can think of three ways to address this.

Do Nothing — the first option is easy: don’t add any new syntax to the language and just rely on the index builtin. TCL does with lindex, as does Lisp with nth, so I’ll be in good company here.

Use Only The Dot — the second option is to add support for the dot and not the square brackets. This is what the Go templating language does for keys of maps or structs fields. They also have an index builtin too, which will work with slice elements.

I’d probably do something similar but I may extend it to support index elements. Getting the value of a field would be what you’d expect, but to get the element of a list, the construct .(x) can be used:

echo $x.hello     \# returns the "hello" field
echo $x.(4)       \# returns the forth element of a list

One benefit of this could be that the .(x) construct would itself be a pipeline, meaning that string and calculated values could be used as well:

echo $x.("hello")
echo $x.($key)
echo $x.([1 2 3] | len)
echo $x.("hello" | toUpper)

I can probably get away with supporting this without changing the scanner or compromising the language design too much. It would be nice to add support for ditching the dot completely when using the parenthesis, a.la. BASIC, but I’d probably run into the same issues as with the square brackets if I did, so I think that’s out.

Use Parenthesis To Be Explicit — the last option is to use square brackets, and modify the grammar slightly to only allow the use of suffix expansion within parenthesis. That way, if you’d want to pass a list element as an argument, you have to use parenthesis:

echo ($x[4])       \# forth element of $x
echo $x[4]         \# $x, along with a list containing "4"

This is what you’d see in more functional languages like Elm and I think Haskell. I’ll have see whether this could work with changes to the scanner and parser if I were to go with this option. I think it may be achievable, although I’m not sure how.

An alternative way might be to go the other way, and modify the grammar rules so that the square brackets would bind closer to the list, which would mean that separate arguments involving square brackets would need to be in parenthesis:

echo $x[4]         \# forth element of $x
echo $x ([4])      \# $x, along with a list containing "4"

Or I could modify the scanner to recognise whitespace characters and use that as a guide to determine whether square brackets following a value. At least one space means the square bracket represent a element suffix, and zero mean two separate values.

So that’s where I am at the moment. I guess it all comes down to what works best for the language as whole. I can live with option one but it would be nice to have the syntax. I rather not go with option three as I’d like to keep the parser simple (I rather not add to all the new-line complexities I’ve have already).

Option two would probably be the least compromising to the design as a whole, even if the aesthetics are a bit strange. I can probably get use to them though, and I do like the idea of index elements being pipelines themselves. I may give option two a try, and see how it goes.

Anyway, more on this later.

Tape Playback Site

Sun, 05 May 2024 23:14:02 +1100

Thought I’d take a little break from UCL today.

Mum found a collection of old cassette tapes of us when we were kids, making and recording songs and radio shows. I’ve been digitising them over the last few weeks, and today the first recorded cassette was ready to share with the family.

I suppose I could’ve just given them raw MP3 files, but I wanted to record each cassette as two large files — one per side — so as to not loose much of the various crackles and clatters made when the tape recorder was stopped and started. But I did want to catalogue the more interesting points in the recording, and it would’ve been a bit “meh” simply giving them to others as one long list of timestamps (simulating the rewind/fast-forward seeking action would’ve been a step too far).

Plus, simply emailing MP3 files wasn’t nearly as interesting as what I did do, which was to put together a private site where others could browse and play the recorded tapes:

The landing page, listing the available tapes (of which there's only one right now.

Playback of a tape side, with chapter links for seeking.

The site is not much to talk about — it’s a Hugo site using the Mainroad theme and deployed to Netlify. There is some JavaScript that moves the playhead when a chapter link is clicked, but the rest is just HTML and CSS. But I did want to talk about how I got the audio files into Netlify. I wanted to use `git lfs` for this and have Netlify fetch them when building the site. Netlify doesn’t do this by default, and I get the sense that Netlify’s support for LFS is somewhat deprecated. Nevertheless, I gave it a try by adding an explicit `git lfs` step in the build to fetch the audio files. And it could’ve been that I was using the LFS command incorrectly, or maybe it was invoked at the wrong time. But whatever the reason, the command errored out and the audio files didn’t get pulled. I tried a few more times, and I probably could’ve got it working if I stuck with it, but all those deprecation warnings in Netlify’s documentation gave me pause.

So what I ended up doing was turning off builds in Netlify and using a Github Action which built the Hugo site and publish it to Netlify using the CLI tool. Here’s the Github Action in full:

name: Publish to Netify
on:
  push:
    branches: [main]
jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: true
          fetch-depth: 0    
          lfs: true
      - name: Setup Hugo
        uses: peaceiris/actions-hugo@v3
        with:
            hugo-version: '0.119.0'
      - name: Build Site
        run: |
          npm install
          hugo          
      - name: Deploy
        env:
          NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
          NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
        run: |          
          netlify deploy --dir=public --prod

This ended up working quite well: the audio files made it to Netlify and were playable on the site. The builds are also quite fast; around 55 seconds (an earlier version involved building Hugo from source, which took 5 minutes). So for anyone else interested in trying to serve LFS files via Netlify, maybe try turning off the builds and going straight to using Github Action and the CLI tool. That is… if you can swallow the price of LFS storage in Github. Oof! A little pricy. Might be that I’ll need to use something else for the audio files.

UCL: Brief Integration Update and Modules

Sat, 04 May 2024 22:14:27 +1100

A brief update of where I am with UCL and integrating it into Dynamo-browse. I did managed to get it integrated, and it’s now serving as the interpreter of commands entered in during a session.

It works… okay. I decided to avoid all the complexities I mentioned in the last post — all that about continuations, etc. — and simply kept the commands returning tea.Msg values. The original idea was to have the commands return usable values if they were invoked in a non-interactive manner. For example, the table command invoked in an interactive session will bring up the table picker for the user to select the table. But when invoked as part of a call to another command, maybe it would return the current table name as a string, or something.

But I decided to ignore all that and simply kept the commands as they are. Maybe I’ll add support for this in a few commands down the line? We’ll see. I guess it depends on whether it’s necessary.

Which brings me up to why this is only working “okay” at the moment. Some commands return a tea.Msg which ask for some input from the user. The table command is one; another is set-attr, which prompts the user to enter an attribute value. These are implemented as a message which commands the UI to go into an “input mode”, and will invoke a callback on the message when the input is entered.

This is not an issue for single commands, but it becomes one when you start entering multiple commands that prompt for input, such as two set-attr calls:

set-attr this -S ; set-attr that -S

What happens is that two messages to show the prompt are sent, but only one of them is shown to the user, while the other is simply swallowed.

Fixing this would require some re-engineering, either with how the controllers returning these messages work, or the command handlers themselves. I can probably live with this limitation for now — other than this, the UCL integration is working well — but I may need to revisit this down the line.

Modules

As for UCL itself, I’ve started working on the builtins. I’m planning to have a small set of core builtins for the most common stuff, and the rest implemented in the form of “modules”. The idea is that the core will most likely be available all the time, but the modules can be turned on and off by the language embedder based on what they need or are comfortable having.

Each module is namespaces with a prefix, such as os for operating system operations, or fs for file-system operations. I’ve chosen the colon as the namespace separator, mainly so I can reserve the dot for field dereferencing, but also because I think TCL uses the colon as a namespace separator as well (I think I saw it in some sample code). The first implementation of this was simply adding the colon to the list of characters that make up the IDENT token. This broke the parser as the colon is also use as the map key/value separator, and the parser couldn’t resolve maps anymore. I had to extend the “indent” parse rule to support multiple IDENT tokens separated by colons. The module builtins are simply added to the environment with there fully-qualified name, complete prefix and colon, and invoking them with one of these idents will just “flatten” all these colon-separated tokens into a single string. Not sophisticated, but it’ll work for now.

There aren’t many builtins for these modules at the moment: just a few for reading environment variables and getting files as list of strings. Dynamo-browse is already using this in a feature branch, and it’s allows me to finally add a long-standing feature I’ve been meaning to add for a while: automatically enabling read-only mode when accessing DynamoDB tables in production. With modules, this construct looks a little like the following:

if (eq (os:env "ENV") "prod") {
    set-opt ro
}

It would’ve been possible to do this with the scripting language already used by Dynamo-browse. But this is the motivation of integrating UCL: it makes these sorts of constructs much easier to do, almost as one would do writing a shell-script over something in C.

UCL: Breaking And Continuation

Wed, 01 May 2024 09:01:36 +1100

I’ve started trying to integrate UCL into a second tool: Dynamo Browse. And so far it’s proving to be a little difficult. The problem is that this will be replacing a dumb string splitter, with command handlers that are currently returning a tea.Msg type that change the UI in some way.

UCL builtin handlers return a interface{} result, or an error result, so there’s no reason why this wouldn’t work. But tea.Msg is also an interface{} types, so it will be difficult to tell a UI message apart from a result that’s usable as data.

This is a Dynamo Browse problem, but it’s still a problem I’ll need to solve. It might be that I’ll need to return tea.Cmd types — which are functions returning tea.Msg — and have the UCL caller detect these and dispatch them when they’re returned. That’s a lot of function closures, but it might be the only way around this (well, the alternative is returning an interface type with a method that returns a tea.Msg, but that’ll mean a lot more types than I currently have).

Anyway, more on this in the future I’m sure.

Break, Continue, Return

As for language features, I realised that I never had anything to exit early from a loop or proc. So I added break, continue, and return commands. They’re pretty much what you’d expect, except that break can optionally return a value, which will be used as the resulting value of the foreach loop that contains it:

echo (foreach [5 4 3 2 1] { |n|
  echo $n
  if (eq $n 3) {
    break "abort"
  }
})
--> 5
--> 4
--> 3
--> abort

These are implemented as error types under the hood. For example, break will return an errBreak type, which will flow up the chain until it is handled by the foreach command (continue is also an errBreak with a flag indicating that it’s a continue). Similarly, return will return an errReturn type that is handled by the proc object.

This fits quite naturally with how the scripts are run. All I’m doing is walking the tree, calling each AST node as a separate function call and expecting it to return a result or an error. If an error is return, the function bails, effectively unrolling the stack until the error is handled or it’s returned as part of the call to Eval(). So leveraging this stack unroll process already in place makes sense to me.

I’m not sure if this is considered idiomatic Go. I get the impression that using error types to handle flow control outside of adverse conditions is frowned upon. This reminds me of all the arguments against using expressions for flow control in Java. Those arguments are good ones: following executions between try and catch makes little sense when the flow can be explained more clearly with an if.

But I’m going to defend my use of errors here. Like most Go projects, the code is already littered with all the if err != nil { return err } to exit early when a non-nil error is returned. And since Go developers preach the idea of errors simply being values, why not use errors here to unroll the stack? It’s better than the alternatives: such as detecting a sentinel result type or adding a third return value which will just be yet another if bla { return res } clause.

Continuations

Now, an idea is brewing for a feature I’m calling “continuations” that might be quite difficult to implement. I’d like to provide a way for a user builtin to take a snapshot of the call stack, and resume execution from that point at a later time.

The reason for this is that I’d like all the asynchronous operations to be transparent to the UCL user. Consider a UCL script with a sleep command:

echo "Wait here"
sleep 5
echo "Ok, ready"

sleep could simply be a call to time.Sleep() but say you’re running this as part of an event loop, and you’d prefer to do something like setup a timer instead of blocking the thread. You may want to hide this from the UCL script author, so they don’t need to worry about callbacks.

Ideally, this can be implemented by the builtin using a construct similar to the following:

func sleep(ctx context.Context, arg ucl.CallArgs) (any, error) {
  var secs int
  if err := arg.Bind(&secs); err != nil {
    return err
  }

  // Save the execution stack
  continuation := args.Continuation()

  // Schedule the sleep callback
  go func() {
    <- time.After(secs * time.Seconds)

    // Resume execution later, yielding `secs` as the return value
    // of the `sleep` call. This will run the "ok, ready" echo call
    continuation(ctx, secs)
  })()

  // Halt execution now
  return nil, ucl.ErrHalt
}

The only trouble is, I’ve got no idea how I’m going to do this. As mentioned above, UCL executes the script by walking the parse tree with normal Go function calls. I don’t want to be in a position to create a snapshot of the Go call stack. That a little too low level for what I want to achieve here.

I suppose I could store the visited nodes in a list when the ErrHalt is raised; or maybe replace the Go call stack with an in memory stack, with AST node handlers being pushed and popped as the script runs. But I’m not sure this will work either. It would require a significant amount of reengineering, which I’m sure will be technically interesting, but will take a fair bit of time. And how is this to work if a continuation is made in a builtin that’s being called from another builtin? What should happen if I were to run sleep within a map, for example?

So it might be that I’ll have to use something else here. I could potentially do something using Goroutines: the script is executed on Goroutine and args.Continuation() does something like pauses it on a channel. How that would work with a builtin handler requesting the continuation not being paused themselves I’m not so sure. Maybe the handlers could be dispatched on a separate Goroutine as well?

A simpler approach might be to just offload this to the UCL user, and have them run Eval on a separate Goroutine and simply sleeping the thread. Callbacks that need input from outside could simply be sent using channels passed via the context.Context. At least that’ll lean into Go’s first party support for synchronisation, which is arguably a good thing.

UCL: The Simplifications Paid Off

Fri, 26 Apr 2024 12:13:49 +1100

The UCL simplifications have been implemented, and they seem to be largely successful.

Ripped out all the streaming types, and changed pipes to simply pass the result of the left command as first argument of the right.

"Hello" | echo ", world"
--> "Hello, world"

This has dramatically improved the use of pipes. Previously, pipes could only be used to connect streams. But now, with pretty much anything flowing through a pipe, that list of commands has extended to pretty much every builtins and user-defined procs. Furthermore, a command no longer needs to know that it’s being used in a pipeline: whatever flows through the pipe is passed transparently via the first argument to the function call. This has made pipes more useful, and usable in more situations.

Macros can still know whether there exist a pipe argument, which can make for some interesting constructs. Consider this variant of the foreach macro, which can “hang off” the end of a pipe:

["1" "2" "3"] | foreach { |x| echo $x }
--> 1
--> 2
--> 3

Not sure if this variant is useful, but I think it could be. It seems like a natural way to iterate items passed through the pipe. I’m wondering if this could extend to the if macro as well, but that variant might not be as natural to read.

Another simplification was changing the map builtin to accept anonymous blocks, as well as an “invokable” commands by name. Naturally, this also works with pipes too:

[a b c] | map { |x| toUpper $x }
--> [A B C]

[a b c] | map toUpper
--> [A B C]

As for other language features, I finally got around to adding support for integer literals. They look pretty much how you expect:

set n 123
echo $n
--> 123

One side effect of this is that an identifier can no longer start with a dash followed by a digit, as that would be parsed as the start of a negative integer. This probably isn’t a huge deal, but it could affect command switches, which are essentially just identifiers that start with a dash.

Most of the other work done was behind the scenes trying to make UCL easier to embed. I added the notion of “listable” and “hashable” proxies objects, which allow the UCL user to treat a Go slice or a Go struct as a list or hash respectively, without the embedder doing anything other than return them from a function (I’ve yet to add this support to maps just yet).

A lot of the native API is still a huge mess, and I really need to tidy it up before I’d be comfortable opening the source. Given that the language is pretty featureful now to be useful, I’ll probably start working on this next. Plus adding builtins. Really need to start adding useful builtins.

Anyway, more to come on this topic I’m sure.

Oh, one last thing: I’ve put together an online playground where you can try the language out in the browser. It’s basically a WASM build of the language running in a JavaScript terminal emulator. It was a little bit of a rush job and there’s no reason for building this other than it being a fun little thing to do.

You can try it out here, if you’re curious.

Simplifying UCL

Tue, 23 Apr 2024 09:21:44 +1100

I’ve been using UCL for several days now in that work tool I mentioned, and I’m wondering if the technical challenge that comes of making a featureful language is crowding out what I set out to do: making a useful command language that is easy to embed.

So I’m thinking of making some simplifications.

The first is to expand the possible use of pipes. To date, the only thing that can travel through pipes are streams. But many of the commands I’ve been adding simply return slices. This is probably because there’s currently no “stream” type available to the embedder, but even if there was, I’m wondering if it make sense to allow the embedder to pass slices, and other types, through pipes as well.

So, I think I’m going to take a page out of Go’s template book and simply have pipes act as syntactic sugar over sequential calls. The goal is to make the construct a | b essentially be the same as b (a), where the first argument of b will be the result of a.

As for streams, I’m thinking of removing them as a dedicated object type. Embedders could certainly make analogous types if they need to, and the language should support that, but the language will no longer offer first class support for them out of the box.

The second is to remove any sense of “purity” of the builtins. You may recall the indecision I had regarding using anonymous procs with the map command:

I’m not sure how I can improve this. I don’t really want to add automatic dereferencing of identities: they’re very useful as unquoted string arguments. I suppose I could add another construct that would support dereferencing, maybe by enclosing the identifier in parenthesis.

I think this is the wrong way to think of this. Again, I’m not here to design a pure implementation of the language. The language is meant to be easy to use, first and foremost, in an interactive shell, and if that means sacrificing purity for a map command that supports blocks, anonymous procs, and automatic dereferencing of commands just to make it easier for the user, then I think that’s a trade work taking.

Anyway, that’s the current thinking as of now.

UCL: First Embed, and Optional Arguments

Thu, 18 Apr 2024 09:39:12 +1100

Came up with a name: Universal Control Language: UCL. See, you have TCL; but what if instead of being used for tools, it can be more universal? Sounds so much more… universal, am I right? 😀

Yeah, okay. It’s not a great name. But it’ll do for now.

Anyway, I’ve started integrating this language with the admin tool I’m using at work. This tool I use is the impetus for this whole endeavour. Up until now, this tool was just a standard CLI command usable from the shell. But it’s not uncommon for me to have to invoke the tool multiple times in quick succession, and each time I invoke it, it needs to connect to backend systems, which can take a few seconds. Hence the reason why I’m converting it into a REPL.

Anyway, I added UCL to the tool, along with a readline library, and wow, did it feel good to use. So much better than the simple quote-aware string splitter I’d would’ve used. And just after I added it, I got a flurry of requests from my boss to gather some information, and although the language couldn’t quite handle the task due to missing or unfinished features, I can definitely see the potential there.

I’m trying my best to only use what will eventually be the public API to add the tool-specific bindings. The biggest issue is that these “user bindings” (i.e. the non-builtins) desperately need support for producing and consuming streams. They’re currently producing Go slices, which are being passed around as opaque “proxy objects”, but these can’t be piped into other commands to, say, filter or map. Some other major limitations:

No commands to actually filter or map. In fact, the whole standard library needs to be built out.
No ability to get fields from hashes or lists, including proxy objects which can act as lists or hashes.

One last thing that would be nice is the ability to define optional arguments. I actually started work on that last night, seeing that it’s relatively easy to build. I’m opting for a style that looks like the switches you’d find on the command line, with option names starting with dashes:

join "a" "b" -separator "," -reverse
--> b, a

Each option can have zero or more arguments, and boolean options can be represented as just having the switch. This does mean that they’d have to come after the positional arguments, but I think I can live with that. Oh, and no syntactic sugar for single-character options: each option must be separated by whitespace (the grammar actually treats them as identifiers). In fact, I’d like to discourage the use of single-character option names for these: I prefer the clarity that comes from having the name written out in full (that said, I wouldn’t rule out support for aliases). This eliminates the need for double dashes, to distinguish long option names from a cluster of single-character options, so only the single dash will be used.

I’ll talk more about how the Go bindings look later, after I’ve used them a little more and they’re a little more refined.