Overhead someone say, in not so many words, that my UI design for something was sub-par. 😔 Sad, but can’t deny that it’s true to a degree. Acknowledged area of potential development, I guess.
Used ChatGPT a record number of times today (5). Asked a lot of questions about Stripe, like what happens to invoices when a subscription is cancelled. Proved more useful in giving me a direction to explore, rather than providing me a definitive answer, although that might come with time and trust.
Listening to ATP #528 follow-up about putting ChatGPT in front of Siri. It occurred to me that doing so will completely defeat the purpose of Siri being there at all. After all, if you can train GPT to generate the phrases that Siri understands, wouldn’t it also be possible to train it to just produce JSON and send it to a service to execute? You might as well cut out the middle man at that point.
Updates To Dynamo-Browse
In the off-chance that anyone other than me is reading this, it’s likely that there will be no update next week due to the Easter weekend. It may not be the only weekend without an update either. If I find that I didn’t get much done for a particular week, I probably won’t say anything and leave the well to fill up for the next one (although I do have some topics planned for some of those weekends).
In fact, I was not expecting to say much this week, given that work was going through a bit of a crunch period. But the needs of work finally pushed me to add a few features to Dynamo-Browse that were sorely lacking. So that’s where most of this week’s side-project budget went to.
Dynamo-Browse
A lot of work done to query expressions in Dynamo-Browse this week, some of it touching on a few topics I mentioned in a previous update.
I’ve finally got around to finishing the between keyword, so that it works with the query planner and actually produces a DynamoDB query when used with a range[^sort] key. This means no more falling back on table scans. It’s still in a branch as of this post, but I feel much less embarrassed with merging it now, given that this support has been added.
I’ve also made a decision about how to deal with multiple index candidates. Now, when Dynamo-Browse finds that multiple indices can apply for a specific query expression, it will produce an error, requesting you to specify which index to use. This can be done by adding a using suffix to an expression, which specifies how the query should be evaluated:
color="blue" using index("color-item-index")
This can be used at any time to override the index Dynamo-Browse should use, even if only one index candidate was found. It can also be used to force the query to run as a table scan if you don’t want to use an index at all:
color="blue" using scan
Ideally you shouldn’t need to use this suffix that often. The whole purpose of query expressions was to eliminate the need for specifying details of how the query should be evaluated. But we don’t live in a perfect world, and it makes sense adding this to deal with cases where it helps to be specific.
This is also in a branch, but I’m hoping this would be merged soon as well.
Unified Expression Types and Values
A relatively large change made this week was how how values and types are represented within query expressions.
A query expression, once parsed, can be executed in multiple contexts. It can be used to generate a conditional expression for a DynamoDB query or scan, or it can be evaluated within the app itself to produce a result or alter the fields of a DynamoDB record in memory. Each of these contexts have a different set of types the expression operates on. When interpreting the expression in order to produce a result, the expression operates on types that implement types.AttributeType. This fits nicely with what the expression has to work with, which is usually the raw DynamoDB records returned by the Go client. The context used to produces conditional expressions, however, operate on a sort of hybrid type hieararchy, that supports both AttributeType and Go types. This is because to the client used to build the expression accept native Go values, which are sometimes available — particularly if they show up in the expression as a literal — but sometimes not.
But here’s the problem: I want to be able to add functions to the expression language that can be used in both contexts. I’ll get into what sort of functions I’m thinking of in a minute, but the issue is that with two sets of type hierarchies, I’d have to implement the functions twice.
Another problem is that an evaluation context operating on AttributeTypes feels very inefficient. Numbers are represented as string, and new attribute values are created on the heap. This is probably not too bad in the grand scheme of things, but it would be nice to use native Go values here, even if it’s just to avoid going from strings to numbers constantly.
So I spent most of yesterday trying to fix this. I built a new private Go interface called exprValue and added as implementing subtypes all the types supported by DynamoDB — strings, numbers, booleans, lists, etc. Values of these type implement this new interface, and can be converted to Go values or DynamoDB AttributeType values depending on the need.
Most of the evaluation logic was changed to use these types, including the builtin functions, and already I’m seeing some dramatic improvements of what’s possible now. I can define a function once and it can be evaluated both in the evaluation and query building context (provided that it’s only operating on constant values in the query building context). It also addressed some long standing issues I’ve had with the expression language, such as adding support for using a list with the in keyword; something that was not possible before:
pk in $someList
This could potentially be helpful with the “fan-out” one I mentioned a few weeks ago.
This is still early days, but I think it’s been a huge improvement to what was there before. And it’s super satisfying cleaning out all this tech-debt, especially if it means I can add features easily now.
Date Functions In The Expression Language
Now with the new type hierarchy in place, the time has come to start adding functions. What existed to date were the operators and functions that DynamoDB’s conditional expression language supported, and little else. It’s time to go beyond that. And to be honest, this was always the plan, especially given that operators like “begins with” (^=) have been there since the start.
This first thing I’m pondering now is time and date functions. The immediate issue is one of representation: in that I don’t want to settle on any specific one. I’ve seen dates stored as both string date-stamps, usually in ISO 8601, or as integer seconds from the Unix epoch, and it would be good to operate on both of these, in addition to other possible representations, like milliseconds from the Unix epoch, to some other string encoding scheme.
So what I’m thinking is an abstract date-type, probably something backed by Go’s builtin date.Time type. This will neither be a number or a string, but can be converted to one, maybe by using a keyword like as:
now() as "S" -- represent date as ISO-8601 timestamp
now() as "N" -- represent date as Unix timestamp
now() -- error: need to convert it to something
Or maybe some other mechanism.
The idea is that all the builtin functions will operate on this type, but will prevent the user from assuming a particular representation, and will force them to choose one.
I think this is something that will fit nicely with the new type hierarchy system, but for now (pun unintended), I’ll stick with Unix timestamp, just so that I can use something that is easy to implement. But to make it crystal clear that this is temporary, any such functions will have an annoying prefix.
So two new functions were added this week: the _x_now() function, which returns the current time as seconds from the Unix epoch as a number; and the _x_add(), which returns the sum of two numbers. Much like the time functions, I’d like to eventually add arithmetic operators like + to the expression language, but I needed something now and I didn’t have much time to work on that.
Attribute Commands
Finally, a few random notes about commands dealing with attribute values.
The set-attr command can now accept the switch -to, which can be used to set the attribute to the result of a query expression. No more copying-and-pasting values, and operating on them outside Dynamo-Browse.
The good thing about this is that the previous attribute values are available in the value expression, so you can use this switch to set the value of attribute based on other attributes in a row. This comes in super handy with bulk changes. I’ve used this to adjust the value of TTLs in a table I’m working in. To set the TTL to be 10 minutes into the future, I just marked the rows, entered the command set-attr -to ttl, and use the expression _x_add(_x_now(), 600). Super useful.
Also, I’ve found a bug where the del-attr command does not work with marked items. It’ll only delete attributes from the item that’s selected (i.e. in pink). I haven’t got around to fixing this, but I hope to very soon.
I think that’s all for this week. Until next time.
What would be a nice addition to the spell-check suggestions menu is a brief (3-5 words) definition of the word. I always find myself choosing the wrong suggestion, and a feature like this would help a lot.
When I first saw John Gruber’s post post about Wavelength, I immediately dismissed it as yet another app I couldn’t use, forgetting that I actually do have devices I can try it on. So I’m trying out the Mac app now. First few minutes of using it: yeah, quite a nice app. We’ll see how we go with it.
Currently reading: The Brand You 50 (Reinventing Work) by Tom Peters 📚
When three different bloggers you follow all write about the author retiring, you pay attention.
First Posts Of The Day
It’s bit strange how the first post of the day can always feel like the hardest to get out. Every one after it is so much easier to write.
I wonder if it’s because when faced with an empty text-box, there are these grand plans about what I’m going to write, as if everyone reading this is hanging on my every word: it’ll be my masterpiece of wit, inspiration, and insightfulness that will spread far and wide and blow the minds of everrryyywoonnneee1. Then I write something, and naturally it falls far short of these expectations: mundane, unimportant, already said before2.
Then I say to myself, “ah well, at least it’s written down.” And with that, the expected level of quality for anything else that day has been set.
So, this is today’s first post. More might come, probably along the same level of importance as this one. At least until the tomorrow, when the cycle starts again.
-
Another possibility is that I feel I need to write something at the same level of quality of those that I read. That’s probably not a bad feeling to have; but, at least for me, it can get in the way of writing anything at all that day. ↩︎
-
And lets not forget the bad spelling and grammar I failed to catch. ↩︎
Completely forgot how to commute properly. This is the second day this week I forget to bring my umbrella on a day with forecasted rain. I got lucky last time. Today, not so much. 🌧️
Ugh! Mountains of work to do and only a week to do it all. Gonna be a bird-by-bird sort of day today. But first… coffee (well, coffee number two actually).
Left work late and now caught in a train suspension due to an accident (someone got hit by a train). So… dining out this evening.
Learnt a lot about digital video today. Fascinating stuff. Didn’t realise that frame rates can be non-integers (59.xx FPS). Turns out it’s a legacy of analogue TV, where they try to squeeze colour into bandwidth originally designed for a B&W signal. Shoehorns all the way down I guess.
There are nice things about having separate IDEs for different languages — GoLand for Go, WebStorm for HTML+JS, etc. — but I can see the advantages of using a single one for everything. I was trying to resolve merge conflicts with my customised GoLand key-bindings and I was getting confused as they weren’t working. Turns out the reason was that I was actually using WebStorm, not GoLand.
The two IDEs are so alike I wonder if things like key-bindings shouldn’t just apply to all installed IDEs on a system. I guess since JetBrains is working on a brand new IDE it probably doesn’t matter at this stage.
Love hearing from successful bloggers who’ve recently celebrated a major milestone (Manton, Ben Thompson, Kottke) that they thought that they were late to the party. Just shows that when you start is less important than just keeping at it.
Got a small envelope with a US stamp delivered today. No idea what it was. Certainly wasn’t expecting anything.
(opens it up) Ah, my Incomparable membership notebook has arrived. How cool! I completely forgot about this, which makes it the best sort of mail to get.
🔗 Google is killing most of Fitbit’s social features today
An amusing thought came to me while I read this: Google has an opportunity to play to it’s strength and act like the assassin for features or services. Don’t want to support something? Get Google to acquire you and inevitably shut you down. It’s such a unique niche that companies should be paying Google for this service.
I’m not a Fitbit user, but I know how it feels to be burned by Google’s obsessive need to shut down things I find useful, so I can understand all the upset over this.
The Android Obsidian app has been playing up recently. Sometimes when I try to make a new note, I loose the ability to add new lines. I have no idea what’s going on, but it’s making me sad. The mobile apps were the reason why I tried Obsidian again. But they’re only useful when they work. 🙁
Bocce at Carlton Gardens this afternoon. We probably played our fastest game of speed bocce today, clocking in at 13 minutes. Second fastest was 14 minutes, also played this afternoon. Amazing how fast you could go if parking tickets are on the line.
Updating Bocce Scorecard
I didn’t get to a lot of side-project work this week, but I did have to make a large change to a project we use to track scores for our “bocce club”. So I’d though I’d say a few words about that today.
We had our bocce “grand final” a few weeks ago, and one of the matches resulted in a tie between two players. Unfortunately, the Bocce Scorecard web-app I build could not properly handle these, which meant that I had to fix it.
I’ll be honest in saying that that this was never really well fleshed out in the code, and there was actually a small bug which didn’t handle the ranking of players well. But I was pushing to keep this app as the de-facto source of truth for these matches, and there was a bit riding on this being correct (there’s a $4 trophy involved). So I had to get this fix before our next match, which was today.
Now, I been having trouble coming up with a good description of what the rules should be so I’d figured a simple example would suffice.
Imagine that there are four players: Tom, Dick, Harry, and Sally. They play several bocce matches during a season — which roughly corresponds to one calendar year — plus three “grand final” matches at the end. Each player would be awarded a number of “season points” (we informally call them “cookies”) based on how well they did in the match. The person with the most season points at the end of the last grand final match wins the season, and gets to take home the trophy.
In regular matches, the wining player is awarded one season point, while the remaining players get nothing:
| Player | Score | Season Points |
|---|---|---|
| Tom | 11 | 1 |
| Dick | 8 | 0 |
| Sally | 6 | 0 |
| Harry | 3 | 0 |
In grand final matches, the winning player is awarded 5 points, the one coming in second gets 2, and the one coming in third gets 1:
| Player | Score | Season Points |
|---|---|---|
| Tom | 11 | 5 |
| Dick | 8 | 2 |
| Sally | 6 | 1 |
| Harry | 3 | 0 |
Season points for grand final matches are distributed this way so that a single grand final match is roughly equivalent to an entire season of regular matches. This means that if someone is coming in last during the regular season (fun fact: that person’s usually me) they still has a chance to win the trophy if they do well during the grand final.
Now, let’s say that our four players are playing a grand final match, and Dick and Sally tie for second place. What should happen is that both Dick and Sally should be awarded half the season points they would get for both the second and third rank, given that they are evenly match for these two positions. In other words, they should both get 1.5 season points (1 + 2 = 3 / 2 = 1.5). Harry, who came last, still gets zero.
| Player | Score | Season Points |
|---|---|---|
| Tom | 11 | 5 |
| Dick | 7 | 1.5 |
| Sally | 7 | 1.5 |
| Harry | 3 | 0 |
This was the rule that I needed to change.
What I found when I started working on this is that the rule definitions themselves needed to be closer to how the players are ranked. What was previously done was that the players were sorted based on their match score, and then the rules were applied to each one by checking the win condition and awarding the points if they match it. But this didn’t fit nicely with this new approach to ties.
So instead of the conditions and awards approach, I simplified the rule definitions such that it simply defines the number of season points based on the players rank. This effectively makes it a simple map between rank and points. For normal matches the mapping would look like this:
| Rank | Season Points |
|---|---|
| 1 | 1 |
and for grand final matches, like this:
| Rank | Season Points |
|---|---|
| 1 | 5 |
| 2 | 2 |
| 3 | 1 |
Now, when a match is over, the logic that awards the season points first sorts the players based on their match score, and then groups the players into buckets such that all the players with same match score are lumped together in the same bucket. Ranks are then assigned to the players in descending score order. If two players have the same score, they will be given two ranks (e.g. Dick and Sally would have both rank two and three). Finally, season points are awarded with the rule definition and the following formula:
season_points(player) = sum[over player_ranks](rules.rank_scores[rank]) / no_of_players_in_bucket
This new logic works for ties between any number of players with any ranks.
But the introduction of division now means that the season points can be a decimal, and the database row that holds the season points is an integer type. I didn’t want to make it a floating point, so I took a page from Stripe and simply changed the representation of the season scores such that 1 season point is represented as 100 in the database. This is exposed in the rules configuration, which now looks like this:
{
"rank_scores": [
{
"points": 500,
"rank": 1
},
{
"points": 200,
"rank": 2
},
{
"points": 100,
"rank": 3
}
]
}
although all the non-admin screens properly represents the score as a decimal number.
I managed to get all finished and pushed to the server, but there was one other thing I think I’d like to get done down the line. My friends have been asking me about the outcome of previous seasons recently and I’d like to make it easier for them to view it themselves. The data exists, but it’s super hacky to get: you need to “open” a previous season so that the leader board is shown on the home page, then close it again once the info is seen. This can only be done by the admin user (i.e. me) and the screens to do it leave a lot to be desired:
What I’m thinking is adding a “Seasons” section in the web-app. Clicking “Seasons” in the nav will bring up the following screen:
The game variant will appear the top as a tab, and below them are all the current and past seasons arranged in descending chronological order. Clicking the > will bring up the season results display:
This will show the final outcome of the season, any metadata associated with the season, and the matches of the season, along with the winner. Clicking the location will bring up the particular bocce session so that all the matches played that day can be seen.
We’ll see when I get around to building this. It’s actually been a while since I’ve last touched this project while making such a large feature.
Oh, and since it’s been a while, this usually means I needed to upgrade Buffalo, the framework this app is using. Doing this usually means that you’ll need to change your app in some way to handle the new build process. This time, it’s moving the main.go file, previously in the project directory, into a cmd/app directory. When you see output like this:
leonmika@Stark bocce-scorecard % buffalo build -o /tmp/app
Usage:
buffalo build [flags]
Aliases:
build, b, bill, install
Flags:
--build-flags strings Additional comma-separated build flags to feed to go build
--clean-assets will delete public/assets before calling webpack
--dry-run runs the build 'dry'
--environment string set the environment for the binary (default "development")
-e, --extract-assets extract the assets and put them in a distinct archive
-h, --help help for build
--ldflags string set any ldflags to be passed to the go build
--mod string -mod flag for go build
-o, --output string set the name of the binary
-k, --skip-assets skip running webpack and building assets
--skip-build-deps skip building dependencies
--skip-template-validation skip validating templates
-s, --static build a static binary using --ldflags '-linkmode external -extldflags "-static"'
-t, --tags string compile with specific build tags
-v, --verbose print debugging information
ERRO[0000] Error: open cmd/app/main.go: no such file or directory
You’ll need to create a cmd/app directory and move main.go into the cmd/app directory.
This will get the build working again but it will break buffalo dev as it could no longer find the main file in the project directory. To fix that, you’ll need to open up .buffalo.dev.yml and add the following property:
build_target_path: "./cmd/app"
This will get the dev build working again.
I don’t know why the dev command honours this config, yet the build command chooses to look at a hard coded path. Wouldn’t it have been easier to express this in a single configuration file?
And let’s not leave Node out of the cold. If you’re trying to run buffalo build and you’re getting this error:
#21 12.21 node:internal/crypto/hash:71
#21 12.21 this[kHandle] = new _Hash(algorithm, xofLen);
#21 12.21 ^
#21 12.21
#21 12.21 Error: error:0308010C:digital envelope routines::unsupported
#21 12.21 at new Hash (node:internal/crypto/hash:71:19)
#21 12.21 at Object.createHash (node:crypto:133:10)
#21 12.21 at BulkUpdateDecorator.hashFactory (/src/bocce_scorecard/node_modules/webpack/lib/util/createHash.js:145:18)
#21 12.21 at BulkUpdateDecorator.update (/src/bocce_scorecard/node_modules/webpack/lib/util/createHash.js:46:50)
#21 12.21 at RawSource.updateHash (/src/bocce_scorecard/node_modules/webpack/node_modules/webpack-sources/lib/RawSource.js:77:8)
#21 12.21 at NormalModule._initBuildHash (/src/bocce_scorecard/node_modules/webpack/lib/NormalModule.js:888:17)
#21 12.21 at handleParseResult (/src/bocce_scorecard/node_modules/webpack/lib/NormalModule.js:954:10)
#21 12.21 at /src/bocce_scorecard/node_modules/webpack/lib/NormalModule.js:1048:4
#21 12.21 at processResult (/src/bocce_scorecard/node_modules/webpack/lib/NormalModule.js:763:11)
#21 12.21 at /src/bocce_scorecard/node_modules/webpack/lib/NormalModule.js:827:5 {
#21 12.21 opensslErrorStack: [ 'error:03000086:digital envelope routines::initialization error' ],
#21 12.21 library: 'digital envelope routines',
#21 12.21 reason: 'unsupported',
#21 12.21 code: 'ERR_OSSL_EVP_UNSUPPORTED'
#21 12.21 }
You’ll need to enable the legacy OpenSSL provider using a Node option:
export NODE_OPTIONS=--openssl-legacy-provider
Yeah, building to a framework is always fun. 😏
So that’s it for this weeks update. I spent some time on Dynamo-Browse this week as well, but I haven’t actually finished that work and this log entry is long enough, so I might say more about that next week.
Doing some changes for a hobby software project I’ve made for a few friends. The whole “software is never finished” can be a drag sometimes: you’re forever on the hook to make changes. But I guess the flip side of that is that your friends find value in what you built. Otherwise, they wouldn’t ask.