UCL - Bytes Slices

Spent a bit of time working on UCL this evening. This is in service of integrating it further into Dequoter, and introducing some facilities for working with hashing algorithms. More on that in a little bit, but for now, there was a need to deal with what these hashing algorithms return.

I suppose I could’ve used strings for this, since they’re little more than large arrays of runes. But I’m aware of the pitfalls that other scripting languages ran into when they did this, particularly when strings became more than simple byte arrays. And although character encoding is pretty much a solved problem now (UTF-8 FTW), using strings for this seems like such a hack. Are you really going to have a string with, say, 100 \0 escape sequences? Strings are meant for characters, and nothing more. As for arbitrary byte slices, well, something else would be needed for that. Which comes to what I’ve been doing this evening.

The first is a new object type called BytesObject, which encapsulates a simple byte slice. To create this object, I added a new builtin with the name bytes:from. This produces a BytesObject from, well, pretty much anything actually. Pass in a string, and it will return a BytesObject from the raw bytes encoding the UTF-8 runes. Provide it with a list or iterator of integers, and it will convert each one into a byte, and assemble it into a BytesObject. It also takes other BytesObject, which will copy (or concatenate) the individual args into a new slice. Passing it nil or providing no arguments will produce empty slices:

bytes:from "hello"
--> bytes[68 65 6c 6c 6f]

bytes:from [202 254 186 190]
--> bytes[ca fe ba be]

bytes:from "hello" (bytes:from "world")
--> bytes[68 65 6c 6c 6f 77 6f 72 6c 64]

bytes:from ()
--> bytes[]

Because BytesObject are collections, it’s possible to use the standard conventions to get the length or peek a byte at a given index:

bts = bytes:from "hello"

len $bts
--> 5

$bts.(2)
--> 108

A BytesObject could be converted into a UTF-8 string using the str builtin, providing the reverse of bytes:from with a string argument. But this isn’t guaranteed to work for slices that don’t encode a proper UTF-8 string.

str (bytes:from "hello")
--> hello

str (bytes:from [202 254 186 190])
--> ????

This is actually the only builtin that does this. Anything else that attempts to render a slice will get the internal representation:

echo "My bytes are " (bytes:from [202 254 186 190]) "!"
My bytes are bytes[ca fe ba be]!
(nil)

There are a few other ways to get a BytesObject from a string, such as producing a hash using the bytes:hash builtin:

bytes:hash (bytes:from "My dog at my homework") md5
--> bytes[f3 80 15 76 95 19 57 d6 54 49 10 9f 3a 42 9e 5c]

This hash builtin accepts either MD5, SHA-1, or SHA-256. And while these byte slices are distinct from strings, going from string to byte slice is usually quite safe. So for now, all the builtins that accept a BytesObject would also accept a string:

bytes:hash "My dog at my homework" md5
--> bytes[f3 80 15 76 95 19 57 d6 54 49 10 9f 3a 42 9e 5c]

As for rendering a BytesObject, so far there’s hex encoding and base64, both which produce the result as string:

dog = bytes:hash "My dog at my homework" md5

bytes:hex $dog
--> f3801576951957d65449109f3a429e5c

bytes:base64 $dog
--> 84AVdpUZV9ZUSRCfOkKeXA==

And that’s pretty much it so far. In the grand scheme of things, there’s not a lot here. But it’s enough for my immediate needs, and I think it provides a good foundation for working with bytes going forward. It would also unblock other features that have been hanging around the backlog, like decent file I/O that doesn’t assume text files. Not that those will be coming anytime soon — this is still a toy language that changes based on my needs — but their delay won’t be because of this.

Anyway, this is now in a working version of Dequoter. Why it’s there at all will be explained in due time, so stay tuned for that.