Planet Scheme

Wednesday, May 22, 2024

Andy Wingo

growing a bootie

Following on last week’s egregious discussion of the Hoot Scheme-to-WebAssembly compiler bootie, today I would like to examine another axis of boot, which is a kind of rebased branch of history: not the hack as it happened, but the logic inside the hack, the structure of the built thing, the history as it might have been. Instead of describing the layers of shims and props that we used while discovering what were building, let’s look at how we would build Hoot again, if we had to.

I think many readers of this blog will have seen Growing a Language, a talk / performance art piece in which Guy L. Steele—I once mentioned to him that Guy L. was one of the back-justifications for the name Guile; he did not take it well—in which Steele takes the set of monosyllabic words as primitives and builds up a tower of terms on top, bootstrapping a language as he goes. I just watched it again and I think it holds up, probably well enough to forgive the superfluous presence of the gender binary in the intro; ideas were different in the 1900s.

It is in the sense of that talk that I would like to look at growing a Hoot: how Hoot defines nouns and verbs in terms of smaller, more primitive terms: terms in terms of terms.

(hoot features) features (hoot primitives) primitives (ice-9 match) match (ice-9 match):s->(hoot primitives):n (hoot eq) eq (ice-9 match):s->(hoot eq):n (hoot pairs) pairs (ice-9 match):s->(hoot pairs):n (hoot vectors) vectors (ice-9 match):s->(hoot vectors):n (hoot equal) equal (ice-9 match):s->(hoot equal):n (hoot lists) lists (ice-9 match):s->(hoot lists):n (hoot errors) errors (ice-9 match):s->(hoot errors):n (hoot numbers) numbers (ice-9 match):s->(hoot numbers):n (fibers scheduler) scheduler (hoot ffi) ffi (fibers scheduler):s->(hoot ffi):n (guile) (guile) (fibers scheduler):s->(guile):n (fibers channels) channels (fibers channels):s->(ice-9 match):n (fibers waiter-queue) waiter-queue (fibers channels):s->(fibers waiter-queue):n (fibers operations) operations (fibers channels):s->(fibers operations):n (fibers channels):s->(guile):n (srfi srfi-9) srfi-9 (fibers channels):s->(srfi srfi-9):n (fibers waiter-queue):s->(ice-9 match):n (fibers waiter-queue):s->(fibers operations):n (fibers waiter-queue):s->(guile):n (fibers waiter-queue):s->(srfi srfi-9):n (fibers promises) promises (fibers promises):s->(fibers operations):n (hoot exceptions) exceptions (fibers promises):s->(hoot exceptions):n (fibers promises):s->(hoot ffi):n (fibers promises):s->(guile):n (fibers conditions) conditions (fibers conditions):s->(ice-9 match):n (fibers conditions):s->(fibers waiter-queue):n (fibers conditions):s->(fibers operations):n (fibers conditions):s->(guile):n (fibers conditions):s->(srfi srfi-9):n (fibers timers) timers (fibers timers):s->(fibers scheduler):n (fibers timers):s->(fibers operations):n (scheme time) time (fibers timers):s->(scheme time):n (fibers timers):s->(guile):n (fibers operations):s->(ice-9 match):n (fibers operations):s->(fibers scheduler):n (hoot boxes) boxes (fibers operations):s->(hoot boxes):n (fibers operations):s->(guile):n (fibers operations):s->(srfi srfi-9):n (hoot eq):s->(hoot primitives):n (hoot syntax) syntax (hoot eq):s->(hoot syntax):n (hoot strings) strings (hoot strings):s->(hoot primitives):n (hoot strings):s->(hoot eq):n (hoot strings):s->(hoot pairs):n (hoot bytevectors) bytevectors (hoot strings):s->(hoot bytevectors):n (hoot strings):s->(hoot lists):n (hoot bitwise) bitwise (hoot strings):s->(hoot bitwise):n (hoot char) char (hoot strings):s->(hoot char):n (hoot strings):s->(hoot errors):n (hoot strings):s->(hoot numbers):n (hoot match) match (hoot strings):s->(hoot match):n (hoot pairs):s->(hoot primitives):n (hoot bitvectors) bitvectors (hoot bitvectors):s->(hoot primitives):n (hoot bitvectors):s->(hoot bitwise):n (hoot bitvectors):s->(hoot errors):n (hoot bitvectors):s->(hoot match):n (hoot vectors):s->(hoot primitives):n (hoot vectors):s->(hoot pairs):n (hoot vectors):s->(hoot lists):n (hoot vectors):s->(hoot errors):n (hoot vectors):s->(hoot numbers):n (hoot vectors):s->(hoot match):n (hoot equal):s->(hoot primitives):n (hoot equal):s->(hoot eq):n (hoot equal):s->(hoot strings):n (hoot equal):s->(hoot pairs):n (hoot equal):s->(hoot bitvectors):n (hoot equal):s->(hoot vectors):n (hoot records) records (hoot equal):s->(hoot records):n (hoot equal):s->(hoot bytevectors):n (hoot not) not (hoot equal):s->(hoot not):n (hoot values) values (hoot equal):s->(hoot values):n (hoot hashtables) hashtables (hoot equal):s->(hoot hashtables):n (hoot equal):s->(hoot numbers):n (hoot equal):s->(hoot boxes):n (hoot equal):s->(hoot match):n (hoot exceptions):s->(hoot features):n (hoot exceptions):s->(hoot primitives):n (hoot exceptions):s->(hoot pairs):n (hoot exceptions):s->(hoot records):n (hoot exceptions):s->(hoot lists):n (hoot exceptions):s->(hoot syntax):n (hoot exceptions):s->(hoot errors):n (hoot exceptions):s->(hoot match):n (hoot cond-expand) cond-expand (hoot exceptions):s->(hoot cond-expand):n (hoot parameters) parameters (hoot parameters):s->(hoot primitives):n (hoot fluids) fluids (hoot parameters):s->(hoot fluids):n (hoot parameters):s->(hoot errors):n (hoot parameters):s->(hoot cond-expand):n (hoot records):s->(hoot primitives):n (hoot records):s->(hoot eq):n (hoot records):s->(hoot pairs):n (hoot records):s->(hoot vectors):n (hoot symbols) symbols (hoot records):s->(hoot symbols):n (hoot records):s->(hoot lists):n (hoot records):s->(hoot values):n (hoot records):s->(hoot bitwise):n (hoot records):s->(hoot errors):n (hoot ports) ports (hoot records):s->(hoot ports):n (hoot records):s->(hoot numbers):n (hoot records):s->(hoot match):n (hoot keywords) keywords (hoot records):s->(hoot keywords):n (hoot records):s->(hoot cond-expand):n (hoot dynamic-wind) dynamic-wind (hoot dynamic-wind):s->(hoot primitives):n (hoot dynamic-wind):s->(hoot syntax):n (hoot bytevectors):s->(hoot primitives):n (hoot bytevectors):s->(hoot bitwise):n (hoot bytevectors):s->(hoot errors):n (hoot bytevectors):s->(hoot match):n (hoot error-handling) error-handling (hoot error-handling):s->(hoot primitives):n (hoot error-handling):s->(hoot pairs):n (hoot error-handling):s->(hoot exceptions):n (hoot write) write (hoot error-handling):s->(hoot write):n (hoot control) control (hoot error-handling):s->(hoot control):n (hoot error-handling):s->(hoot fluids):n (hoot error-handling):s->(hoot errors):n (hoot error-handling):s->(hoot ports):n (hoot error-handling):s->(hoot numbers):n (hoot error-handling):s->(hoot match):n (hoot error-handling):s->(hoot cond-expand):n (hoot ffi):s->(hoot primitives):n (hoot ffi):s->(hoot strings):n (hoot ffi):s->(hoot pairs):n (hoot procedures) procedures (hoot ffi):s->(hoot procedures):n (hoot ffi):s->(hoot lists):n (hoot ffi):s->(hoot not):n (hoot ffi):s->(hoot errors):n (hoot ffi):s->(hoot numbers):n (hoot ffi):s->(hoot cond-expand):n (hoot debug) debug (hoot debug):s->(hoot primitives):n (hoot debug):s->(hoot match):n (hoot symbols):s->(hoot primitives):n (hoot symbols):s->(hoot errors):n (hoot assoc) assoc (hoot assoc):s->(hoot primitives):n (hoot assoc):s->(hoot eq):n (hoot assoc):s->(hoot pairs):n (hoot assoc):s->(hoot equal):n (hoot assoc):s->(hoot lists):n (hoot assoc):s->(hoot not):n (hoot procedures):s->(hoot primitives):n (hoot procedures):s->(hoot syntax):n (hoot write):s->(hoot primitives):n (hoot write):s->(hoot eq):n (hoot write):s->(hoot strings):n (hoot write):s->(hoot pairs):n (hoot write):s->(hoot bitvectors):n (hoot write):s->(hoot vectors):n (hoot write):s->(hoot records):n (hoot write):s->(hoot bytevectors):n (hoot write):s->(hoot symbols):n (hoot write):s->(hoot procedures):n (hoot write):s->(hoot bitwise):n (hoot write):s->(hoot char):n (hoot write):s->(hoot errors):n (hoot write):s->(hoot ports):n (hoot write):s->(hoot numbers):n (hoot write):s->(hoot keywords):n (hoot lists):s->(hoot primitives):n (hoot lists):s->(hoot pairs):n (hoot lists):s->(hoot values):n (hoot lists):s->(hoot numbers):n (hoot lists):s->(hoot match):n (hoot lists):s->(hoot cond-expand):n (hoot not):s->(hoot syntax):n (hoot syntax):s->(hoot primitives):n (hoot values):s->(hoot primitives):n (hoot values):s->(hoot syntax):n (hoot control):s->(hoot primitives):n (hoot control):s->(hoot parameters):n (hoot control):s->(hoot values):n (hoot control):s->(hoot cond-expand):n (hoot bitwise):s->(hoot primitives):n (hoot char):s->(hoot primitives):n (hoot char):s->(hoot bitvectors):n (hoot char):s->(hoot bitwise):n (hoot char):s->(hoot errors):n (hoot char):s->(hoot match):n (hoot dynamic-states) dynamic-states (hoot dynamic-states):s->(hoot primitives):n (hoot dynamic-states):s->(hoot vectors):n (hoot dynamic-states):s->(hoot debug):n (hoot dynamic-states):s->(hoot lists):n (hoot dynamic-states):s->(hoot values):n (hoot dynamic-states):s->(hoot errors):n (hoot dynamic-states):s->(hoot numbers):n (hoot dynamic-states):s->(hoot match):n (hoot read) read (hoot read):s->(hoot primitives):n (hoot read):s->(hoot eq):n (hoot read):s->(hoot strings):n (hoot read):s->(hoot pairs):n (hoot read):s->(hoot bitvectors):n (hoot read):s->(hoot vectors):n (hoot read):s->(hoot exceptions):n (hoot read):s->(hoot symbols):n (hoot read):s->(hoot lists):n (hoot read):s->(hoot not):n (hoot read):s->(hoot values):n (hoot read):s->(hoot char):n (hoot read):s->(hoot errors):n (hoot read):s->(hoot ports):n (hoot read):s->(hoot numbers):n (hoot read):s->(hoot match):n (hoot read):s->(hoot keywords):n (hoot hashtables):s->(hoot primitives):n (hoot hashtables):s->(hoot eq):n (hoot hashtables):s->(hoot pairs):n (hoot hashtables):s->(hoot vectors):n (hoot hashtables):s->(hoot procedures):n (hoot hashtables):s->(hoot lists):n (hoot hashtables):s->(hoot values):n (hoot hashtables):s->(hoot bitwise):n (hoot hashtables):s->(hoot errors):n (hoot hashtables):s->(hoot numbers):n (hoot fluids):s->(hoot primitives):n (hoot fluids):s->(hoot cond-expand):n (hoot errors):s->(hoot primitives):n (hoot atomics) atomics (hoot atomics):s->(hoot primitives):n (hoot ports):s->(hoot primitives):n (hoot ports):s->(hoot eq):n (hoot ports):s->(hoot strings):n (hoot ports):s->(hoot pairs):n (hoot ports):s->(hoot vectors):n (hoot ports):s->(hoot parameters):n (hoot ports):s->(hoot bytevectors):n (hoot ports):s->(hoot procedures):n (hoot ports):s->(hoot lists):n (hoot ports):s->(hoot not):n (hoot ports):s->(hoot values):n (hoot ports):s->(hoot bitwise):n (hoot ports):s->(hoot char):n (hoot ports):s->(hoot errors):n (hoot ports):s->(hoot numbers):n (hoot ports):s->(hoot boxes):n (hoot ports):s->(hoot match):n (hoot ports):s->(hoot cond-expand):n (hoot numbers):s->(hoot primitives):n (hoot numbers):s->(hoot eq):n (hoot numbers):s->(hoot not):n (hoot numbers):s->(hoot values):n (hoot numbers):s->(hoot bitwise):n (hoot numbers):s->(hoot errors):n (hoot numbers):s->(hoot match):n (hoot boxes):s->(hoot primitives):n (hoot match):s->(hoot primitives):n (hoot match):s->(hoot errors):n (hoot keywords):s->(hoot primitives):n (hoot cond-expand):s->(hoot features):n (hoot cond-expand):s->(hoot primitives):n (scheme lazy) lazy (scheme lazy):s->(hoot primitives):n (scheme lazy):s->(hoot records):n (scheme lazy):s->(hoot match):n (scheme base) base (scheme lazy):s->(scheme base):n (scheme load) load (scheme load):s->(hoot primitives):n (scheme load):s->(hoot errors):n (scheme load):s->(scheme base):n (scheme complex) complex (scheme complex):s->(hoot numbers):n (scheme time):s->(hoot primitives):n (scheme time):s->(scheme base):n (scheme file) file (scheme file):s->(hoot primitives):n (scheme file):s->(hoot errors):n (scheme file):s->(hoot ports):n (scheme file):s->(hoot match):n (scheme file):s->(scheme base):n (scheme write) write (scheme write):s->(hoot write):n (scheme eval) eval (scheme eval):s->(hoot errors):n (scheme eval):s->(scheme base):n (scheme inexact) inexact (scheme inexact):s->(hoot primitives):n (scheme inexact):s->(hoot numbers):n (scheme char) char (scheme char):s->(hoot primitives):n (scheme char):s->(hoot bitwise):n (scheme char):s->(hoot char):n (scheme char):s->(hoot numbers):n (scheme char):s->(scheme base):n (scheme process-context) process-context (scheme process-context):s->(hoot primitives):n (scheme process-context):s->(hoot errors):n (scheme process-context):s->(scheme base):n (scheme cxr) cxr (scheme cxr):s->(hoot pairs):n (scheme read) read (scheme read):s->(hoot read):n (scheme base):s->(hoot features):n (scheme base):s->(hoot primitives):n (scheme base):s->(hoot eq):n (scheme base):s->(hoot strings):n (scheme base):s->(hoot pairs):n (scheme base):s->(hoot vectors):n (scheme base):s->(hoot equal):n (scheme base):s->(hoot exceptions):n (scheme base):s->(hoot parameters):n (scheme base):s->(hoot dynamic-wind):n (scheme base):s->(hoot bytevectors):n (scheme base):s->(hoot error-handling):n (scheme base):s->(hoot symbols):n (scheme base):s->(hoot assoc):n (scheme base):s->(hoot procedures):n (scheme base):s->(hoot write):n (scheme base):s->(hoot lists):n (scheme base):s->(hoot not):n (scheme base):s->(hoot syntax):n (scheme base):s->(hoot values):n (scheme base):s->(hoot control):n (scheme base):s->(hoot char):n (scheme base):s->(hoot read):n (scheme base):s->(hoot errors):n (scheme base):s->(hoot ports):n (scheme base):s->(hoot numbers):n (scheme base):s->(hoot match):n (scheme base):s->(hoot cond-expand):n (scheme base):s->(srfi srfi-9):n (scheme repl) repl (scheme repl):s->(hoot errors):n (scheme repl):s->(scheme base):n (scheme r5rs) r5rs (scheme r5rs):s->(scheme lazy):n (scheme r5rs):s->(scheme load):n (scheme r5rs):s->(scheme complex):n (scheme r5rs):s->(scheme file):n (scheme r5rs):s->(scheme write):n (scheme r5rs):s->(scheme eval):n (scheme r5rs):s->(scheme inexact):n (scheme r5rs):s->(scheme char):n (scheme r5rs):s->(scheme process-context):n (scheme r5rs):s->(scheme cxr):n (scheme r5rs):s->(scheme read):n (scheme r5rs):s->(scheme base):n (scheme r5rs):s->(scheme repl):n (scheme case-lambda) case-lambda (scheme case-lambda):s->(hoot primitives):n (fibers) (fibers) (fibers):s->(fibers scheduler):n (fibers):s->(guile):n (guile):s->(hoot features):n (guile):s->(hoot primitives):n (guile):s->(ice-9 match):n (guile):s->(hoot eq):n (guile):s->(hoot strings):n (guile):s->(hoot pairs):n (guile):s->(hoot bitvectors):n (guile):s->(hoot vectors):n (guile):s->(hoot equal):n (guile):s->(hoot exceptions):n (guile):s->(hoot parameters):n (guile):s->(hoot dynamic-wind):n (guile):s->(hoot bytevectors):n (guile):s->(hoot error-handling):n (guile):s->(hoot symbols):n (guile):s->(hoot assoc):n (guile):s->(hoot procedures):n (guile):s->(hoot write):n (guile):s->(hoot lists):n (guile):s->(hoot not):n (guile):s->(hoot syntax):n (guile):s->(hoot values):n (guile):s->(hoot control):n (guile):s->(hoot bitwise):n (guile):s->(hoot char):n (guile):s->(hoot dynamic-states):n (guile):s->(hoot read):n (guile):s->(hoot fluids):n (guile):s->(hoot errors):n (guile):s->(hoot ports):n (guile):s->(hoot numbers):n (guile):s->(hoot boxes):n (guile):s->(hoot keywords):n (guile):s->(hoot cond-expand):n (guile):s->(scheme lazy):n (guile):s->(scheme time):n (guile):s->(scheme file):n (guile):s->(scheme char):n (guile):s->(scheme process-context):n (guile):s->(scheme base):n (guile):s->(srfi srfi-9):n (srfi srfi-9):s->(hoot primitives):n (srfi srfi-9):s->(hoot records):n

If you are reading this on the web, you should see above a graph of dependencies among the 50 or so libraries that are shipped as part of Hoot. (Somehow I doubt that a feed reader will plumb through the inline SVG, but who knows.) It’s a bit of a mess, but still I think it’s a useful illustration of a number of properties of how the Hoot language is grown from small to large. Click on any box to visit the source code for that module.

the root of the boot

Firstly, let us note that the graph is not a forest: it is a single tree. There is no module that does not depend (possibly indirectly) on (hoot primitives). This is because there are no capabilities that Hoot libraries can access without importing them, and the only way into the Hootosphere from outside is via the definitions in the primitives module.

So what are these definitions, you might ask? Well, these are the “well-known” bindings, for example + for which the compiler might have some special understanding, the sort of binding that gets translated to a primitive operation at the compiler IR level. They are used in careful ways by the modules that use (hoot primitives) to ensure that their uses are all open-coded by the compiler. (“Open coding” is inlining. But inlining to me implies that the whole implementation is inlined, with no slow-path callouts, whereas open coding implies to me that it’s the compiler that knows what the op does and may or may not inline the actual asm.)

But, (hoot primitives) also exposes some other definitions, for example define and let and lambda and all that. Scheme doesn’t have keywords in the sense that Python has def and with and such: there is no privileged way to associate a name with its meaning. It is in this sense that it is impossible to avoid (hoot primitives): the most simple (define x 42) depends on the lexical meaning of define, which is provided by the primitives module.

Syntax definitions are an expander construct; they are not present at run-time. Using a syntax definition causes the expander to invoke code, and the expander runs on the host system, which is Guile and not WebAssembly. So, syntax definitions belong to the host. This goes also for some first-order definitions such as syntax->datum and so on, which are only used in syntax expanders; these definitions are plumbed through (hoot primitives), but can only ever be used by macro definitions, which run on the meta-level.

(Is this too heavy? Allow me to lighten the mood: when I was 22 or so and working in Namibia, I somehow got an advance copy of Notes from the Metalevel. I was working on algorithmic music synthesis, and my chief strategy was knocking hubris together with itself, as one does. I sent the author a bunch of uninvited corrections to his book. I think it was completely unwelcome! Anyway, moral of the story, at 22 you get a free pass to do whatever you want, and come to think of it, now that I am 44 I think I should get some kind of hubris loyalty award or something.)

powerful primitives

So, there are expand-time primitives and run-time primitives. The expander knows about expand-time primitives and the compiler knows about run-time primitives. One particularly powerful primitive is %inline-wasm, which takes an inline snippet of WebAssembly as an s-expression and applies it to a number of arguments passed at run-time. Consider make-bytevector:

(define* (make-bytevector len #:optional (init 0))
  (%inline-wasm
   '(func (param $len i32) (param $init i32)
      (result (ref eq))
      (struct.new
       $mutable-bytevector
       (i32.const 0)
       (array.new $raw-bytevector
                  (local.get $init)
                  (local.get $len))))
   len init))

We have an inline snippet of wasm that makes a $mutable-bytevector. It passes 0 as the hash field, meaning that the hashq of this value will be lazily initialized, and the contents are a new array of a given size and initial value. Inputs will be unboxed to the appropriate type (two i32s in this case), and likewise with outputs; here we produce the universal (ref eq) representation.

The nice thing about %inline-wasm is that the compiler didn’t have to be taught about make-bytevector: this definition suffices, because %inline-wasm can access a number of lower-level capabilities.

dual denotations

But as we learned in my notes on whole-program compilation, any run-time definition is available at compile-time, if it is reachable from a syntax transformer. So this definition above isn’t quite sufficient; we can’t call make-bytevector as part of a procedural macro, which we might want to do. What we need instead is to provide one definition when residualizing wasm at run-time, and another when loading a module at expand-time.

In Hoot we do this with cond-expand, where we expand to %inline-wasm when targetting Hoot, and... what, precisely, at expand-time? Really we need to make a Guile bytevector, so in this sort of case, we end up having to include a run-time make-bytevector definition in the (hoot primitives) module. This happens whereever we end up using %inline-wasm.

building to guile

Returning to our graph, we see that there is a red-colored block for Hoot modules, a teal-colored layer on top for those modules that are defined by R7RS, a few oddballs, and then (guile) and Fibers built on top. The (guile) module provides a shim that implements Guile’s own default set of bindings, allowing Guile modules to be loaded on a Hoot system. (guile) is layered on top of the low-level Hoot libraries, and out of convenience, on top of the various R7RS libraries as well, because it was easiest to remember what was where in R7RS than our ad-hoc nest of Hoot internal libraries.

Having (guile) lets Guile hackers build on Hoot. It’s still incomplete but I think eventually it will be capital-G Good. Even for a library that needed more porting like Fibers (Hoot has no threads so much of the parallel concurrent ML implementation can be simplified, and we use an event loop from the Wasm run-time instead of an epoll-based scheduler), it was still pleasant to be able to use define-module and keyword arguments and all of that.

next layers

I mentioned that this tower of terms is incomplete, and so that is one of the next work items for Hoot: complete support for Guile’s run-time library. At that point we’d probably want to merge it into Guile, but that is another topic.

But let’s leave that for another day; until then, happy hacking!

by Andy Wingo at Wednesday, May 22, 2024

Tuesday, May 21, 2024

Idiomdrottning

Modern XMPP

Modern XMPP:

Modern XMPP is an independent project launched to improve the quality of user-to-user messaging applications that use XMPP. […]

We are developing a handful of simple documents aimed at people who wish to build on top of XMPP. The recommendations are derived from healthy discussions between developers from multiple XMPP projects and other members of the XMPP community.

I’m glad XMPP is getting some love. It’s a protocol that has some advantages over Matrix and ActivityPub and even over IRC. (even though adding OMEMO to IRC might be possible, the way XMPP federates is a li’l more flexible than how IRC federates). Not that I’m ready to take all my proverbial eggs out of the ActivityPub, email, or IRC baskets just yet. Email is my doll my dear my darling♥︎ but ultimately from a user perspective, choosing a protocol is about the people on it. I find a lot of awesome stuff on email, on ActivityPub, on IRC and on XMPP so they’re all great in terms of what’s on it. Philosophically is where it doesn’t make sense to keep spending time developing a protocol when there’s a better protocol for the same thing and I don’t know yet which of these four li’l creatures is the best one. I suspect email is the best♥︎

Matrix has a long way to go before it can even go on the same table as these four other pillars of communication but maybe it’ll get there? I’m more inclined to spend my time on the other horses in that particular race but I’ve been wrong plenty of times.

We also intend to provide a comprehensive set of guidelines for UI and UX design.

Not into that. One thing I love about email, XMPP and ActivityPub is the breadth of experiences available.

Having a concrete set of guidelines will help to provide a more uniform user experience between different applications, ensuring they use the same terminology, and provide interoperable feature sets.

Interoperable feature sets through XEP curation, that’s awesome! That’s great! Uniform user experience is the part that sounds like a nightmare. Like, there’s just no way bitlbee + MS Comic Chat can have the same UI or workflow as Snikket, and that’s a good thing. But this is a resource, a tool, not a straitjacket.

by Idiomdrottning (sandra.snan@idiomdrottning.org) at Tuesday, May 21, 2024

Thursday, May 16, 2024

The Racket Blog

Racket v8.13

posted by Stephen De Gabrielle


We are pleased to announce Racket v8.13 is now available from https://download.racket-lang.org/.

As of this release:

  • The racket/treelist and racket/mutable-treelist libraries provide list-like containers that support many operations in effectively constant time, including appending and extracting sub-lists without mutating the given list. Treelists are implemented as RRB Vectors, invented by Stucki, Riompf, Ureche, and Bagwell. (see 4.16 Treelists and RRB vector: a practical general purpose immutable sequence, ICFP 2015)

  • The hash-filter-keys and hash-filter-values functions allow users to filter hashes using a predicate on either keys or values. (see 4.15 Hash Tables: hash-filter-keys, hash-filter-values)

  • The vector-extend and vector*-extend functions provide a way to pre-populate the prefix of a newly allocated vector using the elements of an existing vector. (see 4.12 Vectors: vector-extend)

  • Command-line raco setup, package update, and package installation use terminal control (when available) to show what they are working on more compactly and with a progress bar.

  • Racket v8.13 uses Unicode 15.1 for character and string operations.

  • Machine-specific cross-module optimization allows improved support for static generation of foreign-function bindings.

  • The scribble/acmart language uses v2.01, which avoids errors concerning the hyperref package in some latex installations.

Thank you

The following people contributed to this release:

Alec Mills, Ben Greenman, Bob Burger, Bogdan Popa, dr-neptune, Fred Fu, Gustavo Massaccesi, Jason Hemann, Jay McCarthy, John Clements, Jordan Johnson, Justin Dhillon, Mao Yifu, Matias Eyzaguirre, Matthew Flatt, Matthias Felleisen, Mike Sperber, olopierpa, Oscar Waddell, Pavel Panchekha, Philip McGrath, Robby Findler, Sam Phillips, Sam Tobin-Hochstadt, Siddhartha Kasivajhula, Sorawee Porncharoenwase, Stephen De Gabrielle, Tim Standen, William E. Byrd, and Wing Hei Chan.

Racket is a community developed open source project and we welcome new contributors. See racket/README.md to learn how you can be a part of this amazing project.

Feedback Welcome

Questions and discussion welcome at the Racket community Discourse or Discord

Please share

If you can - please help get the word out to users and platform specific repo packagers

Racket - the Language-Oriented Programming Language - version 8.13 is now available from https://download.racket-lang.org

See https://blog.racket-lang.org/2024/05/racket-v8-13.html for the release announcement and highlights.

by John Clements, Stephen De Gabrielle at Thursday, May 16, 2024

Andy Wingo

on hoot, on boot

I realized recently that I haven’t been writing much about the Hoot Scheme-to-WebAssembly compiler. Upon reflection, I have been too conscious of its limitations to give it verbal tribute, preferring to spend each marginal hour fixing bugs and filling in features rather than publicising progress.

In the last month or so, though, Hoot has gotten to a point that pleases me. Not to the point where I would say “accept no substitutes” by any means, but good already for some things, and worth writing about.

So let’s start today by talking about bootie. Boot, I mean! The boot, the boot, the boot of Hoot.

hoot boot: temporal tunnel

The first axis of boot is time. In the beginning, there was nary a toot, and now, through boot, there is Hoot.

The first boot of Hoot was on paper. Christine Lemmer-Webber had asked me, ages ago, what I thought Guile should do about the web. After thinking a bit, I concluded that it would be best to avoid compromises when building an in-browser Guile: if you have to pollute Guile to match what JavaScript offers, you might as well program in JavaScript. JS is cute of course, but Guile is a bit different in some interesting ways, the most important of which is control: delimited continuations, multiple values, tail calls, dynamic binding, threads, and all that. If Guile’s web bootie doesn’t pack all the funk in its trunk, probably it’s just junk.

So I wrote up a plan something to which I attributed the name tailification. In retrospect, this is simply a specific flavor of a continuation-passing-style (CPS) transmutation, late in the compiler pipeline. I’ll elocute more in a future dispatch. I did end up writing the tailification pass back then; I could have continued to target JS, but it was sufficiently annoying and I didn’t prosecute. It sat around unused for a few years, until Christine’s irresistable charisma managed to conjure some resources for Hoot.

In the meantime, the GC extension for WebAssembly shipped (woot woot!), and to boot Hoot, I filled in the missing piece: a backend for Guile’s compiler that tailified and then translated primitive operations to snippets of WebAssembly.

It was, well, hirsute, but cute and it did compute, so we continued to boot. From this root we grew a small run-time library, written in raw WebAssembly, used for slow-paths for the various primitive operations that are part of Guile’s compiler back-end. We filled out Guile primcalls, in minute commits, growing the WebAssembly runtime library and toolchain as we went.

Eventually we started constituting facilities defined in terms of those primitives, via a Scheme prelude that was prepended to all programs, within a nested lexical environment. It was never our intention though to drown the user’s programs in a sea of predefined bindings, as if the ultimate program were but a vestigial inhabitant of the lexical lake—don’t dilute the newt!, we would often say [ed: we did not]— so eventually when the prelude became unmanageable, we finally figured out how to do whole-program compilation of a set of modules.

Then followed a long month in which I would uproot the loot from the boot: take each binding from the prelude and reattribute it into an appropriate module. User code could import all the modules that suit, as long as they were known to Hoot, but no others; it was only until we added the ability for users to programmatically consitute an environment from their modules that Hoot became a language implementation of any repute.

Which brings us to the work of the last month, about which I cannot be mute. When you have existing Guile code that you want to distribute via the web, Hoot required you transmute its module definitions into the more precise R6RS syntax. Precise, meaning that R6RS modules are static, in a way that Guile modules, at least in absolute terms, are not: Guile programs can use first-class accessors on the module systems to pull out bindings. This is yet another example of what I impute as the original sin of 1990s language development, that modules are just mutable hash maps. You see it in Python, for example: because you don’t know for sure to what values global names are bound, it is easy for any discussion of what a particular piece of code means to end in dispute.

The question is, though, are the semantics of name binding in a language fixed and absolute? Once your language is booted, are its aspects definitively attributed? I think some perfection, in the sense of becoming more perfect or more like the thing you should be, is something to salute. Anyway, in Guile it would be coherent with Scheme’s lexical binding heritage to restitute some certainty as to the meanings of names, at least in a default compilation node. Lexical binding is, after all, the foundation of the Macro Writer’s Statute of Rights. Of course if you are making a build for development purposes, not to distribute, then you might prefer a build that marks all bindings as dynamic. Otherwise I think it’s reasonable to require the user to explicitly indicate which definitions are denotations, and which constitute locations.

Hoot therefore now includes an implementation of the static semantics of Guile’s define-module: it can load Guile modules directly, and as a tribute, it also has an implementation of the ambient (guile) module that constitutes the lexical soup of modules that aren’t #:pure. (I agree, it would be better if all modules were explicit about the language they are written in—their imported bindings and so on—but there is an existing corpus to accomodate; the point is moot.)

The astute reader (whom I salute!) will note that we have a full boot: Hoot is a Guile. Not an implementation to substitute the original, but more of an alternate route to the same destination. So, probably we should scoot the two implementations together, to knock their boots, so to speak, merging the offshoot Hoot into Guile itself.

But do I circumlocute: I can only plead a case of acute Hoot. Tomorrow, we elocute on a second axis of boot. Until then, happy compute!

by Andy Wingo at Thursday, May 16, 2024

Idiomdrottning

Hardening newlines

Sometimes I just want all newlines in a region to become hardened, for example when writing code or poetry.

Markdown

I have this brev snippet that I run with shell-command-on-region:

(make-sloppy-tree-accessor scadr)

(define (poetry-md)
  (pair-fold
    (fn (if (any empty? (list (car x) (scadr x)))
          (print (car x))
          (print (car x) "  ")))
    (void)
    ((over (strse x " +$" "")) (read-lines))))

use-hard-newlines mode

Here’s a function to harden all newlines in the region:

(defun harden-region (start end)
  (interactive "r")
  (when use-hard-newlines
    (save-excursion
      (goto-char start)
      (while (< (point) (min (point-max) end))
        (end-of-line)
        (forward-char)
        (set-hard-newline-properties
         (- (point) 1) (point))))))

by Idiomdrottning (sandra.snan@idiomdrottning.org) at Thursday, May 16, 2024

Wednesday, May 15, 2024

spritely.institute

Make a game with Hoot for the Lisp Game Jam!

Spring Lisp Game Jam

Twice per year I co-host the Lisp Game Jam: a 10-day game jam where participants create small games in their favorite Lisp dialect. Unlike classic game jams like Ludum Dare, which have an intensive 2-3 day long duration, the Lisp Game Jam is much more casual and spans two whole weekends to give busy people (most of us!) more of a chance to complete a game. There are no prizes, though there is a judging period where we play each other's games and give feedback. Really, the joy of getting creative with Lisp is its own reward. Game jams are the perfect excuse to prototype a gameplay idea and get some practice managing the scope of a project to meet a due date.

The Spring edition of the Lisp Game Jam starts this Friday, May 17th and is on track to be the biggest one yet with over 100 registered participants so far! Will you be among them? On behalf of the Spritely Institute, I would like to encourage you to join this spring's jam and try out Hoot, our Scheme to WebAssembly compiler!

We're so excited for you to join that we've put together a game jam template repository that has everything you need to make a 2D game using Hoot and HTML5 canvas. It even features a mini-Breakout clone as an example of how to put all the pieces together into a playable game!

I can't overstate how web browser support is a huge benefit in a game jam. An HTML5 build embedded directly on the itch.io page for your game makes it relatively frictionless for players to try it out. HTML5 games consistently get more players and ratings on average than native games.

It is much harder to ship native releases that will work reliably on many different computers. I've spent many hours debugging such issues in the past. Native releases also typically need one build per supported operating system. That's a lot of time and energy that could be spent building the actual game instead! When it comes to game jams, people play entries for minutes, not hours. If your game is hard to launch then they will move on to another one that is easier to try. I know I'm on the development team, but ease of distribution is probably the biggest motivating factor for making my jam games with Hoot.

Last year, during the Autumn jam, I used a much more primitive version of Hoot to make a space shooter called Strigoform, and David Wilson (of System Crafters fame) made a Solitaire clone called Cybersol. You can read more about our experiences using Hoot in last year's jam here, but trust me when I say that since then, it's become a lot easier to get started!

Strigoform gameplay animation

We love making small games here at Spritely not only because they make for eye-catching demos, but also for their tendency to stress language/library implementations, routinely exposing bugs and performance issues. Game jams are a great opportunity to "eat your own dog food" and put a technology stack to the test under time and resource constraints in a low stakes, fun way. Spritely staff, myself included, will be participating in the jam, but what I'm really looking forward to is seeing what the community creates and taking note of all the ways that Hoot can be improved moving forward!

For help with Hoot or the game template during the jam, you can post questions in this itch.io forum thread.

To talk about all things Spritely, during the jam and beyond, join our community forum or the #spritely channel on the Libera.Chat IRC network.

For some advice on making games, I recommend reading "How to Make Good Small Games" by Far Away Times. I think I will re-read it before the jam starts.

And finally, for our dedicated and intrepid game jam readers, tessa, our developer advocate, has created a mobile wallpaper to help you celebrate the jam all day every day, wherever you may be:

Painting of Spritely characters hanging out and playing

Happy hooting, jammers! 🦉

by Dave Thompson (contact@spritely.institute) at Wednesday, May 15, 2024

Monday, May 13, 2024

Idiomdrottning

Tooticki

I like toot as a good complement to other ActivityPub clients. It’s friendly for scripting.

I can’t really use the TUI version (maybe there’s some issue with my shell) but I use the CLI interface all the time.

One way I use it is in Emacs shell-mode. I hacked in some Lisp so I can favourite, reblog and even reply right from there; I move the cursor to the body of the post and then hit the key bound to one of these interactive functions. (If I wanna do even more fancy stuff I still can, just typing it in the command line as usual, or for some things I still schlep out another client.)

Here is the elisp:

(defun tooticki-find-id ()
  (interactive)
  (save-excursion
    (beginning-of-line)
    (re-search-forward "^ID ")
    (let ((st (point)))
      (forward-word)
      (buffer-substring-no-properties st (point)))))

(defun tooticki-reply-markdown ()
  (interactive)
  (let* ((id (tooticki-find-id))
	 (topic
	  (string-trim-right
	   (shell-command-to-string
	    (s-concat "toot status --json " id "|jq  .spoiler_text"))))
	 (targs (if (< (length topic) 3) "" (s-concat " -p " topic))))
    (async-shell-command
     (s-concat "toot post -r " id targs))))

(defun tooticki-skele (str)
  (shell-command (s-concat "toot " str " " (tooticki-find-id))))

(defun tooticki-fave ()
  (interactive)
  (tooticki-skele "favourite"))

(defun tooticki-boost ()
  (interactive)
  (tooticki-skele "reblog"))

Except that where I wrote “toot post -r” here, in the version I’ve got bound it’s instead a wrapper around written in zshbrev that tries to guess language (currently doing a really bad job at it since for some reason many emojis trigger false positives making it guess that some English language posts are Swedish) and that offers to post links to linkhut.

One limitation is that it doesn’t pre-populate the reply with @usernames; that’s a work in progress. I currently just manually paste them in. Today I added the feature that it preserves the spoiler text a.k.a. subject line, and a similar route, parsing status --json, could be the ticket for getting the user names in there. One SMOP over the line, sweet Jesus.

Update

Yeah, finding the usernames was possible:

toot-get-usernames () {
	toot status --json $1 | jq -r '..|select(.acct?).acct' |
		mawk '!s[$0]++' | sed 's;^;@;'
}

by Idiomdrottning (sandra.snan@idiomdrottning.org) at Monday, May 13, 2024

Saturday, May 11, 2024

Idiomdrottning

RFC on e-ink readers

I used httrack and Calibre’s ebook-convert for what I thought was the best results.

First, go to rfc-editor.org and select the HTML version of your RFC, for example RFC 9421.

Then, I downloaded it with:

httrack -g https://www.rfc-editor.org/rfc/rfc9421.html --levels 1

Then I converted that like this:

ebook-convert rfc9421.html "RFC 9421: HTTP Message Signatures.epub" --max-levels 1

Background

Since I was heading down this particular yak-shaving rabbit-hole this morning, with a couple of false starts and dead ends, I thought I’d note down for posterity what I landed on.

I tried out three or four other solutions, like RFC2Kindle.py and rfc2epub.php, but they didn’t work very well. I didn’t realize that rfc-editor has HTML versions now.

Truth is, I already had an iPad shortcut set up to grab any URL, SSH into my Debian machine and run httrack and ebook-convert, and calibre-smtp to send it. Sometimes the Kindle app gives better results, usually this httrack method is the best, often neither works since in the grim darkness of the world wide web there is only war.

It just took me a while to think of using that approach for this as opposed to solutions trying to parse the traditional RFC format, and the results ended up good, at least for 9421.

Here’s the shortcut I’ve been using, in zshbrev:

(define (kindlepage url #!key (levels 0))
  (if (strse?* url "youtube")
    (kindletube url)
    (with-temp-dir
      (eif levels (httrack '-g url '-D (conc '-r levels)) (httrack '-g url))
      (with (conc (broodal-title (car (glob "*.html"))) '.epub)
            (ebook-convert (car (glob "*.html")) it '--max-levels levels)
            (kindlesend it)))))

(define (broodal-title (= ->string hfile))
  (strse (with-input-from-file hfile read-string)
         (: "<title>" (* space) (=> keep (* nonl)) (* space) "</title>")
         (return keep)))

(define-ir-syntax*
  (with-temp-dir . body)
  `(let ((cur (current-directory)) (,(inject 'it) (mktemp '-d)))
     (change-directory ,(inject 'it))
     ,@body
     (change-directory cur)
     (delete-directory ,(inject 'it) #t)))

(And then kindlesend is just a wrapper for calibre-smtp that prefills the args with the right email addresses.)

I’ve had this set up for years but didn’t think of using it for RFCs since I was like “I’m sure someone else musta figured out a better solution for reading RFCs on e-ink many years ago” but nope.

by Idiomdrottning (sandra.snan@idiomdrottning.org) at Saturday, May 11, 2024

Thursday, May 9, 2024

spritely.institute

Distributed System Daemons: More Than a Twinkle in Goblins' Eye

Shepherd goblin

The great promise of Spritely Goblins is to make networking easier and safer. For the past few years, we've focused on developing a toolkit to facilitate this goal. Today, we're proud to announce an incredibly exciting project that will put this toolkit to the test in the largest real-world deployment of Spritely technology so far: a port of the GNU Shepherd system layer to Guile Goblins — the first step in making Guix the object-capability operating system!

The tale of the Shepherd

What is the Shepherd and what does it do?

The Shepherd is Guix's system layer. That's just jargon saying it both starts the system and manages daemons (background programs needed by the system itself) running on Guix system installations. It's also the first system layer ever, originally developed for the GNU Hurd as dmd, the daemon-managing daemon. You may have heard of systemd, which was inspired by Apple's launchd. These were preceded, and possibly inspired, by the Shepherd!

Over the next decade and change, the Shepherd languished with relatively little development, until it was picked up for Guix by Ludovic Courtès, the founder of the Guix project and the Shepherd's lead developer. Now, the Shepherd is rapidly catching up to its descendants in both functionality and reliability. This new initiative will empower the Shepherd with unique new abilities which no other system layers possess, returning this venerable daemon to the forefront of technological development at last.

Putting Goblins to the test

An ambitious demonstration of Goblins OCaps

At this point you might be thinking, "Okay Juli, that's cool and all, but what are you actually doing?" Great question. At the most minimal level, I will port the Shepherd to Goblins. Because the Shepherd is written in Guile and already uses the actor model internally — as does Goblins (we'll come back to this) — the port is a natural extension of the Shepherd's existing architecture.

Integrating Goblins and the Shepherd has been thrown around as an idea for a while within the Guix community for the last couple of years, and Ludovic has been doing work to convert Shepherd to an actor model implementation over time, making this proposal achieve feasibility. But just changing the internal details of the program isn't particularly interesting, and Goblins promises to let us do magic. So why not do some magic?

In addition to a basic port, this project will leverage Goblins to expose the system daemon to users without compromising security. Currently, user services require a separate Shepherd daemon to avoid giving unprivileged users access to system services.

Even more excitingly, we plan to implement a Unix domain socket netlayer in Goblins and use it to facilitate distributed computing via the Shepherd by enabling Shepherd daemons on different machines to talk to each other and control each others' services. This new functionality will be further edified by allowing services to pass service dependencies as capabilities, including capabilities on remote services. Or to put it simply, users will be able to connect multiple distinct machines together at the system layer to act as one machine for some computing tasks. As an example, in combination with Guix's deploy, this work will allow Guix to replace solutions like Kubernetes!

These new features will have an immediate positive impact on a large number of users. As mentioned, the Shepherd is the system layer for Guix, which has a substantial userbase; and there have been examples of running the Shepherd in user space on other distributions. In other words, the Shepherd is already widely deployed in many real-world contexts, including clustered computing contexts. With the integration of Goblins and the Unix domain socket netlayer into Shepherd, this project will constitute the single largest real-world deployment of Spritely code to date. We will be putting our tools into users' hands at an unprecedented scale, and we cannot wait to see them put to use!

No goblin is an island

Who else is joining us on this journey?

This work would not be possible without the generous support of the NLnet Foundation. In fact, this is not the only NLnet grant related to Spritely's work! We previously shared our work on bootstrapping OCapN, funded by the Foundation. Today, NLnet is funding two additional grants related to Spritely's work: Whippet and Pre-Scheme!

Whippet is a state-of-the-art, embeddable, plain-C garbage collector intended for inclusion in Guile being developed by Andy Wingo, one of Guile's maintainers and Hoot's lead engineer. It aims to improve the performance characteristics of Guile's garbage collection process, increasing the performance of both Guile itself and all programs written in it, such as Guile Goblins. For those interested in the gory details, Andy explains more in his 2023 FOSDEM talk. We can't wait for this work to land upstream, and congratulate Andy on his grant!

Just as Spritely is bringing past ideas to life in new ways, Andrew Whatson, a well-respected Scheme hacker and long-time member of the Spritely community, is working to port the venerable Pre-Scheme to Guile. This new Pre-Scheme will serve as a modern, statically-typed systems programming language in the Guile — and thus Spritely — ecosystem, a continuation of an idea our own Christine Lemmer-Webber proposed some time ago as "Guile Steele"!

Pre-Scheme and Pre-Spritely

Pre-Scheme is a statically-typed, compiled (Scheme-to-C) systems programming language related to Scheme which was used to implement the VM and garbage collector for Scheme48. Modernizing this proven systems language will strengthen the Scheme world with new tools to build amazing programs.

Additionally, Scheme48 is the language described in Jonathan Rees's "A Security Kernel Based on the Lambda-Calculus". The dissertation is based on the observation that object-capability security is a restricted form of the actor model. Scheme, the language family of which Guile is a member and from which Racket is descended, was initially formulated to implement and explore the actor model. This paper is why Spritely uses Scheme — Scheme already does much of the work needed for object-capability security.

All of these connections mean that we are also very excited for Andrew's work. If you're interested in the more technical aspects, he also gave a talk at FOSDEM 2023. Congratulations on your NLnet grant, Andrew!

Onward and outward

What all this means for the future

Alongside the Goblins 0.13 release and ongoing Hoot development development, these NLnet projects show that the future is bright for Spritely and our work.

The Shepherd port demonstrates that Spritely tech is ready to improve end-user computing security while enabling new kinds of connections. We delight that Goblins is mature enough to serve as the foundation for such interesting work. As we look toward tomorrow, we're excited to see what new paradigm shifts our work on the Shepherd, Andy's Whippet work, and Andrew's Pre-Scheme work will empower!

Be sure to subscribe to our RSS feed or check back here for progress on the Shepherd project and other updates in the Spritely space, and join us on our forum to discuss these exciting developments with the team and our wider community!

by Juliana Sims (contact@spritely.institute) at Thursday, May 9, 2024

Tuesday, May 7, 2024

GNU Guix

Authenticate your Git checkouts!

You clone a Git repository, then pull from it. How can you tell its contents are “authentic”—i.e., coming from the “genuine” project you think you’re pulling from, written by the fine human beings you’ve been working with? With commit signatures and “verified” badges ✅ flourishing, you’d think this has long been solved—but nope!

Four years after Guix deployed its own tool to allow users to authenticate updates fetched with guix pull (which uses Git under the hood), the situation hasn’t changed all that much: the vast majority of developers using Git simply do not authenticate the code they pull. That’s pretty bad. It’s the modern-day equivalent of sharing unsigned tarballs and packages like we’d blissfully do in the past century.

The authentication mechanism Guix uses for channels is available to any Git user through the guix git authenticate command. This post is a guide for Git users who are not necessarily Guix users but are interested in using this command for their own repositories. Before looking into the command-line interface and how we improved it to make it more convenient, let’s dispel any misunderstandings or misconceptions.

Why you should care

When you run git pull, you’re fetching a bunch of commits from a server. If it’s over HTTPS, you’re authenticating the server itself, which is nice, but that does not tell you who the code actually comes from—the server might be compromised and an attacker pushed code to the repository. Not helpful. At all.

But hey, maybe you think you’re good because everyone on your project is signing commits and tags, and because you’re disciplined, you routinely run git log --show-signature and check those “Good signature” GPG messages. Maybe you even have those fancy “✅ verified” badges as found on GitLab and on GitHub.

Signing commits is part of the solution, but it’s not enough to authenticate a set of commits that you pull; all it shows is that, well, those commits are signed. Badges aren’t much better: the presence of a “verified” badge only shows that the commit is signed by the OpenPGP key currently registered for the corresponding GitLab/GitHub account. It’s another source of lock-in and makes the hosting platform a trusted third-party. Worse, there’s no notion of authorization (which keys are authorized), let alone tracking of the history of authorization changes (which keys were authorized at the time a given commit was made). Not helpful either.

Being able to ensure that when you run git pull, you’re getting code that genuinely comes from authorized developers of the project is basic security hygiene. Obviously it cannot protect against efforts to infiltrate a project to eventually get commit access and insert malicious code—the kind of multi-year plot that led to the xz backdoor—but if you don’t even protect against unauthorized commits, then all bets are off.

Authentication is something we naturally expect from apt update, pip, guix pull, and similar tools; why not treat git pull to the same standard?

Initial setup

The guix git authenticate command authenticates Git checkouts, unsurprisingly. It’s currently part of Guix because that’s where it was brought to life, but it can be used on any Git repository. This section focuses on how to use it; you can learn about the motivation, its design, and its implementation in the 2020 blog post, in the 2022 peer-reviewed academic paper entitled Building a Secure Software Supply Chain with GNU Guix, or in this 20mn presentation.

To support authentication of your repository with guix git authenticate, you need to follow these steps:

  1. Enable commit signing on your repo: git config commit.gpgSign true. (Git now supports other signing methods but here we need OpenPGP signatures.)

  2. Create a keyring branch containing all the OpenPGP keys of all the committers, along these lines:

    git checkout --orphan keyring
    git reset --hard
    gpg --export alice@example.org > alice.key
    gpg --export bob@example.org > bob.key
    …
    git add *.key
    git commit -m "Add committer keys."

    All the files must end in .key. You must never remove keys from that branch: keys of users who left the project are necessary to authenticate past commits.

  3. Back to the main branch, add a .guix-authorizations file, listing the OpenPGP keys of authorized committers—we’ll get back to its format below.

  4. Commit! This becomes the introductory commit from which authentication can proceed. The introduction of your repository is the ID of this commit and the OpenPGP fingerprint of the key used to sign it.

That’s it. From now on, anyone who clones the repository can authenticate it. The first time, run:

guix git authenticate COMMIT SIGNER

… where COMMIT is the commit ID of the introductory commit, and SIGNER is the OpenPGP fingerprint of the key used to sign that commit (make sure to enclose it in double quotes if there are spaces!). As a repo maintainer, you must advertise this introductory commit ID and fingerprint on a web page or in a README file so others know what to pass to guix git authenticate.

The commit and signer are now recorded on the first run in .git/config; next time, you can run it without any arguments:

guix git authenticate

The other new feature is that the first time you run it, the command installs pre-push and pre-merge hooks (unless preexisting hooks are found) such that your repository is automatically authenticated from there on every time you run git pull or git push.

guix git authenticate exits with a non-zero code and an error message when it stumbles upon a commit that lacks a signature, that is signed by a key not in the keyring branch, or that is signed by a key not listed in .guix-authorizations.

Maintaining the list of authorized committers

The .guix-authorizations file in the repository is central: it lists the OpenPGP fingerprints of authorized committers. Any commit that is not signed by a key listed in the .guix-authorizations file of its parent commit(s) is considered inauthentic—and an error is reported. The format of .guix-authorizations is based on S-expressions and looks like this:

;; Example ‘.guix-authorizations’ file.

(authorizations
 (version 0)               ;current file format version

 (("AD17 A21E F8AE D8F1 CC02  DBD9 F8AE D8F1 765C 61E3"
   (name "alice"))
  ("2A39 3FFF 68F4 EF7A 3D29  12AF 68F4 EF7A 22FB B2D5"
   (name "bob"))
  ("CABB A931 C0FF EEC6 900D  0CFB 090B 1199 3D9A EBB5"
   (name "charlie"))))

The name bits are hints and do not have any effect; what matters is the fingerprints that are listed. You can obtain them with GnuPG by running commands like:

gpg --fingerprint charlie@example.org

At any time you can add or remove keys from .guix-authorizations and commit the changes; those changes take effect for child commits. For example, if we add Billie’s fingerprint to the file in commit A, then Billie becomes an authorized committer in descendants of commit A (we must make sure to add Billie’s key as a file in the keyring branch, too, as we saw above); Billie is still unauthorized in branches that lack A. If we remove Charlie’s key from the file in commit B, then Charlie is no longer an authorized committer, except in branches that start before B. This should feel rather natural.

That’s pretty much all you need to know to get started! Check the manual for more info.

All the information needed to authenticate the repository is contained in the repository itself—it does not depend on a forge or key server. That’s a good property to allow anyone to authenticate it, to ensure determinism and transparency, and to avoid lock-in.

Interested? You can help!

guix git authenticate is a great tool that you can start using today so you and fellow co-workers can be sure you’re getting the right code! It solves an important problem that, to my knowledge, hasn’t really been addressed by any other tool.

Maybe you’re interested but don’t feel like installing Guix “just” for this tool. Maybe you’re not into Scheme and Lisp and would rather use a tool written in your favorite language. Or maybe you think—and rightfully so—that such a tool ought to be part of Git proper.

That’s OK, we can talk! We’re open to discussing with folks who’d like to come up with alternative implementations—check out the articles mentioned above if you’d like to take that route. And we’re open to contributing to a standardization effort. Let’s get in touch!

Acknowledgments

Thanks to Florian Pelz and Simon Tournier for their insightful comments on an earlier draft of this post.

by Ludovic Courtès at Tuesday, May 7, 2024

Monday, May 6, 2024

Scheme Requests for Implementation

SRFI 251: Mixing groups of definitions with expressions within bodies

SRFI 251 is now in final status.

Scheme has traditionally required procedure bodies and the bodies of derived constructs such as let to contain definitions followed by commands/expressions. This SRFI proposes to allow mixing commands and groups of definitions in such bodies, so that each command/expression is in the scope of all local definition groups preceding it, but not in scope of the local definition groups following it. This approach is backwards compatible with R7RS and upholds the intuitive rule that to find the definition of a lexical variable, one has to look up the source code tree.

by Sergei Egorov at Monday, May 6, 2024

Tuesday, April 30, 2024

spritely.institute

Guile Hoot v0.4.1 released!

Hoot version 0.4.1

We are excited to announce the release of Guile Hoot v0.4.1! Hoot is a Scheme to WebAssembly compiler backend for Guile, as well as a general purpose WebAssembly toolchain. In other words, Scheme in the browser!

This is a maintenance release that primarily fixes many bugs present in Hoot 0.4.0, but there are also a few developer experience improvements. Read on for the full release notes.

New features

  • Added initial support for Guile's define-module syntax with the caveat that modules must be marked as #:pure. Impure modules implicitly import (guile) which Hoot does not yet provide.

  • Added hashq procedure to (hoot hashtables).

  • Added --run flag to guild compile-wasm for quickly testing programs in the terminal.

  • Added --mode flag to guild compile-wasm to control how the Hoot ABI is exported/imported in the resulting module.

  • Added --dump-tree-il flag to guild compile-wasm for debugging.

  • Added (hoot web-server) module which provides a development static file web server. Updated tutorial and project template to use it.

  • Non-continuable exceptions now include a message explaining what such an error means.

  • Improved define-foreign macro to validate that the module and name fields are string literals.

  • Removed requirement to have an expression at the end of a program.

Bug fixes

  • Fixed prompt unwinding in tailify that caused miscompilations.

  • Fixed known-return-arity calls in tailify that caused miscompilations.

  • Fixed kargs-splitting in tailify that caused compilation errors.

  • Fixed (ref extern) parameters not working in define-foreign.

  • Fixed records and parameters being unable to be created at expansion-time.

  • Fixed several small issues in port implementation.

  • Fixed call-with-values to allow arbitrary consumer expression, not just a lambda.

  • Fixed weak map imports for Hoot VM.

  • Fixed syntax definitions in programs not being evaluated.

  • Fixed linker to allow exports to pull in tag definitions.

  • Worked around partial evaluator giving up inlining dynamic-wind.

Browser compatibility

  • Compatible with Firefox 121 or later.

  • Compatible with Google Chrome 119 or later.

  • Safari/WebKit is unsupported, unfortunately, as Wasm GC and tail calls are still not available.

Get Hoot 0.4.1!

Hoot is already available in GNU Guix:

$ guix pull
$ guix install guile-next guile-hoot

(Hoot currently requires a bleeding-edge version of Guile, hence guile-next above.)

Otherwise, Hoot can be built from source via our release tarball. See the Hoot homepage for a download link and GPG signature.

Documentation for Hoot 0.4.1, including build instructions, can be found here.

Make a game with Hoot!

The annual Spring Lisp Game Jam is coming up! Starting on Friday, May 17th, the Lisp Game Jam is a 10-day long event in which participants make small games using their favorite Lisp dialect. If this sounds like a fun time to you, we encourage you to register for the jam and try making a game with Hoot!

To help you get started right away, we've put together a game jam template repository that has everything you need to make a 2D game using Hoot and HTML5 canvas.

Get in touch!

For bug reports, pull requests, or just to follow along with development, check out the Hoot project on GitLab.

If you build something cool with Hoot, let us know on our community forum!

The code in this release was brought to you by Andy Wingo and David Thompson. Thanks and apologies to Juliana Sims, who found so many bugs. The lovely Hoot art is by tessa. Special thanks to the MetaMask folks for funding this work!

Until next time, happy hooting! 🦉

by Dave Thompson (contact@spritely.institute) at Tuesday, April 30, 2024

Thursday, April 25, 2024

Scheme Requests for Implementation

SRFI 252: Property Testing

SRFI 252 is now in final status.

This defines an extension of the SRFI 64 test suite API to support property testing. It uses SRFI 158 generators to generate test inputs, which allows for the creation of custom input generators. It uses SRFI 194 as the source of random data, so that the generation of random test inputs can be made deterministic. For convenience, it also provides procedures to create test input generators for the types specified in R7RS-small. The interface to run property tests is similar to that of SRFI 64, and a property-testing-specific test runner is specified in order to display the results of the propertized tests.

by Antero Mejr at Thursday, April 25, 2024

spritely.institute

Organizational changes at the Spritely Institute

Today we’re announcing some organizational changes at the Spritely Networked Communities Institute. My co-founder and colleague Randy Farmer is transitioning out of the Executive Director role. I (Christine Lemmer-Webber) will be stepping into the role of Executive Director, and David Thompson, formerly our Core Infrastructure Architect, will be stepping up as Chief Technology Officer.

Personally, I am very grateful for the two years that Randy served as Executive Director of this organization. Randy stepped up to fill a necessary role, making major positive impacts on the Institute. I consider myself incredibly lucky to have co-founded the Spritely Institute with someone who always inspired, understood, and fostered Spritely’s vision. Indeed, much of Spritely’s long-term vision came directly out of Randy’s work at Electric Communities and his entire career shaping online community structures. You can read Randy’s own statement about his departure. Please join us in applauding Randy for serving the organization and working so hard for the last couple of years. Randy will always be co-founder of the Spritely Institute, and we are happy to say that Randy will continue to work with the Institute as a consultant.

Meanwhile, the Spritely Institute continues in its work and goals as we have always had them: building the future of decentralized networks to empower individuals and communities on the internet. I am excited to move forward: we have an incredible team with David Thompson as our new CTO, Jessica Tallon as our Founding Technologist, Juliana Sims who works with us as Integration Engineer, Andy Wingo working via Igalia with us as Hoot's Compiler Specialist, tessa as our Developer Relations Manager, and Briana Jacobs our Human Resources Administrator. And of course, Spritely is not just the employees of the Spritely Institute: it is also all the users and contributors to Goblins, Hoot, and the OCapN pre-standardization group. If you are excited to join our community, we have a forum at community.spritely.institute and would be happy to speak to you there!

by Christine Lemmer-Webber (contact@spritely.institute) at Thursday, April 25, 2024

Tuesday, April 23, 2024

spritely.institute

Spritely Goblins v0.13.0: Object Persistence and Easier IO!

Goblins version 0.13.0

We are thrilled to announce the release of our distributed programming environment Spritely Goblins version 0.13.0 for Goblins on Guile! This release has two major new features: a powerful new object persistence mechanism (to save your running programs to disk and wake them up later!), as well as a new abstraction which makes writing IO code much easier!

(Note that there are no major protocol updates to OCapN functionality in this release, and there is not an accompanying release of Racket Goblins this time.)

Persistence comes to Goblins

Imagine you're building out a fantasy dungeon crawler, modifiable by your users at runtime. Your users are building rooms and putting NPCs here and there and your world is evolving. But oh no -- it's time to restart your computer! How can you keep track of all these objects and bring them back later?

Now Spritely Goblins has the answer, incorporating a powerful new persistence mechanism (codenamed "Aurie") which empowers you to save relevant parts of a running Goblins program and wake them up later! But that's not all -- you can also use Goblins' persistence system to upgrade your saved objects as they evolve, and it even helps making live hacking even more powerful since you can change objects and reload them with upgraded behavior live!

Here's an example of our persistence system in action:

Terminal Phase game with user saving and restoring

Above we see a user playing a space shooter game named Terminal Phase written on top of Spritely Goblins. In it, you can observe the player saving, quitting, and reopening the game. Although internally, reopening the game initializes a distinct new process, which previously meant a new game state starting at the beginning of the game, with the release of Goblins 0.13.0, we were able to save all the state-setting Goblins objects to disk in the first session and set them as the state of the new one. In other words, the player can resume their session right where they left off!

Unlike persistence systems common to some other Lisps and Smalltalk, Goblins' persistence system does not use "orthogonal persistence". Orthogonal persistence relies on the underlying programming language to implement persistence support and saves a "running snapshot" of the program as the programming language itself sees it. While a powerful approach, orthogonal persistence suffers from generally saving much more than the programmer may need and tends to result in difficult to upgrade systems.

Goblins' persistence system takes a different approach by using a technique which qualifies as "manual persistence", but in reality Goblins has also provided tooling which makes most actors "just work" in terms of persistence, providing the best of both of the worlds of manual and orthogonal persistence.

Goblins' persistence system was also designed so that only the information which needs to be saved can be what is saved, assuming that the author of the relevant object's code can determine what to store and how to restore things. For instance, in the Terminal Phase example above, the entire level need not be stored to memory, only the parts which are on screen and the filename of the current level and progress therein in terms of how far the player has gotten need be stored. Every persisted object is also tagged with a version (defaulting to 0 if none is provided) permitting the user to perform an uprade at restoration time if appropriate.

For most use cases, adopting Goblins' persistence system will be straightforward. Here is a classic and simple "cell" actor which may be familiar to many Goblins users:

(define (^cell bcom #:optional val)
  (case-lambda
    ;; Called with no arguments; return the current value
    [() val]
    ;; Called with one argument, we become a version of ourselves
    ;; with this new value
    [(new-val)
     (bcom (^cell bcom new-val))]))

The only thing that is needed to convert this cell to a persistence-compatible cell is to switch the top-level define to the new define-actor syntax:

(define-actor (^cell bcom #:optional val)
  (case-lambda
    ;; Called with no arguments; return the current value
    [() val]
    ;; Called with one argument, we become a version of ourselves
    ;; with this new value
    [(new-val)
     (bcom (^cell bcom new-val))]))

That's it! define-actor is a fancy new macro which takes the boilerplate out of writing most persisted objects. The above roughly expands to:

(define (^cell bcom #:optional val)
  (define cell-behavior
    (case-lambda
      ;; Called with no arguments; return the current value
      [() val]
      ;; Called with one argument, we become a version of ourselves
      ;; with this new value
      [(new-val)
       (bcom (^cell bcom new-val))]))
  ;; The self-portrait gives the information necessary to restore this
  ;; object later, by default by applying to the actor's constructor.
  (define (self-portrait)
    (list val))
  (portraitize cell-behavior self-portrait))

However, users in general will rarely need to use portraitize... for most users, define-actor will be enough!

Even though objects self-describe, Goblins' serialization system follows the same capability security properties as Goblins itself. Objects cannot describe themselves with more power than they actually have access to.

Our persistence system also takes advantage of Goblins' transactional nature; we can serialize only the delta of objects which have changed between messages handled in the event loop by introspecting Goblins' transaction data, resulting in faster snapshots and less data saved to disk!

This is the first release of Goblins containing our persistence system; as such, we expect that usage will have some rough edges. In particular, persistence really only works for references to objects contained within one vat. We hope to remove this restriction in a future version of Goblins.

New, simpler IO

One of the most frequent requests for Goblins has been to introduce an IO system which "felt like" using Goblins. Previously, users had to understand the details of programming in Fibers, the event loop system on which Goblins programs run by default.

No longer! We are happy to announce a new addition to Goblins' actor-lib standard library, the ^io actor from (goblins actor-lib io)! This object wraps a protected IO resource and ensures that only one IO operation is handled at a time without blocking the vat (event loop) in which the IO actor is spawned.

Getting the release

This release also includes a number of other bugfixes and minor features. See the NEWS file for more information!

You can also get the latest Guile Goblins release using Guix:

$ guix pull
$ guix install guile-goblins

See also the README for more ways to get guile-goblins for your operating system or distribution!

Bonus history!

Goblins' persistence system was a major effort with a long history behind it. The codename "Aurie" (and the flame-and-crystal-character associated with it, named after the Aurora Borealis) came from its predecessor, an independent library for the Racket verison of Goblins.

The original conceptual designs from original-Aurie remain intact within Spritely Goblins, but the implementation and interface have been significantly refactored.

The original version of Aurie was cumbersome to use; it is thanks to Jessica Tallon, Spritely's enchantress of the goblin realm, who took Christine Lemmer-Webber's original proof-of-concept and transformed it into something much easier to use. We are all excited that Spritely Goblins now includes a powerful persistence system.

For more on the ideas behind Aurie, read Safe Serialization Under Mutual Suspicion by Mark S. Miller, by which Aurie is designed. Aurie also borrows heavily from an observation by Jonathan A. Rees made to Mark Miller, that serialization is uneval / unapply... we take exactly this technique in our implementation, as Aurie can be thought of as a metacircular evaluator running in reverse!

Onwards we go!

Of course, there's plenty more to do, and we're already working and looking ahead to the next release! We are planning for v0.14 of Goblins to focus on improvements to networking tooling in particular as well as various quality of life improvements.

If you're making something with Goblins, or want to contribute to Goblins itself, join our community: community.spritely.institute for our forum and #spritely on irc.libera.chat for real-time chat! Hope to see you there!

by Christine Lemmer-Webber (contact@spritely.institute) at Tuesday, April 23, 2024