# Planet Scheme

## Monday, January 17, 2022

### Jérémy Korwin-Zmijowski

#### 2022 Sem.2 - Bibliographie

En lisant le livre « Clean Architecture » de Robert C. Martin, il m'arrive de revoir l'architecture de mon application « Your Next Meal » (alias YNM), pour la science !

Guile, ou plus largement Scheme, n'étant pas un langage très utilisé pour illustrer les concepts de « Clean Architecture » ou « Clean Code », je m'inspire de ce qui est fait dans d'autres langages.

J'ai aussi lu quelques ressources sur le langage Ruby, dont j'apprécie la philosophie de son créateur (même si je ne me suis jamais vraiment intéressé au langage en lui-même). À noter que je n'a fait que survoler certaines de ces ressources à la recherche d'une information précise sans les lire dans leur totalité.

## Design techniques

Merci beaucoup d'avoir lu cet article !

N'hésite pas à me contacter pour me donner ton avis, proposer une idée d'amélioration, ou poser une question !

Abonnes-toi pour ne pas manquer les prochains :

1. articles via Mastodon @jeko@write.as et RSS
2. screencasts via Peertube jeko@video.tedomum.net et RSS

Et encore plus important, partages ce blog et dis à tes amis que c'est le meilleur blog de l'histoire du logiciel libre ! Sans dec'

## Wednesday, January 12, 2022

### GNU Guix

#### Announcing the second online Guix Days

The Guix hackers are very happy to announce the second online Guix Days Conference on 19 & 20 February 2022. This conference is open to everyone and will be held entirely online. Want to speak? Submit your proposal!

Important dates:

1. February 8: Deadline for talks proposal.
3. February 14: Release of the schedule.
4. February 19: Conference day!
5. February 20: Conference day!

The agenda of these two days is:

• pre-recorded talks with live question and answer sessions
• birds of a feather (BoF) sessions
• lightning round talks, if possible
• retrospective and future of Guix
• hack together

Talks will be released before the conference day, watch them as soon as possible! And: no registration fee.

##### Until February 8: talk proposals

Propose your talks by sending them to guix-days@gnu.org. Feel free to drop in #guix on irc.libera.chat to discuss what you would like to talk about before submitting. :)

You can choose one of the following formats:

• Standard talk. 15-45 minutes pre-recorded presentation and a 5 minutes lightning talk. The 5-minute presentation will be live, to refresh our minds, followed by a 30 minutes live Q&A.
• BoF (birds of a feather, for a session with a small group who wants to talk about a specific topic) with no presentation. You may prepare something live to spark conversations.
• Lightning talk with a 5 minutes live presentation

In addition to the format you would like to choose, please describe your session with 10 lines or more (for lightning talks, at least 1 sentence).

Once you have sent your proposal, you will be notified in the following days whether your talk will be part of the Guix Days. Submit earlier to get more time to prepare your session!

Even for live presentation, please prepare a back-up pre-recorded talk, so we can play it if you cannot attend or have a technical problem during the Guix days. The deadline for short presentations (5 minutes) is February 16.

We welcome all kinds of topics from the community, especially your own experience with Guix, your cool projects that involve Guix in some way, infrastructure around guix (translations, continuous integration, ...), and any subject you feel should be discussed during the conference.

We particularly encourage people who consider themselves part of a group underrepresented in Guix and the broader free software movement to submit a talk. Do not hesitate to get in touch with the organizers at guix-days@gnu.org if unsure or if you would like guidance on how to prepare your talk.

Please make sure your talk is accessible to a non-expert audience, for instance by explaining the general context before diving into technical descriptions, and by avoiding acronyms and jargon.

We accept talks in languages other than English provided English subtitles are included.

Have a look at the topics from the last conference for ideas, but don't hesitate to innovate in your proposals!

##### February 8 (or before) – 12: prepare your session

The aim of the pre-recorded talks is to demonstrate new features, what you are hacking on, introduce the subject for easing the live question and answer sessions or BoFs. These pre-recorded talks should be 15–45 minutes long. Feel free to ask if you need help with the recording.

You are free to choose whichever storage platform you want (e.g., your own website, a PeerTube instance, a Nextcloud instance, etc.), but we will need to have access to the original file so we can publish it later on audio-video.gnu.org. Your video must be released under a license that at least allows anyone to copy and share it, for any purpose.

You will have to release the video publicly before February 12, so everyone has a chance to see it before the conference. If you are not able to do so (for instance your server cannot handle a huge load), you can alternatively send us a private link to the video and we will upload it on audio-video.gnu.org. If you decide to do so, you will need to have the video ready by February 10.

##### February 12–18: watch the talks

But don't miss the Fosdem conference either!

Be sure to watch the pre-recorded talks before the conference. There will be no presentations on the 19 nor 20.

##### February 19–20: participate

Coming soon! Stay tuned.

#### Code of Conduct

This online conference is an official Guix event. Therefore, the Code of Conduct applies. Please be sure to read it beforehand!

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

## Sunday, January 9, 2022

### Mark Damon Hughes

#### Formatting Strings in Scheme

Most of the time I use primitive display, or print functions:

;; displays a series of args to stdout
(define (print . args) (for-each display args) (flush-output-port) )

;; displays a series of args to stdout, then newline
(define (println . args) (for-each display args) (newline) (flush-output-port) )

;; displays a series of args to given port
(define (fprint port . args) (for-each (λ (x) (display x port)) args) (flush-output-port port) )

;; displays a series of args to given port, then newline
(define (fprintln port . args) (for-each (λ (x) (display x port)) args) (newline port) (flush-output-port port) )

;; displays a series of args to stderr, then newline
(define (errprintln . args)  (let [ (port (current-error-port)) ]
(for-each (λ (x) (display x port)) args) (newline port) (flush-output-port port)
))


but sometimes I actually need to format things:

(import (prefix (srfi s19 time) tm: ))

(format #f "~8a ~10:d ~20a" name score
(tm:date->string (tm:current-date) "~Y-~m-~d ~H:~M:~S") )


Common Lisp format works as described in Chez Scheme, using /#f for destination, and some other Schemes as well; but most Schemes only have the nearly-useless SRFI 28. I'm aware of cat/fox/etc combinatorial formatters, but they're very verbose.

Chez also has date/time functions, but no formatter, so using SRFI 19 - nicely, SRFI 19 mostly does sane things, it's not like C's strftime.

## Friday, January 7, 2022

### Scheme Requests for Implementation

#### SRFI 232: An advanced currying form

SRFI 232 is now in final status.

Scheme lacks a flexible way to create and apply curried procedures. This SRFI describes lambda*, a variant of lambda that creates true curried procedures which also behave just like ordinary Scheme procedures. They can be applied to their arguments one-by-one, all at once, or anywhere in between, without any novel syntax. lambda* also supports nullary and variadic procedures, and procedures created with it have predictable behavior when applied to surplus arguments.

## Thursday, January 6, 2022

### Mark Damon Hughes

#### Gambit hits a mark

(see, the X-Men Gambit has perfect aim and a stupid accent, which still makes him more interesting than Hawkeye; and of course I'm Mark and so is Marc)

With much appreciated help from Marc Feeley, got maintest running.

A couple of lessons: I very much think include paths should include the path of the main source doing the including. Chibi's default is correct, Gambit's default is wrong and requires fixing in every user program. It's "more secure", but if you're running source code from a directory, you can probably trust whatever else is in that dir.

main was frustrating: Gambit manual 2.6 (highlighting mine)

After the script is loaded the procedure main is called with the command line arguments. The way this is done depends on the language specifying token. For scheme-r4rs, scheme-r5rs, scheme-ieee-1178-1990, and scheme-srfi-0, the main procedure is called with the equivalent of (main (cdr (command-line))) and main is expected to return a process exit status code in the range 0 to 255. This conforms to the “Running Scheme Scripts on Unix SRFI” (SRFI 22). For gsi-script and six-script the main procedure is called with the equivalent of (apply main (cdr (command-line))) and the process exit status code is 0 (main’s result is ignored). The Gambit system has a predefined main procedure which accepts any number of arguments and returns 0, so it is perfectly valid for a script to not define main and to do all its processing with top-level expressions (examples are given in the next section).

So your code that looks fine with 1 arg will break with 2, depending on the version. (main . argv) works. I'm in the process of making sure every one of my maintests parses args consistently, and every Scheme disagrees.

Gambit's compiler worked very simply once I got the library on the command line; it doesn't seek out & include them the way Chez does, even though it takes what looks like a search path.

The upside of all this is at least now there's one maintained, fast, R7-compatible Scheme compiler. I'm sticking with Chez (R6) for my code, but it's nice having something 100x faster (gut feeling, not benchmarked) than Chibi to test R7 code on.

### Joe Marshall

#### Idle puzzles 2: Revenge of the Shift

The idle puzzles got some web traffic, so here are a couple more in the same vein. Not much new, just a variation on a theme. They can be done in your head, but I spent a few minutes coding up some solutions to see what was involved.

In the previous puzzles, you were given these numeric primitives:

(import 'cl:zerop (find-package "PUZZLE"))
(defun puzzle::shr (n) (floor n 2))
(defun puzzle::shl0 (n) (* n 2))
(defun puzzle::shl1 (n) (1+ (* n 2)))

and the task was to implement basic arithmetic on non-negative integers.

These puzzles extend the task to include negatve numbers. We are given one additional primitive:

(defun puzzle::-1? (n) (= n -1))

If you want challenge yourself, you could speedrun the problems from start to finish. Or try to adapt the solutions to the prior puzzles with the minimal amount of editing and new code (minimize the diff). Another thing you could try is instrumenting shr, shl0, and shl1 to count the amount of shifting taking place and try to minimize that.

Here is the puzzle:

;;; -*- Lisp -*-

(defpackage "PUZZLE"
(:use)
(:import-from "COMMON-LISP"
"AND"
"COND"
"DEFUN"
"IF"
"FUNCALL"
"FUNCTION"
"LAMBDA"
"LET"
"MULTIPLE-VALUE-BIND"
"NIL"
"NOT"
"OR"
"T"
"VALUES"
"ZEROP"))

(defun puzzle::shr (n) (floor n 2))
(defun puzzle::shl0 (n) (* n 2))
(defun puzzle::shl1 (n) (1+ (* n 2)))

(defun puzzle::-1? (n) (= n -1)) ;; new primitive

(in-package "PUZZLE")

;;; You can only use the symbols you can access in the PUZZLE package.

;;; Problem -1 (Example).  Fix = to handle negative numbers.

(defun = (l r)
(cond ((zerop l) (zerop r))
((-1? l) (-1? r))     ;; new base case
((zerop r) nil)
((-1? r) nil)         ;; new base case
(t (multiple-value-bind (l* l0) (shr l)
(multiple-value-bind (r* r0) (shr r)
(if (zerop l0)
(and (zerop r0)
(= l* r*))
(and (not (zerop r0))
(= l* r*))))))))

;;; Problem 0.  Implement minusp and plusp.

;;; Problem 1.  Fix > to handle negative numbers.

;;; Problem 2.  Fix inc and dec to handle negative numbers.

;;; Problem 3.  Implement logand, logior, and logxor.

;;; Problem 4.  Implement neg (unary minus).

;;; Problem 5.  Fix add and sub to handle negative numbers.

;;; Problem 6.  Fix mul to handle negative numbers.

;;; Problem 7.  Implement floor and ceiling for both positive and
;;;             negative numbers


## My Solutions

The reason we're given a new primitive, -1?, is because the shr function has two fixed points: 0, and -1. So when we write code that recurs over a shr, the recursion is going to bottom out in one of those two base cases and we need to distinguish between them. The earlier puzzles ensured we'd bottom out at zero by specifying non-negative numbers, but if we allow negative numbers, our recursions could bottom out at -1.

;;; Problem 0.  Implement minusp and plusp

(defun minusp (n)
(cond ((zerop n) nil)
((-1? n) t)
(t (minusp (shr n)))))

(defun plusp (n)
(not (or (zerop n)
(minusp n))))

We have to handle both base cases for both the arguments:

;;; Problem 1.  Fix > to handle negative numbers.

(defun > (l r)
(cond ((zerop l) (minusp r))
((-1? l) (and (minusp r)
(not (-1? r))))
((or (zerop r)
(-1? r)) (plusp l))
(t (multiple-value-bind (l* l0) (shr l)
(multiple-value-bind (r* r0) (shr r)
(if (and (not (zerop l0)) (zerop r0))
(not (> r* l*))
(> l* r*)))))

This is interesting. The base case handles when one or the other argument is 0 or -1, but the recursive case doesn't know if the arguments are positive or negative. It doesn't seem to care, either. What is going on? This is the result of using floor on a negative number. The remainder is still a positive number, so when we operate on l0 and r0 we treat them as positive numbers regardless of whether l or r are positive or negative.

;;; Problem 2.  Fix inc and dec to handle negative numbers.

(defun inc (n)
(if (-1? n)
0
(multiple-value-bind (n* n0) (shr r)
(if (zerop n0)
(shl1 n*)
(shl0 (inc n*))))))

(defun dec (n)
(if (zerop n)
-1
(multiple-value-bind (n* n0) (shr r)
(if (zerop n0)
(shl1 (dec n*))
(shl0 n*)))))

Well that was easy, we just had to handle the zero crossing.

No doubt you've noticed that shr shifts a number to the right as if it were held in a register. If you shift the bits out of a negative number, you will notice that the bits come out as if the number were "stored" in two's complement form, with negative numbers being infinitely extended to the left with 1s. This is curious because we didn't design or choose a two's complement representation, it just sort of appears.

;;; Problem 3.  Implement logand, logior, and logxor.

(defun logand (l r)
(cond ((zerop l) 0)
((-1? l) r)
((zerop r) 0)
((-1? r) l)
(t (multiple-value-bind (l* l0) (shr l)
(multiple-value-bind (r* r0) (shr r)
(if (or (zerop l0) (zerop r0))
(shl0 (logand l* r*))
(shl1 (logand l* r*))))))))

(defun logior (l r)
(cond ((zerop l) r)
((-1? l) -1)
((zerop r) l)
((-1? r) -1)
(t (multiple-value-bind (l* l0) (shr l)
(multiple-value-bind (r* r0) (shr r)
(if (and (zerop l0) (zerop r0))
(shl0 (logior l* r*))
(shl1 (logior l* r*))))))))

(defun complement (n)
(cond ((zerop n) -1)
((-1? n) 0)
(t (multiple-value-bind (n* n0) (shr n)
(if (zerop n0)
(shl1 (complement n*))
(shl0 (complement n*)))))))

(defun logxor (l r)
(cond ((zerop l) r)
((-1? l) (complement r))
((zerop r) l)
((-1? r) (complement l))
(t (multiple-value-bind (l* l0) (shr l)
(multiple-value-bind (r* r0) (shr r)
(if (or (and (zerop l0) (zerop r0))
(and (not (zerop l0)) (not (zerop r0))))
(shl0 (logxor l* r*))
(shl1 (logxor l* r*))))))))

;;; Problem 4.  Implement neg (unary minus).

(defun neg (n)
(cond ((zerop n) 0)
((-1? n) 1)
(t (multiple-value-bind (n* n0) (shr n)
(if (zerop n0)
(shl0 (neg n*))
(shl1 (complement n*)))))))

This is basically (inc (complement n)), which is how you negate a two's complement number, but inc and the complement step have been folded together to reduce the amount of shifting.

;;; Problem 5.  Fix add and sub to handle negative numbers.

(cond ((zerop l) r)
((-1? l) (dec r))
((zerop r) l)
((-1? r) (dec l))
(t (multiple-value-bind (l* l0) (shr l)
(multiple-value-bind (r* r0) (shr r)
(if (zerop l0)
(if (zerop r0)
(if (zerop r0)

(cond ((zerop l) (inc r))
((-1? l) r)
((zerop r) (inc l))
((-1? r) l)
(t (multiple-value-bind (l* l0) (shr l)
(multiple-value-bind (r* r0) (shr r)
(if (zerop l0)
(if (zerop r0)
(if (zerop r0)

(defun sub (l r) (add l (neg r)))

The great thing about two's complement is that you can handle negative numbers without changing how you handle the low order bits. For add and addc, I only had to add the two additional base cases for l or r being -1.

By the way, you shouldn't define sub this way. It's double the number of shifts.

;;; Problem 6.  Fix mul to handle negative numbers.

(defun fma (l r a)
(cond ((zerop r) a)
((-1? r) (sub a l))   ;; added this line
(t (multiple-value-bind (r* r0) (shr r)
(fma (shl0 l) r* (if (zerop r0) a (add l a)))))))

(defun mul (l r) (fma l r 0))

Two's complement to the rescue again. The loop can treat the low-order bits of r the same way regardless of whether r is positive or negative.

;;; Problem 7.  Implement floor and ceiling for both positive and
;;;             negative numbers

(defun floor0 (n d)
(if (> d n)
(values 0 n)
(multiple-value-bind (q r) (floor0 n (shl0 d))
(if (> d r)
(values (shl0 q) r)
(values (shl1 q) (sub r d))))))

(defun ceil0 (n d)
(if (not (> n d))
(values 1 (sub n d))
(multiple-value-bind (q r) (ceil0 n (shl0 d))
(if (plusp r1)
(values (shl0 q) r)
(values (dec (shl0 q)) r1))))))

(defun floor (n d)
(if (minusp n)
(multiple-value-bind (q r) (ceiling (neg n) d)
(values (neg q) (neg r)))
(if (minusp d)
(multiple-value-bind (q r) (ceil0 n (neg d))
(values (neg q) r))
(floor0 n d))))

(defun ceiling (n d)
(if (minusp n)
(multiple-value-bind (q r) (floor (neg n) d)
(values (neg q) (neg r)))
(if (minusp d)
(multiple-value-bind (q r) (floor0 n (neg d))
(values (neg q) r))
(ceil0 n d))))

My original method for division doesn't work too well with negative numbers. I worked around that by converting the problem to positive numbers and converting the answer back to negative numbers where appropriate. Supporting negative numbers for division is an exercise in combinatorics. All this checking for minusp and calls to neg cause a lot of shifting of numbers. There is no doubt a better way, but my brain hurts now.

### GNU Guix

#### GNU Guix maintainer rotation

For some time already, Ludovic and Marius have voiced their desire to step down from the Guix maintainers collective. An email announcing the news and calling for new maintainers was sent more than 4 months ago. We're very happy to announce that Efraim Flashner has responded to the call and accepted to join the Guix maintainers collective! Efraim has been with Guix for a long time -- the first recorded Git history of their activity goes back to 2015, and they have since authored more than 6000 commits! More importantly, Efraim has demonstrated traits we value for a co-maintainer, such as good communication and collaboration abilities, values that align well with those of the GNU project and overall a person you'd like to hang out with at FOSDEM :-).

We're sad to see Ludovic and Marius step down from their current duties, but we take comfort knowing they will remain among us for the good hacks and occasional guidance. Losing them as co-maintainers will be a difficult test for the remaining Guix co-maintainers. Ludovic's wit, relentless energy, tactful communication and wise planning have been invaluable to the Guix project throughout the years. Ludovic has mentioned this decision was at least partly based on their desire to guard Guix against the Founder's syndrome, which is something we can only applaud. Marius has served as co-maintainer since October 2019. Their calm, composed attitude has helped us navigate through at times difficult situations, and their technical wizardry has brought us the likes of ungoogled-chromium and a Ganeti service, among many others.

Let's take a moment to say thank you to Ludovic and Marius for their valuable years as maintainers, and wish they can enjoy their newfound extra time :-).

Let's also wish a warm welcome to Efraim. Thank you for stepping up to become a co-maintainer, Efraim!

#### The Guix co-maintainers

The Guix maintainer collective now consists of Efraim Flashner, Mathieu Othacehe, Maxim Cournoyer and Tobias Geerinckx-Rice. You can reach us all by email at guix-maintainers@gnu.org, a private alias.

For information about the responsibilities assumed by the Guix co-maintainers, you are encouraged to read a previous blog post that covered the topic.

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

## Wednesday, January 5, 2022

### Scheme Requests for Implementation

#### SRFI 231: Nonempty Intervals and Generalized Arrays (Updated^2)

SRFI 231 is now in draft status.

This SRFI specifies an array mechanism for Scheme. Arrays as defined here are quite general; at their most basic, an array is simply a mapping, or function, from multi-indices of exact integers $i_0,\ldots,i_{d-1}$ to Scheme values. The set of multi-indices $i_0,\ldots,i_{d-1}$ that are valid for a given array form the domain of the array. In this SRFI, each array's domain consists of the cross product of nonempty intervals of exact integers $[l_0,u_0)\times[l_1,u_1)\times\cdots\times[l_{d-1},u_{d-1})$ of $\mathbb Z^d$, $d$-tuples of integers. Thus, we introduce a data type called $d$-intervals, or more briefly intervals, that encapsulates this notion. (We borrow this terminology from, e.g., Elias Zakon's Basic Concepts of Mathematics.) Specialized variants of arrays provide portable programs with efficient representations for common use cases.

This is a revised version of SRFI 179.

## Tuesday, January 4, 2022

### Mark Damon Hughes

#### Gambit is a risky scheme

(puns about "scheme" names are mandatory)

Neat: New version 4.9.4 of Gambit Scheme is out and they have a web site again after like 3 years.

OK: So I start adapting my little module/how do you run example: here

Bad: Not only does the R7 library system not work, their version of this hello example
will load code from fucking github at runtime! NPM viruses & sabotage are baked into the system. See Modules in Gambit at 30

SIGH.

## Monday, January 3, 2022

### Jérémy Korwin-Zmijowski

#### Following the book « Test Driven Development by Example » with Guile

I love Test Driven Development. You might already know it if you are following my work. Kent Beck wrote the book in 2002. He has chosen to provide examples using Java and Python in an Object Oriented way.

Now, I want to use the Guile programming language to follow the examples in the book and see how things are different ! Guile has Object Oriented capabilities, thanks to its GOOPS module. But here I would like to have a better feeling about the differences between Object Oriented and Functional paradigms.

Guile is not a pure functioral language. However, I will try to avoid non pure features of the language (like mutations).

I will focus on the Part 2, the one where Kent is writing a small version of xUnit.

P.S : Thank you Kent for your work and energy. It has made my dev journey a lot much more enjoyable !

Don't hesitate to give me your opinion, suggest an idea for improvement, or ask a question! To do so : contact me.

Don't miss out on the next ones...

## Saturday, January 1, 2022

### Idiomdrottning

#### Matrix and its bridges

Generally, people on Matrix can use its bridges to join rooms on other networks, even query individual people on that network. In other words, they can use Element or other Matrix clients as a fancy email client or a fancy IRC client or a fancy XMPP client. If they want to. Matrix users can join any room on Libera, for example.

What’s not as easy is vice versa, and that kind of makes sense. It’s not as if every Matrix user out there just automatically should an account on every other protocol ever, like an email account anyone could just spam to. Instead, it’s the Matrix side that can use bridges, like how the main Matrix server is bridged to Libera.

However, that was how I misinterpreted Matrix’ original pitch. “Don’t worry, you can still use all your old networks.” I was like “OK, great, I don’t need Matrix. I’ll just join the rooms from IRC or email.” That’s what sent me down a road of frustration. Instead, it’s “You can use Matrix to reach all your old networks.” They can join our rooms, not vice versa.

What’s the problem? Just that that’s not a particularly unique feature. XMPP had gateways way back when, and Pidgin can do IRC or email just fine. (Maybe this will get better: Gitter, which has been bought by New Vector, creators of Matrix and Element, had an IRC interface that any IRC client could just connect to straight up.) For now, this isn’t a unique selling point for Matrix at all.

Instead, the purpose of Matrix is the room-oriented data structure (as opposed to how IRC and XMPP have message-oriented data). You preserve a chat, sort of like on Mattermost or Gitter, as opposed to display a chat temporarily pieced together from messages. (This room structure is a bad fit for video/audio, which is why it shells out to XMPP for that.)

TL;DR: I’m probably not gonna hop on Matrix any time soon but I’ll check in from time to time to see if any good-quality reverse bridges are created, as opposed to bridges from the Matrix side.

This is a rewrite of an older, sprawling, exploratory post.

## Reverse bridges I know of

An IRC server that connects to Matrix. Not sure if ready for production. Apparently obsoleted in favor of

A currently pretty minimal IRC bridge. A better starting point for hacking efforts?

A Prosody plugin. Unfinished sketch/prototype.

A libpurple plugin in alpha state. No OLM encryption.

A double-puppeteer between a synapse server and a jabber server. You can’t operate the bridge from the jabber side yet.

Let me know if there are more.

## Wednesday, December 29, 2021

### Joe Marshall

#### Idle puzzles

I've been amusing myself with these little puzzles. They're simple enough you can do them in your head, but I coded them up just for fun and to see if they worked.

;;; -*- Lisp -*-

(defpackage "PUZZLE"
(:use)
(:import-from "COMMON-LISP"
"AND"
"COND"
"DEFUN"
"IF"
"FUNCALL"
"FUNCTION"
"LAMBDA"
"LET"
"MULTIPLE-VALUE-BIND"
"NIL"
"NOT"
"OR"
"T"
"VALUES"
"ZEROP"))

(defun puzzle::shr (n) (floor n 2))
(defun puzzle::shl0 (n) (* n 2))
(defun puzzle::shl1 (n) (1+ (* n 2)))

(in-package "PUZZLE")

;;; You can only use the symbols you can access in the PUZZLE package.

;;; Problem -1 (Example).  Define =

(defun = (l r)
(cond ((zerop l) (zerop r))
((zerop r) nil)
(t (multiple-value-bind (l* l0) (shr l)
(multiple-value-bind (r* r0) (shr r)
(if (zerop l0)
(and (zerop r0)
(= l* r*))
(and (not (zerop r0))
(= l* r*))))))))

;;; Problem 0.  Define >

;;; Problem 1.  Define (inc n), returns n + 1 for any non-negative n.

;;; Problem 2.  Define (dec n), returns n - 1 for any positive n.

;;; Problem 3.  Define (add l r), returns the sum of l and r where l
;;;             and r each are non-negative numbers.

;;; Problem 4.  Define (sub l r), returns the difference of l and r
;;;             where l and r are non-negative numbers and l >= r.

;;; Problem 5.  Define (mul l r), returns the product of l and r where
;;;             l and r are non-negative integers.

;;; Problem 6.  Define (pow l r), returns l raised to the r power,
;;;             where l is positive and r is non-negative.

;;; Problem 7.  Define (div l r), returns the quotient and remainder
;;;             of l/r.  l is non-negative and r is positive.

;;; Problem 8.  Define (log l r), returns the integer logarithm of l
;;;             base r


### My Solutions

;;; Problem 0.  Define >

(defun > (l r)
(cond ((zerop l) nil)
((zerop r) t)
(t (multiple-value-bind (l* l0) (shr l)
(multiple-value-bind (r* r0) (shr r)
(if (and (not (zerop l0)) (zerop r0))
(not (> r* l*))
(> l* r*)))))))

This one turned out to be trickier than I thought. I figured you’d basically discard the low order bit and just compare the high ones. And you do, but for this one case where the left bit is one and the right bit is zero. In this case, l > r if l* >= r*, so we swap the arguments and invert the sense of the conditional.

;;; Problem 1.  Define (inc n), returns n + 1 for any non-negative n.

(defun inc (n)
(multiple-value-bind (n* n0) (shr n)
(if (zerop n0)
(shl1 n*)
(shl0 (inc n*)))))

;;; Problem 2.  Define (dec n), returns n - 1 for any positive n.

(defun dec (n)
(multiple-value-bind (n* n0) (shr n)
(if (zerop n0)
(shl1 (dec n*))
(shl0 n*))))
;;; Problem 3.  Define (add l r), returns the sum of l and r where l
;;;             and r each are non-negative numbers.

(cond ((zerop l) r)
((zerop r) l)
(t (multiple-value-bind (l* l0) (shr l)
(multiple-value-bind (r* r0) (shr r)
(if (zerop l0)
(if (zerop r0)
(if (zerop r0)

(cond ((zerop l) (inc r))
((zerop r) (inc l))
(t (multiple-value-bind (l* l0) (shr l)
(multiple-value-bind (r* r0) (shr r)
(if (zerop l0)
(if (zerop r0)
(if (zerop r0)

;;; Problem 4.  Define (sub l r), returns the difference of l and r
;;;             where l and r are non-negative numbers and l >= r.

(defun sub (l r)
(cond ((zerop l) 0)
((zerop r) l)
(t (multiple-value-bind (l* l0) (shr l)
(multiple-value-bind (r* r0) (shr r)
(if (zerop l0)
(if (zerop r0)
(shl0 (sub l* r*))
(shl1 (subb l* r*)))
(if (zerop r0)
(shl1 (sub l* r*))
(shl0 (sub l* r*)))))))))

(defun subb (l r)
(cond ((zerop l) 0)
((zerop r) (dec l))
(t (multiple-value-bind (l* l0) (shr l)
(multiple-value-bind (r* r0) (shr r)
(if (zerop l0)
(if (zerop r0)
(shl1 (subb l* r*))
(shl0 (subb l* r*)))
(if (zerop r0)
(shl0 (sub l* r*))
(shl1 (subb l* r*)))))))))

The presence of a carry or borrow is encoded by which procedure you are in. Effectively, we're encoding the carry or borrow in the program counter.

;;; Problem 5.  Define (mul l r), returns the product of l and r where
;;;             l and r are non-negative integers.

(defun fma (l r a)
(if (zerop r)
a
(multiple-value-bind (r* r0) (shr r)
(fma (shl0 l) r* (if (zerop r0) a (add l a))))))

(defun mul (l r) (fma l r 0))

This has a nice iterative solution if we define a “fused multiply add” operation that given l, r, and a, computes (+ (* l r) a).

Exponentiation has an obvious analogy to multiplication, but instead of doubling l each iteration, we square it, and we multiply into the accumulator rather than adding into it.

;;; Problem 6.  Define (pow l r), returns l raised to the r power,
;;;             where l is positive and r is non-negative.

(defun fem (b e m)
(if (zerop e)
m
(multiple-value-bind (e* e0) (shr e)
(fem (mul b b) e* (if (zerop e0) m (mul b m))))))

(defun pow (b e) (fem b e 1))

For division we use a curious recursion. To divide a big number n by a divisor d, we first pass the buck and divide by (* d 2) and get a quotient and remainder. The quotient we return is twice the quotient we got back, plus 1 if the remainder we got back is bigger than d. We either return the remainder we got back or we subtract d from it.

I find this curious because one usually performs a recursion by making one of the arguments in some way smaller on each recursive call. The recursion bottoms out when the argument can get no smaller. In this recursion, however, we keep trying to divide by bigger and bigger divisors until we cannot anymore.

;;; Problem 7.  Define (div l r), returns the quotient and remainder
;;;             of l/r.  l is non-negative and r is positive.

(defun div (n d)
(if (> d n)
(values 0 n)
(multiple-value-bind (q r) (div n (shl0 d))
(if (> d r)
(values (shl0 q) r)
(values (shl1 q) (sub r d))))))

Logarithm should be analagous (mutatis mutandis).

;;; Problem 8.  Define (log l r), returns the integer logarithm of l
;;;             base r

(defun log (l r)
(if (> r l)
(values 0 l)
(multiple-value-bind (lq lr) (log l (mul r r))
(if (> r lr)
(values (shl0 lq) lr)
(values (shl1 lq) (div lr r))))))

## Sunday, December 19, 2021

### Idiomdrottning

#### Changing inflexible code

There is this Ron Jeffries quote:

I once saw Beck declare two patches of almost completely different code to be “duplication”, change them so that they WERE duplication, and then remove the newly inserted duplication to come up with something obviously better.

And it’s great and I’m into it. I read it back in the day and I’ve since lived that same dream myself, many times.♥ Duplication is the one thing I always do wanna extract.

Extracted duplication is great when you later need to make the same kind of change to both places. You have a single point of truth.

However, you need to know how to inline. Always keep inlining in mind, because you’re gonna need it when you need to make different changes to both places.

Take the reverse of that Beck story:

Declare two calls to the same method to be two patches of almost completely different code, change them so that they are different, and remove the extraction.

If this becomes a tool you’re comfortable with using in both directions, code is going to become flexible and strong, even legacy code.

I wanted to extract a repeated long weird incantation like get into frobnicate lambda line noise rho rho rho your boat iota down the stream, and my pal was like “No, don’t introduce a new abstraction here. Every experienced programmer can instantly grok that idiom you wanted to extract.”

I’m like, wow, that’s great. I’m happy to learn it. But what every experienced programmer need to learn is get comfortable with inlining and extracting. Get as comfortable with refactoring basics as a prose writer is changing two words around.

## Thursday, December 16, 2021

### Idiomdrottning

#### The Inliniac Goes to Town

Let’s see what happens when we apply our Inliniac principles while refactoring some Java code.

The code example we’ll start with comes from the book Clean Code by Robert C. Martin. It’s a good example because the starting code is divided into methods in order to match a set of preconceived semantics, and the whole deal with the Inliac policy is to instead let the code itself dictate the factoring and at which levels we should abstract.

import java.util.ArrayList;

public class PrimeGenerator {
private static int[] primes;
private static ArrayList<Integer> multiplesOfPrimeFactors;

protected static int[] generate(int n) {
primes = new int[n];
multiplesOfPrimeFactors = new ArrayList<Integer>();
set2AsFirstPrime();
checkOddNumbersForSubsequentPrimes();
return primes;
}

private static void set2AsFirstPrime() {
primes[0] = 2;
}

private static void checkOddNumbersForSubsequentPrimes() {
int primeIndex = 1;
for (int candidate = 3;
primeIndex < primes.length;
candidate += 2) {
if (isPrime(candidate))
primes[primeIndex++] = candidate;
}
}

private static boolean isPrime(int candidate) {
if (isLeastRelevantMultipleOfNextLargerPrimeFactor(candidate)) {
return false;
}
return isNotMultipleOfAnyPreviousPrimeFactor(candidate);
}

private static boolean
isLeastRelevantMultipleOfNextLargerPrimeFactor(int candidate) {
int nextLargerPrimeFactor = primes[multiplesOfPrimeFactors.size()];
int leastRelevantMultiple = nextLargerPrimeFactor * nextLargerPrimeFactor;
return candidate == leastRelevantMultiple;
}

private static boolean
isNotMultipleOfAnyPreviousPrimeFactor(int candidate) {
for (int n = 1; n < multiplesOfPrimeFactors.size(); n++) {
if (isMultipleOfNthPrimeFactor(candidate, n))
return false;
}
return true;
}

private static boolean
isMultipleOfNthPrimeFactor(int candidate, int n) {
return
candidate == smallestOddNthMultipleNotLessThanCandidate(candidate, n);
}

private static int
smallestOddNthMultipleNotLessThanCandidate(int candidate, int n) {
int multiple = multiplesOfPrimeFactors.get(n);
while (multiple < candidate)
multiple += 2 * primes[n];
multiplesOfPrimeFactors.set(n, multiple);
return multiple;
}
}


Ok, fun. Let’s start by inlining absolutely everything.

import java.util.ArrayList;

public class PrimeGenerator {
protected static int[] generate(int n) {
int[] primes = new int[n];
ArrayList<Integer> multiples = new ArrayList<Integer>();
primes[0] = 2;
candidates:
for (int primeIndex = 1, candidate = 3;
primeIndex < primes.length;
candidate += 2) {
if (candidate == primes[multiples.size()]
* primes[multiples.size()]) {
continue;
}
for (int i = 1; i < multiples.size(); i++) {
int multiple = multiples.get(i);
while (multiple < candidate)
multiple += 2 * primes[i];
multiples.set(i, multiple);
if (candidate == multiple)
continue candidates;
}
primes[primeIndex++] = candidate;
}
return primes;
}
}


We could stop here and that’d be fine, I’d be pretty happy with that. This version is also pretty nifty because there’s no side effects.

If we’re looking for more to do, I don’t like dealing with the continue labels so let’s start by extracting that part:

import java.util.ArrayList;

public class PrimeGenerator {
private static int[] primes;
private static ArrayList<Integer> multiples;

protected static int[] generate(int n) {
primes = new int[n];
multiples = new ArrayList<Integer>();
primes[0] = 2;
for (int i = 1, c = 3; i < primes.length; i++, c = findNextPrime(c + 2))
primes[i] = c;
return primes;
}

private static int findNextPrime(int candidate) {
if (candidate == primes[multiples.size()]
* primes[multiples.size()]) {
return findNextPrime(candidate+2);
}
for (int i = 1; i < multiples.size(); i++) {
int multiple = multiples.get(i);
while (multiple < candidate)
multiple += 2 * primes[i];
multiples.set(i, multiple);
if (candidate == multiple)
return findNextPrime(candidate+2);
}
return candidate;
}
}


Now, I’m not much of a mathematician so I wouldn’t’ve been able to make a “find next prime” function normally. It’s just the life-changing magic of refactoring.

Here, by scope-lifting the vectors, we reintroduced state, which, depending on what you want to do is kinda iffy. At this point we could still instead pass those in to the subroutine—still stateful, but more explicit.

But let’s just call it, uh, “dynamic programming” and move on.

Next, I see that we’re introducing a variable, int multiple, which is fine, but often it’s interesting to consider introducing variables as arg bindings. This is definitively not a hard and fast rule, just something I’ve been toying with lately. In other words, let’s replace temp with query:

import java.util.ArrayList;

public class PrimeGenerator {
private static int[] primes;
private static ArrayList<Integer> multiples;

protected static int[] generate(int n) {
primes = new int[n];
multiples = new ArrayList<Integer>();
primes[0] = 2;
for (int i = 1, c = 3; i < primes.length; i++, c = findNextPrime(c + 2))
primes[i] = c;
return primes;
}

private static int findNextPrime(int candidate) {
if (candidate == primes[multiples.size()]
* primes[multiples.size()]) {
return findNextPrime(candidate+2);
}
for (int i = 1; i < multiples.size(); i++) {
multiples.set(i, ensureBigger(multiples.get(i), primes[i], candidate));
if (candidate == multiples.get(i))
return findNextPrime(candidate+2);
}
return candidate;
}

private static int ensureBigger(int multiple, int prime, int candidate) {
while (multiple < candidate)
multiple += 2 * prime;
return multiple;
}
}


I’m not sure about that one. It’s a matter of taste. Either way works. Since Java is such a boiler-plate-alicious language, every time we extract, the entire file gets way bigger.

Finally, let’s extract that pesky square (I’m sure a lot of you readers were aching to do so, probably the first one I’d do too), and, optionally but until Java gets tail call elimination, let’s factor those candidates into a while loop or we’ll smash the stack on huge primes. (So weird that Java still does this.)

import java.util.ArrayList;

public class PrimeGenerator {
private static int[] primes;
private static ArrayList<Integer> multiples;

protected static int[] generate(int n) {
primes = new int[n];
multiples = new ArrayList<Integer>();
primes[0] = 2;
for (int i = 1, c = 3; i < primes.length; i++, c = findNextPrime(c + 2))
primes[i] = c;
return primes;
}

private static int findNextPrime(int candidate) {
while (!isPrime(candidate))
candidate += 2;
return candidate;
}

private static boolean isPrime(int candidate) {
if (candidate == square(primes[multiples.size()])) {
return false;
}
for (int i = 1; i < multiples.size(); i++) {
multiples.set(i, ensureBigger(multiples.get(i), primes[i], candidate));
if (candidate == multiples.get(i))
return false;
}
return true;
}

private static int square(int x) {
return x * x;
}

private static int ensureBigger(int multiple, int prime, int candidate) {
while (multiple < candidate)
multiple += 2 * prime;
return multiple;
}
}


That square is a good illustration of the principle that names should be introduced as args (in this case the x) as opposed to as variables at the call site, if only because that’s usually a pretty clean slice through the code. This principle says that if you’re finding yourself making an assigmnent:

int foo = bar + baz;
foo * foo;


You should consider making that a function call instead.

frobnicate(bar + baz);


When I’m making functions this way, I usually call them “frobnicate” and similar Zork names because I have no idea what they’re gonna be. That’s how I found findNextPrime, I just extracted that core loop, called it frobnicate, read through it, realized that it finds the next prime. All about looking at where, uh, this is gonna sound so dumb… where the code wants you to make the perfect cut, where the semantics are actually interfacing each other.

Now, which version is “better”, I dunno. Maybe you think in terms of isLeastRelevantMultipleOfNextLargerPrimeFactor and that’s a meaningful semantic abstraction to you. Refactoring is about giving you the tools you need to make your own choices. And then the Inliniac mindset is to inline everything you’re only saying once, and try to find where the code tells us to extract (for example, duplication like that square) as opposed to our preconceived abstractions.

To me, I don’t see a lot of inherent value in making a

private static int functionThatReturnsTwoPlusTwo() {
int two = 2;
int otherTwo = 2;
return two + otherTwo;
}


Let me rephrase and summarize this entire post, what I think is the core of it:

My philosophy is making functions that my factoring needs, and then naming them. Love it.

The opposing philosphy is to make functions out of the names you want, and then factoring the code to match. That becomes especially cart-before-horse when the function names you want describe what you think the algorithm is, as opposed to describing the problem domain. That’s not so much my jam.

git clone https://idiomdrottning.org/inliniac-primes

## Wednesday, December 15, 2021

### GNU Guix

#### The Big Change

Making cross-cutting changes over a large code base is difficult, but it's occasionally necessary if we are to keep the code base tidy and malleable. With almost 800K source lines of code, Guix can reasonably be called a large code base. One might argue that almost 80% of this code is package definitions, which “doesn't count”. Reality is that it does count, not only because those package definitions are regular Scheme code, but also they are directly affected by the big change Guix has just undergone.

This post looks at what’s probably the biggest change Guix has seen since it started nine years ago and that anyone writing packages will immediately notice: simplified package inputs. Yes, we just changed how each of the 20K packages plus those in third-party channels can declare their dependencies. Before describing the change, how we implemented it, and how packagers can adapt, let’s first take a look at the previous situation and earlier improvements that made this big change possible.

# Packages and inputs

Packages in Guix are defined using a domain-specific language embedded in the Scheme programming language—an EDSL, for the programming language geeks among us. This is a departure from other designs such as that of Nix, which comes with a dedicated language, and it gives packagers and users access to a rich programming interface while retaining the purely declarative style of package definitions.

This package EDSL is one of the oldest bits of Guix and was described in a 2013 paper. Although embedded in Scheme, the package “language” was designed to be understandable by people who’ve never practiced any kind of Lisp before—you could think of it as a parenthesized syntax for JSON or XML. It’s reasonable to say that it’s been successful at that: of the 600+ people who contributed to Guix over the years, most had never written Scheme or Lisp before. The example given in that paper remains a valid package definition:

(define hello
(package
(name "hello")
(version "2.8")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/hello/hello-"
version ".tar.gz"))
(sha256 (base32 "0wqd8..."))))
(build-system gnu-build-system)
(arguments
'(#:configure-flags
("--disable-color"
,(string-append "--with-gawk="
(assoc-ref %build-inputs "gawk")))))
(inputs (("gawk" ,gawk)))
(synopsis "GNU Hello")
(description "An illustration of GNU's engineering practices.")
(home-page "http://www.gnu.org/software/hello/")
(license gpl3+)))

Of particular interest here is the inputs field, which lists build-time dependencies. Here there’s only one, GNU Awk; it has an associated label, "gawk". The ,gawk bit lets us insert the value of the gawk variable, which is another package. We can list more dependencies like so:

(inputs (("gawk" ,gawk)
("gettext" ,gnu-gettext)
("pkg-config" ,pkg-config)))

Quite a bit of boilerplate. Unless you’re into Lisp, this probably looks weird to you—manageable, but weird. What’s the deal with this backquote and those commas? The backquote is shorthand for quasiquote and commas are shorthand for unquote; it’s a facility that Lisp and Scheme provide to construct lists.

Lispers couldn’t live without quasiquote, it’s wonderful. Still, exposing newcomers to this syntax has always been uncomfortable; in tutorials you’d end up saying “yeah don’t worry, just write it this way”. Our goal though is to empower users by giving them abstractions they can comprehend, hopefully providing a smooth path towards programming without noticing. This seemingly opaque backquote-string-unquote construct works against that goal.

Then, you ask, why did Guix adopt this unpleasant syntax for inputs in the first place? Input syntax had to satisfy one requirement: that it’d be possible for “build-side code” to refer to a specific input. And what’s build-side code? It’s code that appears in the package definition that is staged for later execution—code that’s only evaluated when the package is actually built. The bit that follows #:configure-flags in the example above is build-side code: it’s an expression evaluated if and when the package gets built. That #:configure-flags expression refers to the gawk package to construct a flag like "--with-gawk=/gnu/store/…-gawk-5.0.1"; it does so by referring to the special %build-inputs variable, which happens to contain an association list that maps input labels to file names. The "gawk" label in inputs is here to allow build-side code to get at an input’s file name.

Still here? The paragraphs above are a demonstration of the shortcoming of this approach. That we have to explain so much suggests we’re lacking an abstraction that would make the whole pattern clearer.

# G-expressions and self-referential records

The missing abstraction came to Guix a couple of years later: G-expressions. Without going into the details, which were covered elsewhere, notably in a research article, g-expressions, or “gexps”, are traditional Lisp s-expressions (“sexps”) on steroids. A gexp can contain a package record or any other “file-like object” and, when that gexp is serialized for eventual execution, the package is replaced by its /gnu/store/… file name.

Gexps have been used since 2014–2015 in all of Guix System and they’ve been great to work with, but package definitions were stuck with old-style sexps. One reason is that a big chunk of the code that deals with packages and build systems had to be ported to the gexp-related programming interfaces; a first attempt had been made long ago but performance was not on par with that of the old code, so postponing until that was addressed seemed wiser. The second reason was that using gexps in package definitions could be so convenient that packagers might unwillingly find themselves creating inconsistent packages.

We can now rewrite our hello package such that configure flags are expressed using a gexp:

(define hello
(package
(name "hello")
;; …
(arguments
(list #:configure-flags
#~("--disable-color"
,(string-append "--with-gawk=" #$gawk)))) (inputs (("gawk" ,gawk))))) The reference inserted here with #$gawk (#$ is synonymous for ungexp, the gexp equivalent of traditional unquote) refers to the global gawk variable. This is more concise and semantically clearer than the (assoc-ref %build-inputs "gawk") snippet we had before. Now suppose you define a package variant using this common idiom: (define hello-variant (package (inherit hello) (name "hello-variant") (inputs (("gawk" ,gawk-4.0))))) Here the intent is to create a package that depends on a different version of GNU Awk—the hypothetical gawk-4.0 instead of gawk. However, the #:configure-flags gexp of this variant still refers to the gawk variable, contrary to what the inputs field prescribes; in effect, this variant depends on the two Awk versions. To address this discrepancy, we needed a new linguistic device, to put it in a fancy way. It arrived in 2019 in the form of self-referential records. Within a field such as the arguments field above, it’s now possible to refer to this-package to get at the value of the package being defined. (If you’ve done object-oriented programming before, you might be thinking that we just rediscovered the this or self pointer, and there’s some truth in it. :-)) With a bit of syntactic sugar, we can rewrite the example above so that it refers to the Awk package that appears in its own inputs field: (define hello (package (name "hello") ;; … (arguments (list #:configure-flags #~(list (string-append "--with-gawk=" #$(this-package-input "gawk")))))
(inputs (("gawk" ,gawk)))))

With this in place, we can take advantage of gexps in package definitions while still supporting the common idiom to define package variants, wheee!

That was a long digression from our input label theme but, as you can see, all this interacts fairly tightly.

# Getting rid of input labels

Now that we have gexps and self-referential records, it looks like we can finally get rid of input labels: they’re no longer strictly necessary because we can insert packages in gexps instead of looking them up by label in build-side code. We “just” need to devise a backward compatibility plan…

Input labels are pervasive; they’re visible in three contexts:

1. in the inputs fields of package definitions;
2. on the build side with the inputs keyword parameter of build phases;
3. in the Scheme programming interface since package-inputs and related functions are expected to return a list of labeled inputs.

We’re brave but not completely crazy, so we chose to focus on #1 for now—it’s also the most visible of all three—, with an plan to incrementally address #2, leaving #3 for later.

To allow for label-less inputs, we augmented the record interface with field sanitizers. This feature allows us to define a procedure that inspects and transforms the value specified for the inputs, native-inputs, and propagated-inputs. Currently that procedure reintroduces input labels when they’re missing. In a sense, we’re just changing the surface syntax but under the hood everything is unchanged. With this change, our example above can now be written like this:

(define hello
(package
(name "hello")
;; …
(inputs (list gawk))))

Much nicer, no? The effect is even more pleasant for packages with a number of inputs:

(define-public guile-sqlite3
(package
(name "guile-sqlite3")
;; …
(build-system gnu-build-system)
(native-inputs (list autoconf automake guile-3.0 pkg-config))
(inputs (list guile-3.0 sqlite))))

That’s enough to spark joy to anyone who’s taught the previous syntax. Currently this is transformed into something like:

(define-public guile-sqlite3
(package
;; …
(native-inputs
(list autoconf automake guile-3.0 pkg-config)))))

… where the add-input-label function turns a package into a label/package pair. It does add a little bit of run-time overhead, but nothing really measurable.

There are also cases where package definitions, in particular for package variants, would directly manipulate input lists as returned by package-inputs and related procedures. It’s a case where packagers had to be aware of input labels, and they would typically use association list (or “alist”) manipulation procedures and similar construct—this is context #3 above. To replace those idioms, we defined a higher-level construct that does not assume input labels. For example, a common idiom when defining a package variant with additional dependencies goes like this:

(define hello-with-additional-dependencies
(package
(inherit hello)
(name "hello-with-bells-and-whistles")
(inputs (("guile" ,guile-3.0)
("libtextstyle" ,libtextstyle)
,@(package-inputs hello)))))

The variant defined above adds two inputs to those of hello. We introduced a macro, modify-inputs, which allows packagers to express that in a higher-level (and less cryptic) fashion, in a way that does not refer to input labels. Using this other linguistic device (ha ha!), the snippet above becomes:

(define hello-with-additional-dependencies
(package
(inherit hello)
(name "hello-with-bells-and-whistles")
(inputs (modify-inputs (package-inputs hello)
(prepend guile-3.0 libtextstyle)))))

Similarly, modify-inputs advantageously subsumes alist-delete and whatnot when willing to replace or remove inputs, like so:

(modify-inputs (package-inputs hello)
(replace "gawk" my-special-gawk)
(delete "guile"))

On the build side—context #2 above—, we also provide new procedures that allow packagers to avoid relying on input labels: search-input-file and search-input-directory. Instead of having build phases that run code like:

(lambda* (#:key inputs #:allow-other-keys)
;; Replace references to “/sbin/ip” by references to
;; the actual “ip” command in /gnu/store.
(substitute* "client/scripts/linux"
(("/sbin/ip")
;; Look up the input labeled “iproute”.
(string-append (assoc-ref inputs "iproute")
"/sbin/ip"))))

… you’d now write:

(lambda* (#:key inputs #:allow-other-keys)
;; Replace references to “/sbin/ip” by references to
;; the actual “ip” command in /gnu/store.
(substitute* "client/scripts/linux"
(("/sbin/ip")
;; Search “/sbin/ip” among all the inputs.
(search-input-file inputs "/sbin/ip"))))

Nothing revolutionary but a couple of advantages: code is no longer tied to input labels or package names, and search-input-file raises an exception when the file is not found, which is better than building an incorrect file name.

That was a deep dive into packaging! If you’re already packaging software for Guix, you hopefully see how to do things “the new way”. Either way, it’s interesting to see the wide-ranging implications of what initially looks like a simple change. Things get complex when you have to consider all the idioms that 20,000 packages make use of.

# Adapting to the new style

It’s nice to have a plan to change the style of package definitions, but how do you make it happen concretely? The last thing we want is, for the next five years, to have to explain two package styles to newcomers instead of one.

First, guix import now returns packages in the new style. But what about existing package definitions?

Fortunately, most of the changes above can be automated: that’s the job of the guix style command that we added for this purpose, but which may eventually be extended to make package definitions prettier in all sorts of ways. From a checkout, one can run:

./pre-inst-env guix style

Whenever it can be done safely, package inputs in every package definition are rewritten to the new style: removing input labels, and using modify-inputs where appropriate. If you maintain your own channel, you can also run it for your packages:

guix style -L /path/to/channel my-package1 my-package2 …

We recommend waiting until the next Guix release is out though, which could be a month from now, so that your channel remains usable by those who pinned an older revision of the guix channel.

We ran guix style a couple of days ago on the whole repository, leading to the biggest commit in Guix history:

460 files changed, 37699 insertions(+), 49782 deletions(-)

Woohoo! Less than 15% of the packages (not counting Rust packages, which are a bit special) have yet to be adjusted. In most cases, package inputs that were left unchanged are either those where we cannot automatically determine whether the change would break the package, for instance because build-side code expects certain labels, and those that are “complex”—e.g., inputs that include conditionals.

The key challenge here was making sure guix style transformations are correct; by default, we even want to be sure that changes introduced by guix style do not trigger a rebuild—that package derivations are unchanged.

To achieve that, guix style correlates the source code of each package definition with the corresponding live package record. That allows it to answer questions such as “is this label the name of the input package”. That way, it can tell that, say:

(inputs (("libx11" ,libx11)
("libxcb" ,libxcb)))

can be rewritten without introducing a rebuild, because labels match actual package names, whereas something like this cannot, due to label mismatches:

(inputs (("libX11" ,libx11)
("xcb" ,libxcb)))

guix style can also determine situations where changes would trigger a rebuild but would still be “safe”—without any observable effect. You can force it to make such changes by running:

guix style --input-simplification=safe

Because we’re all nitpicky when it comes to code formatting, guix style had to produce nicely formatted code, and to make local changes as opposed to rewriting complete package definitions. Lisps are famous for being homoiconic, which comes in handy in such a situation.

But the tools at our disposal are not capable enough for this application. First, Scheme’s standard read procedure, which reads an sexp (or an abstract syntax tree if you will) from a byte stream and returns it, does not preserve comments. Obviously we’d rather not throw away comments, so we came up with our own read variant that preserves comments. Similarly, we have a custom pretty printer that can write comments, allowing it to achieve changes like this:

@@ -2749,18 +2707,17 @@ (define-public debops
"debops-debops-defaults-fall-back-to-less.patch"))))
(build-system python-build-system)
(native-inputs
-     (("git" ,git)))
+     (list git))
(inputs
-     (("ansible" ,ansible)
-       ("encfs" ,encfs)
-       ("fuse" ,fuse)
-       ("util-linux" ,util-linux)  ;for umount
-       ("findutils" ,findutils)
-       ("gnupg" ,gnupg)
-       ("which" ,which)))
+     (list ansible
+           encfs
+           fuse
+           util-linux ;for umount
+           findutils
+           gnupg
+           which))
(propagated-inputs
-     (("python-future" ,python-future)
-       ("python-distro" ,python-distro)))
+     (list python-future python-distro))
(arguments
(#:tests? #f

The pretty printer also has special rules for input lists. For instance, lists of five inputs or less go into a single line, if possible, whereas longer lists end up with one input per line, which is often more convenient, especially when visualizing diffs. It also has rules to format modify-inputs in the same way we’d do it in Emacs:

@@ -171,9 +170,9 @@ (define-public arcan-sdl
(inherit arcan)
(name "arcan-sdl")
(inputs
-     (("sdl" ,sdl)
-       ,@(fold alist-delete (package-inputs arcan)
-               '("libdrm"))))
+     (modify-inputs (package-inputs arcan)
+       (delete "libdrm")
+       (prepend sdl)))
(arguments

Overall that makes guix style a pretty fun meta-project!

# “Worse is better” or “the Right Thing”?

There are several lessons here. One is that having an embedded domain-specific language is what makes such changes possible: yes package definitions have a declarative field, but we do not hide the fact that their meaning is determined by the broader Guix framework, starting with the (guix packages) module, which defines the package record type and associated procedures. Having a single repository containing both package definitions and “the package manager” is also a key enabler; we can change the framework, add new linguistic tools, and adjust package definitions at the same time. This is in contrast, for instance, with the approach taken by Nix, where the language implementation lives separately from the package collection.

Another one is that a strong, consistent community leads to consistent changes—not surprisingly in fact. It’s a pleasure to see that we, collectively, can undertake such overarching changes and all look in the same direction.

The last lesson is in how we approach design issues in a project that is now a little more than nine years old. Over these nine years it’s clear that we have usually favored “the right thing” in our design—but not at any cost. This whole change is an illustration of this. It was clear from the early days of Guix that input labels and the associated machinery were suboptimal. But it took us a few years to design an implement the missing pieces: G-expressions, self-referential records, and the various idioms that allow package definitions to benefit from these. In the meantime, we built a complete system and a community and we gained experience. We cannot claim we attained the right thing, if such a thing exists, but certainly package definitions today are closer to the declarative ideal and easier to teach.

# It’s here today!

This big change, along with countless other improvements and package upgrades, is just one guix pull` away! After months of development, we have just merged the “core updates” branch bringing so many new things—from GNOME 41, to GCC 10 by default, to hardened Python packages and improved executable startup times. This paves the way for the upcoming release, most likely labeled “1.4”, unless a closer review of the changes that have landed leads us to think “2.0” would be more appropriate… Stay tuned!