Planet Scheme

Monday, June 13, 2022

GNU Guix

Celebrating 10 years of Guix in Paris, 16–18 September

It’s been ten years of GNU Guix! To celebrate, and to share knowledge and enthusiasm, a birthday event will take place on September 16–18th, 2022, in Paris, France. The program is being finalized, but you can already register!

10 year anniversary artwork

This is a community event with several twists to it:

  • Friday, September 16th, is dedicated to reproducible research workflows and high-performance computing (HPC)—the focuses of the Guix-HPC effort. It will consist of talks and experience reports by scientists and practitioners.
  • Saturday targets Guix and free software enthusiasts, users and developers alike. We will reflect on ten years of Guix, show what it has to offer, and present on-going developments and future directions.
  • on Sunday, users, developers, developers-to-be, and other contributors will discuss technical and community topics and join forces for hacking sessions, unconference style.

Check out the web site and consider registering as soon as possible so we can better estimate the size of the birthday cake!

If you’re interested in presenting a topic, in facilitating a session, or in organizing a hackathon, please get in touch with the organizers at guix-birthday-event@gnu.org and we’ll be happy to make room for you. We’re also looking for people to help with logistics, in particular during the event; please let us know if you can give a hand.

Whether you’re a scientist, an enthusiast, or a power user, we’d love to see you in September. Stay tuned for updates!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

by Ludovic Courtès, Tanguy Le Carrour, Simon Tournier at Monday, June 13, 2022

Thursday, June 9, 2022

Göran Weinholt

Cond-expand and #ifdef

In the C programming language you can ask the macro preprocessor to keep or remove part of a source file. This is done with #ifdef. The equivalent in Scheme is called cond-expand. R7RS Scheme has two different instances of cond-expand, while R6RS Scheme does not have it all. What does R6RS do instead, and is cond-expand a bad idea?

Use cases

What is #ifdef, an its cousins #if and #ifndef, used for? And why might we want its equivalent in Scheme? There are two major use cases for #ifdef: build-time configuration and portability. The build system will usually have some sort of configuration script that detects what system it’s running on. Usually these scripts also let the user enable or disable features.

Chez Scheme runs on top of a chunk of fairly portable C and uses #ifdef as described:

$ grep -h '^#ifdef' ChezScheme/c/*.[ch] |awk '{print $2}' \
   | sort -u | xargs
ARCHYPERBOLIC ARMV6 BSDI CHAFF CHECK_FOR_ROSETTA CLOCK_HIGHRES
CLOCK_MONOTONIC CLOCK_MONOTONIC_HR CLOCK_PROCESS_CPUTIME_ID
CLOCK_REALTIME CLOCK_REALTIME_HR CLOCK_THREAD_CPUTIME_ID DEBUG
DEFINE_MATHERR DISABLE_CURSES EINTR ENABLE_OBJECT_COUNTS
FEATURE_EXPEDITOR FEATURE_ICONV FEATURE_PTHREADS FEATURE_WINDOWS FLOCK
FLUSHCACHE FunCRepl GETWD HANDLE_SIGWINCH HPUX I386 IEEE_DOUBLE ITEST
KEEPSMALLPUPPIES LIBX11 LITTLE_ENDIAN_IEEE_DOUBLE LOAD_SHARED_OBJECT
LOCKF LOG1P LOOKUP_DYNAMIC MACOSX MAP_32BIT __MINGW32__ MMAP_HEAP
NAN_INCLUDE NO_DIRTY_NEWSPACE_POINTERS NOISY
NO_LOCKED_OLDSPACE_OBJECTS PPC32 PROMPT PTHREADS SA_INTERRUPT
SA_RESTART SAVEDHEAPS segment_t2_bits segment_t3_bits SIGBUS SIGQUIT
SOLARIS SPARC SPARC64 TIOCGWINSZ USE_MBRTOWC_L WIN32 _WIN64 WIPECLEAN
X86_64

Macros like FEATURE_EXPEDITOR turn on and off functionality, while macros like HPUX and I386 are used for portability. So, configuration and portability.

Portability?

Does #ifdef truly help with portability? It can certainly seem this way, but there’s a different way to think about this issue. This is what Rob Pike had to say on the TUHS main list:

C with #ifdefs is not portable, it is a collection of 2^n overlaid programs, where n is the number of distinct #if[n]def tags. It’s too bad the problems of that approach were not appreciated by the C standard committee, who mandated the #ifndef guard approach that I’m sure could count as a provable billion dollar mistake, probably much more. The cost of building #ifdef’ed code, especially with C++, which decided to be more fine-grained about it, is unfathomable.

For each C file with #ifdef you need to understand what happens if the condition is true versus if it’s false. It’s simple with just one #ifdef, but the problem grows exponentially.

Configuration?

You can use #ifdef for conditional compilation, to turn on and off features. But this can also create a mess like that described by Pike above. The GNU Coding Standards have this to say:

When supporting configuration options already known when building your program we prefer using if (... ) over conditional compilation, as in the former case the compiler is able to perform more extensive checking of all possible code paths.

The same sentiment is echoed by Douglas McIlroy in the message that preceded Pike’s message above.

This approach is generally a good idea. All those conditionals give you a lot of code paths. Checking that all them even compile is difficult to do by hand, and you need all the help you can get. When you use #ifdef you hide the code from the compiler. The compiler can’t check code that it can’t see. Using if (... ) when the expression is constant at compile time should give the same result as conditional inclusion, at least if you have an optimizing compiler.

Also Considered Harmful

It’s not just people on the Internet saying these things about #ifdef, and the complaints are not new either.

We believe that a C programmer’s impulse to use#ifdef in an attempt at portability is usually a mistake. Portability is generally the result of advance planning rather than trench warfare involving #ifdef. In the course of developing C News on different systems, we evolved various tactics for dealing with differences among systems without producing a welter of #ifdefs at points of difference. We discuss the alternatives to, and occasional proper use of, #ifdef.

Source: SPENCER, Henry; COLLYER, Geoff. #ifdef considered harmful, or portability experience with C News. In: USENIX Summer 1992 Technical Conference (USENIX Summer 1992 Technical Conference). 1992.

You can use Google Scholar to find papers that cite this paper, if you’re interested in more reading.

cond-expand is not as bad…

The first standardization of cond-expand that I know of is Marc Feeley’s SRFI-0. It is also part of R7RS Scheme, where I believe it has seen wider adoption than plain SRFI-0.

There is one major difference between #ifdef and cond-expand. The former is handled by a preprocessor that does not understand the lexical syntax of the language is it working with. You can even use cpp with other languages than C, e.g. assembly. This means you can easily introduce latent syntax errors with #ifdef.

There is a cond-expand available from inside define-library and another one available as syntax in (scheme base). Both of these are handled after the source file has been parsed. This means that cond-expand cannot create an unbalanced syntax tree. What I mean by this is that you can’t somehow use cond-expand wrong in such a way that the parenthesis become unbalanced. To make this mistake with #ifdef is trivial; simply place } before #endif when it should have been after, or vice versa.

You get bonus points, so to speak, if you do this near code that handles portability to operating systems that you can’t test your changes on.

… but not really better

Apart from the differences in how the compiler handles them, they do actually express the same thing. One can look at cond-expand as morally equivalent to a series of #if, #else and #endif directives. Therefore the very same problems that happen with #ifdef also happen with cond-expand.

I’m not optimistic about the future landscape of R7RS code if cond-expand is not recognized for the problems it brings. It may be that each R7RS library will become a jungle of 2n overlaid libraries. I have seen some indication of this process already beginning when looking at the packages in Snow Fort.

Fortunately I have not seen any examples where cond-expand is used to change which identifiers are exported from a library, but that day may yet come.

Back to R6RS

So in the beginning of this article I wrote that R6RS Scheme does not have cond-expand. Does that mean it has another way to handle these problems?

No, but in practice: yes. In the R6RS report there are only libraries as a suggested way to handle this. And there isn’t really a way to conditionally import libraries at compile time.

This situation has given rise to some creative solutions. The configuration and portability problems do not disappear just like that, so people have tried to solve it within the restrictions of the language.

Portability between R6RS implementations

I believe that all R6RS Scheme implementations implement the de facto standard of importing libraries by first looking for them in files that end with the .<impl>.sls suffix before they try .sls. If Chez Scheme sees (import (foo)) then it first tries foo.chezscheme.sls before it tries foo.sls. This mechanism, even though it’s not in R6RS, is widely implemented.

This mechanism is used to create compatibility libraries. One striking example is the (xitomatl common) library from Derick Eddington’s xitomatl. It contains a few procedures that traditionally appear in Scheme implementation, but which are not in the reports. Here is common.chezscheme.sls:

;; Copyright 2009 Derick Eddington.  My MIT-style license is in the file named
;; LICENSE from the original collection this file is distributed with.

(library (xitomatl common)
  (export
    add1 sub1
    format printf fprintf pretty-print
    gensym
    time
    with-input-from-string with-output-to-string
    system
    ;; TODO: add to as needed/appropriate
    )
  (import
    (chezscheme))
)

There are matching libraries for Guile, Ikarus, Larcency, Racket, Mosh and Ypsilon. They export the same identifiers, but they all have some tweaks to adapt them to the various implementations.

This is the “Plan 9” approach to portability, as briefly described in the mailing list thread referenced above. Define APIs and let the implementation of the API hide the portability problems from the rest of the program.

Configuration for R6RS code

What is to be done for configuration? When I need this in my own code, I create a library that exports the configuration as identifier syntax. Here’s an abbreviated example from Loko Scheme:

(library (loko arch amd64 config)
  (export
    ; ...
    use-popcnt)
  (import
    (rnrs))

(define-syntax define-const
  (syntax-rules ()
    ((_ name v)
     (define-syntax name
       (identifier-syntax v)))))

; ...

(define-const use-popcnt #f))

I can then use this identifier syntax as a regular variable, like (if use-popcnt <do-this> <do-that>). But thanks to define-const it becomes inlined at the place where it is used. So if it’s set to #f then the expanded conditional is actually (if #f <do-this> <do-that>), which is trivial to optimize.

Akku supports this stuff

I have included support in Akku for both the R6RS and R7RS approach to portability. Akku will keep track of which Scheme implementation an R6RS library is meant for and adapt the way it installs the library. It will use the .<impl>.sls extension and even escape the file name correctly for that implementation.

With R7RS libraries it merely needs to install the .sld file to the right location. This is simple enough to do. But Akku also translates R7RS libraries into R6RS libraries. Akku has to do some interesting juggling when cond-expand appears at the define-library level.

Akku checks the list of features that appear cond-expand and looks to see if it recognizes any implementation names. For each implementation it then creates a copy of the library that is specific to that implementation. For each such copy it expands all cond-expand expressions at the define-library level, as best as it can, and installs .<impl>.sls files. Kind of dirty, but it mostly works.

For example, this library will be installed as hello.chezscheme.sls, hello.loko.sls, hello.sld and hello.sls:

(define-library (hello)
  (export hello)
  (cond-expand
   ((library (rnrs))
    (import (rnrs)))
   (else
    (import (scheme base))))

  (cond-expand
   (chezscheme
    (begin
      (define (hello)
        (display "Hello Chez!\n"))))
   (loko
    (begin
      (define (hello)
        (display "Hello Loko!\n"))))
   (else
    (begin
      (define (hello)
        (display "Hello world!\n"))))))

(The hello.loko.sls file it creates is actually a symlink to hello.sld because Akku knows that Loko supports R7RS).

Maybe a way forward

The picture I’ve painted seems quite damning for cond-expand. But the problem is not really cond-expand itself. The problem is when the misuse of cond-expand leads to a mess. I’m not advocating for its removal, but I would like it to be better understood for what it is. Some widely distributed guidelines for how to use it would go far in reducing the damage.

The “Plan 9” approach of making APIs can be done with cond-expand just as well as with the R6RS approach of .<impl>.sls>. It just requires that you know what you’re doing; that it’s a bad idea to sprinkle all your code with cond-expand and that you should keep this code in isolated libraries.

Finally, I’d like to say that cond-expand is actually more powerful than the .<impl>.sls approach. This power unfortunately makes static analysis of library declarations more difficult, but it also gives you access to feature identifiers that are more interesting than just the name of the Scheme implementation, such as x86-64 and clr. But please do hide your use of these behind an appropriate API.

by weinholt at Thursday, June 9, 2022

Wednesday, June 8, 2022

Idiomdrottning

Leave my quote lines alone

I can’t believe the resistance to this 😭😭😭😭😭😭😭😭😭😭😭😭

It’s like, the cost/benefit ratio is completely lopsided. 💔

Just please please please change it ��

The fact whenever I try to send a line that looks like this in Delta Chat, it gets tampered with, feels almost cruel and petty and mean spirited.

Like, what’s the harm? Just please let me send lines that look that way������ There’s never anything bad that’s gonna happen from not tampering with these quotes.

Mirror universe Clippy: “Looks like you’re trying to write a quote line, I’d better silently change it to not be a quote line. Hehehehe. Only way to quote is to hit the FOTU button.�

This is actually what RFC 3676 intended. So the windmill I’m tilting at is not only one app of wrongness, it’s eighteen years of wrongness since it came out in 2004.

I even asked the guy who worked on the RFC, who very kindly replied (thank you for that) but… he doubled down on this madness! I was reading the source code of his email replies to me and I was like “See, these are the semantics I want; your mailer reflowed long lines, but kept > at the start of lines you quoted from me� and he was like inserting those is the job of the MUA. He used TOFU by the way. Which, I know traditional hacker culture says TOFU is worse than FOTU while I don’t care. I don’t care how you do it, I just wanna keep my own house in order and that sometimes means excerpting a single phrase with a > line. As is possible in traditional email, traditional paper letters, on Usenet, on Gemini, in Markdown, on Slack, on Discourse, on Reddit, on Stack Exchange, on Slack, on Mattermost, on Matrix, on the moon and the sun and the clouds and the pebbles and trees, on Mars, on Venus, on everything between us. Anywhere but in Delta Chat, where my arms suddenly get chopped off if I try it and I just come across as an embarrassing weirdo to the person on the other side. And I didn’t even know about it for the first year I was using Delta Chat (because it hides this tampering from the sender so I’m embarrassing myself unknowingly) but since I found out about it six months ago I’ve been trying to get this changed.

Like, in what universe is this a desired feature by an end user, someone who goes “Ah yes I’m really glad I don’t have to type space > in order to send space >, I only have to type >, and when I do type space >, I’m glad it still gets displayed as only >, and I’m sure happy I can’t manually quote a line!� That person is my enemy! Luckily, they don’t exist (or if they do, they’re like one out of forty trillion people) while the people who instead occasionally wanna, IDK, sometimes indicate that “hey, this thing you said earlier, this specific part�, those people? They are me 💔

by Idiomdrottning (sandra.snan@idiomdrottning.org) at Wednesday, June 8, 2022

Sunday, June 5, 2022

Idiomdrottning

A conditionally transforming combinator

?-> is a combinator that takes a predicate test and a transformer, and returns a unary procedure that transforms its argument if the predicate applies.

That’s a mouthful. Let’s break it down with some examples:

You can make a car that only cars if the list is not null:

(map (?-> (o not null?) car) '((a b c) () (1 2 3) () ()))
⇒ (a () 1 () ())

Or a reverse that only reverses if it receives a list:

(map (?-> list? reverse) '(a (l u f r e d n o w) "summer's" (y a d)))
⇒ (a (w o n d e r f u l) "summer's" (d a y))

The default when the predicate doesn’t apply is to just pass through the argument, but you can set a different default with a keyword argument:

(map (?-> (o not null?) car default: #f) '((a b c) () (1 2 3) () ()))
⇒ (a #f 1 #f #f)

As for the name, I know I said no new ASCII art names, but ? for predicates and -> for transformation are both grandfathered in.

?-> is available in brev-separate from version 1.52.

by Idiomdrottning (sandra.snan@idiomdrottning.org) at Sunday, June 5, 2022

newline1 and skip1

newline1 is a version of Scheme’s newline that does nothing the first time it’s called.

(for-each (fn (newline1) (display x)) '(a b c d e))

prints

a
b
c
d
e

skip1 is a general combinator that makes these:

(define tab1 (skip1 (fn (newline) (display "	"))))
(for-each (fn (tab1) (display x)) '(ol li li li))

prints

ol
	li
	li
	li

To reset them (so they’ll skip their next invocation), call them with a single argument, 'reset.

(newline1 'reset)
(tab1 'reset)

skip1 can take any number of procedures with any number of arguments, it runs ((compose proc1 proc2 …) arg1 arg2 …).

For example,

(define conv1 (skip1 print add1 *))
(for-each conv1 '(1 2 3) '(10 20 30))

prints

41
91

newline1 and skip1 are available in brev-separate from version 1.52.

by Idiomdrottning (sandra.snan@idiomdrottning.org) at Sunday, June 5, 2022

Saturday, June 4, 2022

Gauche Devlog

Is this an Undefined Behavior?

Is this an Undefined Behavior?

Automated tests of Gauche HEAD on Windows platform started failing since several days ago. The log showed SHA1 digest result didn't match. It's weird, for I haven't touched that part of code for long time.

I isolated the reproducible condition. It happens with the fairly new gcc (11.2.0) with -O2. It doesn't exhibit without optimization, nor with gcc 10.2.0 or other previous versions of gcc I have.

The problematic code is Aaron D. Gifford's SHA implementation sha2.c ( http://www.aarongifford.com/computers/sha.html ). It was last updated in January 2004, so it's pretty old, but I think it's still widely used.

I narrowed down the problem to around here:

        /* Set the bit count: */
#if BYTE_ORDER == LITTLE_ENDIAN
        /* Convert FROM host byte order */
        REVERSE64(context->s1.bitcount,context->s1.bitcount);
#endif
        void *buf56 = &context->s1.buffer[56];
        *(sha_word64*)buf56 = context->s1.bitcount;

        /* Final transform: */
        SHA1_Internal_Transform(context, (sha_word32*)context->s1.buffer);

In our case, BYTE_ORDER is LITTLE_ENDIAN. REVERSE64 is a macro to swap the byte order of a 64bit word. context->s1.bitcount is uint64_t, and context->s1.buffer is an array of unsigned chars. What it does is to store 64bit-word of bitcount into the buffer from 56th octet in the network byte order, and calls SHA1_Internal_Transform.

It compiles to this code with optimization:

   25ca75e9d:   48 8b 53 18             mov    0x18(%rbx),%rdx
   25ca75ea1:   48 0f ca                bswap  %rdx
   25ca75ea4:   48 89 53 18             mov    %rdx,0x18(%rbx)
   25ca75ea8:   48 89 d9                mov    %rbx,%rcx
   25ca75eab:   4c 89 e2                mov    %r12,%rdx
   25ca75eae:   e8 8d fa ff ff          call   25ca75940 <SHA1_Internal_Transform>

Here, %rbx contains the pointer to context, and %r12 to context->s1.buffer. The first three instructions swap the 64bit word. (By the way, REVERSE64 macro is written with shifts and bitmasks. Gcc cleverly figures out its intent and replaces the whole expression by a bswap instrucion.) The next three instruction is the calling sequence of SHA1_Internal_Transform.

Wait. There're no instructions emitted to store bitcount into *buf56. I checked the assembly after this but there're no instructions for the store either.

If I insert a dummy external function call before SHA1_Internal_Transform like this:

        /* Set the bit count: */
#if BYTE_ORDER == LITTLE_ENDIAN
        /* Convert FROM host byte order */
        REVERSE64(context->s1.bitcount,context->s1.bitcount);
#endif
        void *buf56 = &context->s1.buffer[56];
        *(sha_word64*)buf56 = context->s1.bitcount;

        puts("foo");

        /* Final transform: */
        SHA1_Internal_Transform(context, (sha_word32*)context->s1.buffer);

Then the storing to *buf56 appears (mov %rdx, 0x58(%rbx)):

   25ca75e9d:   48 8b 53 18             mov    0x18(%rbx),%rdx
   25ca75ea1:   48 0f ca                bswap  %rdx
   25ca75ea4:   48 89 53 18             mov    %rdx,0x18(%rbx)
   25ca75ea8:   48 8d 0d 8f c7 00 00    lea    0xc78f(%rip),%rcx        # 25ca8263e <.rdata+0x9e>
   25ca75eaf:   48 89 53 58             mov    %rdx,0x58(%rbx)
   25ca75eb3:   e8 60 2e 00 00          call   25ca78d18 <puts>
   25ca75eb8:   4c 89 e2                mov    %r12,%rdx
   25ca75ebb:   48 89 d9                mov    %rbx,%rcx
   25ca75ebe:   e8 7d fa ff ff          call   25ca75940 <SHA1_Internal_Transform>

Now, accessing type punned pointer can break the strict aliasing rule. The gcc might have figured the storing into *buf56 had nothing to do with SHA1_Internal_Transform. But I feel there still needs to be a leap that it completely eliminates the store instruction.

Does *(sha_word64*)buf56 = context->s1.bitcount triggers Undefined Behavior? That's why gcc is entitled to remove that code?

Tags: gcc, Optimization, UndefinedBehavior

Saturday, June 4, 2022

Friday, June 3, 2022

Idiomdrottning

Second second

In GNU’s parse-datetime.y, or as it used to be called, getdate.y, the second ordinal is commented out. I guess for English ambiguity reasons.

  { "THIS",     tORDINAL,        0 },
  { "NEXT",     tORDINAL,        1 },
  { "FIRST",    tORDINAL,        1 },
/*{ "SECOND",   tORDINAL,        2 }, */
  { "THIRD",    tORDINAL,        3 },
  { "FOURTH",   tORDINAL,        4 },
  { "FIFTH",    tORDINAL,        5 },

This leads to weird things.

ellen% date --date "first day"
date --date "first day"
Sat 04 Jun 2022 10:29:20 AM CEST
ellen% date --date "second day"
date --date "second day"
Sat 04 Jun 2022 10:29:33 AM CEST
ellen% date --date "third day"
date --date "third day"
Mon 06 Jun 2022 10:29:40 AM CEST
ellen% date --date "fourth day"
date --date "fourth day"
Tue 07 Jun 2022 10:29:48 AM CEST

by Idiomdrottning (sandra.snan@idiomdrottning.org) at Friday, June 3, 2022

Friday, May 20, 2022

Idiomdrottning

Trivial Patents

I opposed and ended a trivial patent once (I found prior art) but the lesson I learned from that is that it’s important to oppose even non-trivial patents. If I identify and stop only the stupid patents, I’m only doing the work of the patent office for them.

In the case of something like “Bionic Reading” (a proprietary system that bolds part of the words) the claimed part is not the implementation, it’s the idea and the research to verify the idea. That’s exactly the difference between patents and copyright. Copyright restricts an implementation (which, for something like code or a movie can be a very non-trivial implementation of a trivial idea. Like, the movie “The Matrix”, the idea is simple (“guy gets chosen to fight in modem land”) but the implementation cost $63 million. Patents restricts an idea (which can be an idea that’s unintuitive and difficult to come up with and verify), but the implementation is like two lines of JS.

It’s patents and patenting as a whole that’s the problem, not just trivial patents.

(I personally would want fewer—maybe half—fixations than they’re using. It’s very frustrating for me to read “bionic reading” text because it forces me to fixate much more than I otherwise would. It feels like getting a slap after every word.)

by Idiomdrottning (sandra.snan@idiomdrottning.org) at Friday, May 20, 2022

Saturday, May 14, 2022

Jeremy Kun

“Practical Math” Preview: Collect Sensitive Survey Responses Privately

This is a draft of a chapter from my in-progress book, Practical Math for Programmers: A Tour of Mathematics in Production Software.

Tip: Determine an aggregate statistic about a sensitive question, when survey respondents do not trust that their responses will be kept secret.

Solution:

import random

def respond_privately(true_answer: bool) -> bool:
    '''Respond to a survey with plausible deniability about your answer.'''
    be_honest = random.random() < 0.5
    random_answer = random.random() < 0.5
    return true_answer if be_honest else random_answer

def aggregate_responses(responses: List[bool]) -> Tuple[float, float]:
    '''Return the estimated fraction of survey respondents that have a truthful
    Yes answer to the survey question.
    '''
    yes_response_count = sum(responses)
    n = len(responses)
    mean = 2 * yes_response_count / n - 0.5
    # Use n-1 when estimating variance, as per Bessel's correction.
    variance = 3 / (4 * (n - 1))
    return (mean, variance)

In the late 1960’s, most abortions were illegal in the United States. Daniel G. Horvitz, a statistician at The Research Triangle Institute in North Carolina and a leader in survey design for social sciences, was tasked with estimating how many women in North Carolina were receiving illegal abortions. The goal was to inform state and federal policymakers about the statistics around abortions, many of which were unreported, even when done legally.

The obstacles were obvious. As Horvitz put it, “a prudent woman would not divulge to a stranger the fact that she was party to a crime for which she could be prosecuted.” [Abernathy70] This resulted in a strong bias in survey responses. Similar issues had plagued surveys of illegal activity of all kinds, including drug abuse and violent crime. Lack of awareness into basic statistics about illegal behavior led to a variety of misconceptions, such as that abortions were not frequently sought out.

Horvitz worked with biostatisticians James Abernathy and Bernard Greenberg to test out a new method to overcome this obstacle, without violating the respondent’s privacy or ability to plausibly deny illegal behavior. The method, called randomized response, was invented by Stanley Warner in 1965, just a few years earlier. [Warner65] Warner’s method was a bit different from what we present in this Tip, but both Warner’s method and the code sample above use the same strategy of adding randomization to the survey.

The mechanism, as presented in the code above, requires respondents to start by flipping a coin. If heads, they answer the sensitive question truthfully. If tails, they flip a second coin to determine how to answer the question—heads resulting in a “yes” answer, tails in a “no” answer. Naturally, the coin flips are private and controlled by the respondent. And so if a respondent answers “Yes” to the question, they may plausibly claim the “Yes” was determined by the coin, preserving their privacy. The figure below describes this process as a diagram.

A branching diagram showing the process a survey respondent takes to record their response.

Another way to describe the outcome is to say that each respondent’s answer is a single bit of information that is flipped with probability 1/4. This is half way between two extremes on the privacy/accuracy tradeoff curve. The first extreme is a “perfectly honest” response, where the bit is never flipped and all information is preserved. The second extreme has the bit flipped with probability 1/2, which is equivalent to ignoring the question and choosing your answer completely at random, losing all information in the aggregate responses. In this perspective, the aggregate survey responses can be thought of as a digital signal, and the privacy mechanism adds noise to that signal.

It remains to determine how to recover the aggregate signal from these noisy responses. In other words, the surveyor cannot know any individual’s true answer, but they can, with some extra work, estimate statistics about the underlying population by correcting for the statistical bias. This is possible because the randomization is well understood. The expected fraction of “Yes” answers can be written as a function of the true fraction of “Yes” answers, and hence the true fraction can be solved for. In this case, where the random coin is fair, that formula is as follows (where \mathbf{P} stands for “the probability of”).

\displaystyle \mathbf{P}(\textup{Yes answer}) = \frac{1}{2} \mathbf{P}(\textup{Truthful yes answer}) + \frac{1}{4}

And so we solve for \mathbf{P}(\textup{Truthful yes answer})

\displaystyle \mathbf{P}(\textup{Truthful yes answer}) = 2 \mathbf{P}(\textup{Yes answer}) - \frac{1}{2}

We can replace the true probability \mathbf{P}(\textup{Yes answer}) above with our fraction of “Yes” responses from the survey, and the result is an estimate \hat{p} of \mathbf{P}(\textup{Truthful yes answer}). This estimate is unbiased, but has additional variance—beyond the usual variance caused by picking a finite random sample from the population of interest—introduced by the randomization mechanism.

With a bit of effort, one can calculate that the variance of the estimate is

\displaystyle \textup{Var}(\hat{p}) = \frac{3}{4n}

And via Chebyshev’s inequality, which bounds the likelihood that an estimator is far away from its expectation, we can craft a confidence interval and determine the needed sample sizes. Specifically, the estimate \hat{p} has additive error at most q with probability at most \textup{Var}(\hat{p}) / q^2. This implies that for a confidence of 1-c, one requires at least n \geq 3 / (4 c q^2) samples. For example, to achieve error 0.01 with 90 percent confidence (c=0.1), one requires 7,500 responses.

Horvitz’s randomization mechanism didn’t use coin flips. Instead they used an opaque box with red or blue colored balls which the respondent, who was in the same room as the surveyor, would shake and privately reveal a random color through a small window facing away from the surveyor. The statistical principle is the same. Horvitz and his associates surveyed the women about their opinions of the privacy protections of this mechanism. When asked whether their friends would answer a direct question about abortion honestly, over 80% either believed their friends would lie, or were unsure. [footnote: A common trick in survey methodology when asking someone if they would be dishonest is to instead ask if their friends would be dishonest. This tends to elicit more honesty, because people are less likely to uphold a false perception of the moral integrity of others, and people also don’t realize that their opinion of their friends correlates with their own personal behavior and attitudes. In other words, liars don’t admit to lying, but they think lying is much more common than it really is.] But 60% were convinced there was no trick involved in the randomization, while 20% were unsure and 20% thought there was a trick. This suggests many people were convinced that Horvitz’s randomization mechanism provided the needed safety guarantees to answer honestly.

Horvitz’s survey was a resounding success, both for randomized response as a method and for measuring abortion prevalence. [Abernathy70] They estimated the abortion rate at about 22 per 100 conceptions, with a distinct racial bias—minorities were twice as likely as whites to receive an abortion. Comparing their findings to a prior nationwide study from 1955—the so-called Arden House estimate—which gave a range of between 200,000 and 1.2 million abortions per year, Horvitz’s team estimated more precisely that there were 699,000 abortions in 1955 in the United States, with a reported standard deviation of about 6,000, less than one percent. For 1967, the year of their study, they estimated 829,000.

Their estimate was referenced widely in the flurry of abortion law and court cases that followed due to a surging public interest in the topic. For example, it is cited in the 1970 California Supreme Court opinion for the case Ballard v. Anderson, which concerned whether a minor needs parental consent to receive an otherwise legal abortion. [Ballard71, Roemer71] It was also cited in amici curiae briefs submitted to the United States Supreme Court in 1971 for Roe v. Wade, the famous case that invalidated most U.S. laws making abortion illegal. One such brief was filed jointly by the country’s leading women’s rights organizations like the National Organization for Women. Citing Horvitz for this paragraph, it wrote, [Womens71]

While the realities of law enforcement, social and public health problems posed by abortion laws have been openly discussed […] only within a period of not more than the last ten years, one fact appears undeniable, although unverifiable statistically. There are at least one million illegal abortions in the United States each year. Indeed, studies indicate that, if the local law still has qualifying requirements, the relaxation in the law has not diminished to any substantial extent the numbers in which women procure illegal abortions.

It’s unclear how the authors got this one million number (Horvitz’s estimate was 20% less for 1967), nor what they meant by “unverifiable statistically.” It may have been a misinterpretation of the randomized response technique. In any event, randomized response played a crucial role in providing a foundation for political debate.

Despite Horvitz’s success, and decades of additional research on crime, drug use, and other sensitive topics, randomized response mechanisms have been applied poorly. In some cases, the desired randomization is inextricably complex, such as when requiring a continuous random number. In these cases, a manual randomization mechanism is too complex for a respondent to use accurately. Trying to use software-assisted devices can help, but can also produce mistrust in the interviewee. See [Rueda16] for additional discussion of these pitfalls and what software packages exist for assisting in using randomized response. See [Fox16] for an analysis of the statistical differences between the variety of methods used between 1970 and 2010.

In other contexts, analogues to randomized response may not elicit the intended effect. In the 1950’s, Utah used death by firing squad as capital punishment. To avoid a guilty conscience of the shooters, one of five marksmen was randomly given a blank, providing him some plausible deniability that he knew he had delivered the killing shot. However, this approach failed on two counts. First, once a shot was fired the marksman could tell whether the bullet was real based on the recoil. Second, a 20% chance of a blank was not enough to dissuade a guilty marksman from purposely missing. In the 1951 execution of Elisio Mares, all four real bullets missed the condemned man’s heart, hitting his chest, stomach, and hip. He died, but it was neither painless nor instant.

Of many lessons one might draw from the botched execution, one is that randomization mechanisms must take into account both the psychology of the participants as well as the severity of a failed outcome.

References

@book{Fox16,
  title = {{Randomized Response and Related Methods: Surveying Sensitive Data}},
  author = {James Alan Fox},
  edition = {2nd},
  year = {2016},
  doi = {10.4135/9781506300122},
}

@article{Abernathy70,
  author = {Abernathy, James R. and Greenberg, Bernard G. and Horvitz, Daniel G.
            },
  title = {{Estimates of induced abortion in urban North Carolina}},
  journal = {Demography},
  volume = {7},
  number = {1},
  pages = {19-29},
  year = {1970},
  month = {02},
  issn = {0070-3370},
  doi = {10.2307/2060019},
  url = {https://doi.org/10.2307/2060019},
}

@article{Warner65,
  author = {Stanley L. Warner},
  journal = {Journal of the American Statistical Association},
  number = {309},
  pages = {63--69},
  publisher = {{American Statistical Association, Taylor \& Francis, Ltd.}},
  title = {Randomized Response: A Survey Technique for Eliminating Evasive
           Answer Bias},
  volume = {60},
  year = {1965},
}

@article{Ballard71,
  title = {{Ballard v. Anderson}},
  journal = {California Supreme Court L.A. 29834},
  year = {1971},
  url = {https://caselaw.findlaw.com/ca-supreme-court/1826726.html},
}

@misc{Womens71,
  title = {{Motion for Leave to File Brief Amici Curiae on Behalf of Women’s
           Organizations and Named Women in Support of Appellants in Each Case,
           and Brief Amici Curiae.}},
  booktitle = {{Appellate Briefs for the case of Roe v. Wade}},
  number = {WL 128048},
  year = {1971},
  publisher = {Supreme Court of the United States},
}

@article{Roemer71,
  author = {R. Roemer},
  journal = {Am J Public Health},
  pages = {500--509},
  title = {Abortion law reform and repeal: legislative and judicial developments
           },
  volume = {61},
  number = {3},
  year = {1971},
}

@incollection{Rueda16,
  title = {Chapter 10 - Software for Randomized Response Techniques},
  editor = {Arijit Chaudhuri and Tasos C. Christofides and C.R. Rao},
  series = {Handbook of Statistics},
  publisher = {Elsevier},
  volume = {34},
  pages = {155-167},
  year = {2016},
  booktitle = {Data Gathering, Analysis and Protection of Privacy Through
               Randomized Response Techniques: Qualitative and Quantitative Human
               Traits},
  doi = {https://doi.org/10.1016/bs.host.2016.01.009},
  author = {M. Rueda and B. Cobo and A. Arcos and R. Arnab},
}

by j2kun at Saturday, May 14, 2022

Saturday, April 30, 2022

The Racket Blog

Racket v8.5

Racket version 8.5 is now available from https://download.racket-lang.org.

As of this release:

  • Racket’s new -y flag automatically keeps compiled files up to date, reducing subsequent load times.

  • Error-message realms allow Racket-hosted languages to adapt and rewrite error messages to make sense in a particular context.

  • Nonprivileged users can control package installation scope using an “other-version” directory in the addon-dir.

  • Racket CS runs on platforms where native-code generation is not currently supported (e.g., s390x or ppc64). See “README.txt” in the source distribution for more information on the —enable-pb flag to configure.

  • DrRacket’s new ‘Reopen Closed Tab’ file menu item will open previously closed tabs.

  • Typed Racket has support for the xml library; use typed/xml.

  • Rackunit reports source locations for failed test cases in the Typed Racket language.

  • Plot has violin plots and improved box-and-whisker plots.

  • Boxes are supported alongside lists, vectors etc. in place-channel messages.

  • Those who manually configure Racket CS to use Zlib compression for compiled code should be aware of CVE–2018–25032; the next release and the current snapshot builds use a newer, safer version of zlib.

  • The release includes many other repairs and changes!

The following people contributed to this release:

Alex Harsányi, Alexander Shopov, Alexis King, Andrew Mauer-Oats, Ben Greenman, Benedek Szilvasy, Bert De Ketelaere, Bogdan Popa, Cameron Moy, Chung-chieh Shan, Fred Fu, Gustavo Massaccesi, J. Ryan Stinnett, Jamie Taylor, Joel Dueck, John Clements, Joseph Griego, Khadija Sidhpuri, Laurent Orseau, Maciej Barć, Matthew Flatt, Matthias Felleisen, Mike Sperber, Noah Ma, Philip McGrath, Robby Findler, Ryan Culpepper, Sam Tobin-Hochstadt, Sorawee Porncharoenwase, Stephen De Gabrielle, Tim Jervis, and Trevor Paley

Link to package regressions issue for the 8.5 release: https://github.com/racket/racket/issues/4202

by The Unknown Author at Saturday, April 30, 2022

Wednesday, April 27, 2022

Idiomdrottning

preserve

preserve is a combinator that caches the result of a procedure for a given number of seconds.

Example

(define slow-plus (preserve 5 +))

(list (slow-plus 1 2) (slow-plus 3 4))

This returns (3 3). Calling (slow-plus 3 4) five seconds later returns 7.

Code

sudo chicken-install preserve

For a repo,

git clone https://idiomdrottning.org/preserve

by Idiomdrottning (sandra.snan@idiomdrottning.org) at Wednesday, April 27, 2022

Sunday, April 24, 2022

Idiomdrottning

You can tell I really really wanna complain about it but I’m not gonna

There is this popular CSS framework that I wanna complain about (not attack or contact them, of course, just kvetch about it here) but I simultaneously feel like I don’t wanna, because they don’t break any web standards and there are no accessibility issues (unlike SPA frameworks). It’s a little bandwidth-wasteful but I don’t really care, that’s fine.

The framework I have in mind is only bad (just a really bad and self-foot-shootingly way to make web) for the devs that use it, and if they can stomach it, it doesn’t hurt anyone else. I mean, if you use it you are fucking up royal and there are probably a lot of things wrong with your entire pipeline and it’s a sign that something has gone seriously wrong with how we teach web and make tools for web, but, those things only hurt yourself.

So I would only be tearing down someone else’s work (the creators of this particular framework) from my own armchair, which I feel is only legit when there’s a standards and/or accessibility and/or political issue. My instincts are so often to tear down bad things when it’s so much better to build up good things.

If I want good CSS, I should instead lead by example, write good CSS, make tools for writing good CSS, and write about how to write good CSS, instead of just tearing into a bad CSS framework without having good alternatives readily available.

In other words, this is a rare case of the it’s no big deal if someone is wrong on the Internet xkcd actually applies because while they are very, painfully wrong, it doesn’t really hurt me or anyone else except sympathetically, so I should just let it be.

The analogy is seeing someone build a bicycle painstakingly with a toothpick instead of with good tools. If the bike still works, all else is moot.

I could be warning my dev friends about the framework but if they’ve painted themselves into a corner so deep that they’re actually considering it, I don’t wanna restrict their options further with armchair advice. Again, what I should be doing is STFU and write constructive posts instead of this, and the same goes for my previous post on it.

by Idiomdrottning (sandra.snan@idiomdrottning.org) at Sunday, April 24, 2022

Monday, April 18, 2022

GNU Guix

10 years of stories behind Guix

It’s been ten years today since the very first commit to what was already called Guix—the unimaginative name is a homage to Guile and Nix, which Guix started by blending together. On April 18th, 2012, there was very little to see and no actual “project”. The project formed in the following months and became a collective adventure around a shared vision.

Ten years later, it’s amazing to see what more than 600 people achieved, with 94K commits, countless hours of translation, system administration, web design work, and no less than 175 blog posts to share our enthusiasm at each major milestone. It’s been quite a ride!

What follows is a series of personal accounts by some of the contributors who offered their time and energy and made it all possible. Read their stories and perhaps you too will be inspired to join all the nice folks on this journey?

10 year anniversary artwork

Alice Brenon

As a conclusion, Guix is a really neat system and I hope you enjoy it as much as I do!

My story with Guix is a bit topsy-turvy so I thought I might as well start by the end :) I first ranked it last among systems I wanted to test, then was a bit puzzled by it when I had the chance to test it after all the others had disappointed me, and we finally got to know each other once I installed it on my work laptop, because when you need a good, stable system you know you can rely on, why not use the most surprising one? Strangely, the alchemy worked and it has never let me down so far.

Like all good computer things, it looked way scarier from a distance than it really was, and seemed to be very much about ethics and theory while it's actually very pragmatic. I had struggled for years with the myriad of incompatible package formats for systems, then for each specific languages, and was flushed to discover at last what seemed to be a reasonable universal format. That's probably what I like best about it: the ability to use potentially any software I want without trashing my system. The welcoming community eager to help and valuing my contributions made it even better, and submitting patches came naturally. I mostly use it for development, and to keep my sanity in spite of all the data science tools I have to use for work. I sometimes wish it were easier to tweak the core of the system, but I blame my lack of free time at least as much as its design. I would absolutely love to see my Guix system using the runit init system one day but it just works and I finally learnt that it was all that mattered if you wanted to get things done in the end.

Andreas Enge

When I think of Guix, I always kid myself into believing that I had the idea — I remember chatting with Ludovic around a GNU hackers' meeting about Nix; I joked that since Guile is the GNU language, Nix should be rewritten in Guile. But it turned out that Ludovic had already started such a project in earnest... Luckily there is the git history to refresh our memories. Apparently I installed Guix, at the time only the package manager, with Ludovic's help in December 2012, and immediately reported a few bugs. My next action was to update the package for my own software. Learning Scheme was quite difficult, but I fondly remember discussing quotes and quasiquotes with Ludovic. After that, I mostly added packages to Guix, which was possible without knowing much of functional programming; the most tricky packages that stayed in my mind were ImageMagick and TeX Live. I came to appreciate the GNU Autotools — with all their shortcomings, having a uniform (and usually reliable) way of compiling and installing a software makes creating a Guix package almost trivial.

The most compelling feature of Guix was (and still is, I think) the ability to roll back package installations, and now complete system installations — no more fear of updating a package to a non-working state! And on a completely different level, the nice and welcoming atmosphere in the community, in no small part thanks to Ludovic's initial efforts of creating an inclusive environment.

Many formidable adventures are attached to working on Guix. Buying our first server for the build farm was difficult, since we wanted to use a machine with Libreboot that would work well with the GNU system. Eventually we succeeded, and it is still hosted with the non-profit Aquilenet in Bordeaux, so we managed to start our own infrastructure in accordance with our values.

Writing the bylaws of the Guix Europe non-profit was another exciting adventure; again we tried to create a structure in line with our values, where the decisions were taken as collectively as possible.

And personally I have fond memories of taking part in Guix meetings at FOSDEM and co-organising the Guix Days in Brussels; these are such nice occasions to meet people passionate about Guix and free software! I will never forget the Guix Days 2020 when I had just returned from a Reproducible Build Summit where I admired their way of facilitating the workshop, which I then tried to copy for our own meeting.

The Guix system meets all my daily needs now, so I have no technical wishes for the future — but I trust the many creative minds working on advancing the project to come up with nice new ideas. And I wish the human adventure and community building around Guix to continue!

Andrew Tropin

It's all lisp, no distraction, pure consistency! Every few years I migrate to a different workhorse and it has always been a pain to bring my software setup with me: forgot a pam/udev rule here, package here and some tiny hack here and everything is messed up, easier to reinstall everything from ground up. With declarative and reproducible nature of Guix System, Guix Home and rde project it's just a pure pleasure: write the configuration once, use it everywhere! Daemon configurations, package build phases, cron tasks, everything described in Guile Scheme. The one language to rule them all! I look forward for a little more: wider adoption of Guix for managing development environments and infrastructures.

GNU Guix respects you and your freedoms: anyone can explore and study, tweak and adjust, and share everything they want, every program available. Moreover every package is bootstrapped, what a miracle! Yes, some hardware isn't supported, but for a good reason, for this price you get a lot. I look forward for a little more: decentralized substitutes and RISC-V laptops running GNU Guix.

Thank you the community for all the hard work, already enjoy Guix for more than an year and look forward for more exciting things to appear!

Arun Isaac

I was introduced to Guix in 2016. That was around the time I was learning lisp and having my mind blown. As soon as I heard that Guix was written in a lisp and that it was a FSDG compliant distro, I knew that this was going to be the best distro ever and immediately jumped ship.

While I was immediately smitten by the perfection and elegance of Guix, I have stayed this long not for the technical excellence but for the people. The Guix community is the warmest and most caring free software community I have ever been a part of. I honestly did not believe such people could exist online until I saw Guix. In the future, I would love for this community to grow and thrive with a smoother issue tracking and contribution process so that potential contributors, especially newcomers, don't turn away frustrated.

Björn Höfling

In 2016, I searched a GNU/Linux distribution for a friend's laptop and chose GuixSD (now Guix System), inspired by a LibrePlanet video from MediaGoblin. The laptop failed to digest this OS, as it demanded binary, non-free drivers. In contrast, my wetware is 100% GNU FSDG-compatible, and so, generation 1 of Guix burned successfully into my brain, without any headaches. Or at least, they went away with the help from the very friendly, supportive, tolerant (only to the tolerant, thanks to the code of conduct) community.

My contributions started with not understanding Guix, and asking stupid questions on the mailing lists. Sometimes during that process I found bugs and analyzed them further, which helped Ludovic and other developers to fix them. I reviewed mostly Java packages and added some on my own. I very much enjoyed co-mentoring for Outreachy, and learned more on automated video/screencast generation. I should be really more active in the community again!

For the future of Guix, I would like to see more Java and Go packages (hem, this comes not from alone, I should review and contribute more). Internally I wish a more intuitive bug- and patch-tracker instead of Debbugs+Mumi. Externally I wish a much bigger awareness in the commercial "Open Source" world about software freedom, bootstrapping your environment and dependencies from free source code in a real reproducible way, instead of relying on opaque, binary containers. I wish people would much more take care of their dependencies (with Guix, of course!), put much more thoughts in their usage of dependencies, break-up dependency hell and support third parties in building their sources with free software (instead of relying on binary dependencies and opaque containerized dev-environments).

Blake Shaw

New media artists and designers suffer from following dilemma: our work, with its primary medium being code, is at once perhaps the simplest medium to distribute — requiring little more than copying a directory of text files from one hard drive to another — yet the works themselves remain a total nightmare to faithfully reproduce across varying machines at different points in time. Among other reasons, this is because our works are often composed of disparate parts with accompanying technical debt: an audio-visual installation may use the C++ library openFrameworks for high-performance interactive graphics, Haskell's TidalCycles for realtime sequencing, the fantastic FAUST signal processing language for zero-delay audio DSP, plus the usual dependencies; openCV, libfreenect, cairo, gstreamer, ffmpeg and so on. Time and its corresponding ABI changes intensify our predicament; not only is it often an error-prone and laborious task to get all of this running correctly across many machines, the nature of technical debt means that getting an installation from 2014 up and running in 2022 is often more trouble than its worth. Sadly, these seemingly immaterial works of art that are the simplest to copy are simultaneously some of the most difficult to reproduce and the quickest to depreciate.

Guix, on the other hand, offers its users provenance-preserving bit-reproducible builds of their entire operating systems: using Guix's implementation of the functional software deployment model, I should be able to reproduce, bit-for-bit, the exact same results across equivalent hardware. Suddenly our artworks can be deterministically produced not only at the level of the source code's changelog but also as the level of the build, offering the guarantee that our usually glued-together towers of systems that power our installations can be revisited and reproduced in the future, and the deployment processes becomes as simple as packing your system into a container to be deployed to a remote target. These guarantees means that the scaling of our works become simplified: if you want to do an installation that involves 100 Raspberry Pis communicating with one another in a dynamic system, you can focus on working on just a small parameterized subset, and then generate their varying configurations using the infrastructure that Guix provides. I'm currently in the early phases of developing re::producer, a "creative plumber's toolkit" that seeks to simplify this process for artists, tailoring the tools provided by Guix to the domain-specific needs of media art and allowing artists to declaratively define complex media art systems using one of the greatest programming language of all time, Scheme.

This is still new terrain, so there is plenty of work to do. But that’s no excuse to keep up with your old habits, so roll up your sleeves and come hack the good hack!

Cyril Roelandt

Back in early 2013, I was lucky enough to be unemployed for a few months. This gave me a lot of time to try out GNU Guix. Ludovic had told me about it a few months earlier, while we were still working at the same place. It was so easy to add new packages that I naturally ended up submitting a few patches and quickly trolled the whole project by adding my editor of choice, vim. Debugging package definitions was also very simple since the builds were reproducible by default.

I also had some fun writing the first version of the linter and improving the importer/updater for Python packages. I even hacked tox to make it use Guix instead of virtualenv and gave a talk about this at FOSDEM. Even though I left the project a few years ago, I'm glad to see it's doing well and is used in science and has joined forces with Software Heritage.

Efraim Flashner

Back in 2015 or so I had been using GNU/Linux on the desktop for a number of years and I wanted to contribute somehow. I had just finished a course in University using Lisp and Prolog and then I heard about Guix having its 0.8.3 (or so) release and it looked like something that I could try to contribute to. I certainly made a number of mistakes in the beginning; I didn't know that git revert was an actual command and I tried to revert a commit by hand, leaving a dangling parenthesis and breaking the repo. Another time I added Java as a dependency to an image library and broke the graphics stack for half the architectures until I reverted that! I even had a stint as a failed GSoC student. I was working on Bournish, a Gash/Gash-utils like utility to make debugging in the early boot process far easier by providing common CLI utilities. I had some issues with time management and ended up spending more time than I should have updating packages in the repository, as a result I didn't spend enough time working on Bournish and it's languished since then.

Currently, I enjoy working on troublesome packages and expanding the number of packages available on non-popular architectures. Sometimes it's removing compiler flags or ‘ifdef gating’ architecture-specific includes and other times certain parts of programs need to be disabled. Then everything needs to be double-checked for cross-compiling. Right now I'm working on riscv64-linux support in Guix, it has a lot of potential but powerful boards are hard to come by. Also there are some lingering bugs with guix show showing different supported-systems for packages depending on which architecture you run it from; on x86_64-linux only two are shown, from aarch64-linux all 9 architectures are shown.

Ekaitz Zarraga

A friend of mine introduced me to Nix and Guix a while ago but I was hesitant to try it because I hate configuring stuff and this didn't look as an easy distribution to use. Once I discovered we could have separate environments and how easy is to write a package (despite all other difficulties Guix has) I was completely into it. I installed Guix in my laptop and never looked back. In the 10 years I've been using a GNU/Linux distribution I never interacted that directly with my packages: creating custom ones, sending them upstream, making fixes… That's also freedom! Now, a couple of years later, I am working on improving the bootstrap process for RISC-V and using Guix as a mechanism that provides reproducible builds and an easy way to manage all the actors I have to deal with: very old software with old dependencies, colliding libraries, environment variables, custom patches in source code… This would be a pain to build in any other environment, but Guix makes hard things easy. Guix also makes easy things hard sometimes, but we are working on that!

Eric Bavier

As a young(ish) computer programmer, I had been running GNU/Linux systems for about 7 years but wanted to find a project I could contribute back to. Fortunately, I came upon a release announcement for Guix after having poked around the GNU Hurd and Guile spheres. To me at the time Guix had the exact mix of upstart energy, optimism, and long-term vision that I was hoping to find. Over the years I've been able to contribute packages I use in both my personal and work lives, and I'm proud to have implemented the first version of guix refresh --list-dependents. I've really loved how Guix allows me to easily move my environments around to different systems, and "rollback" gives me much peace of mind knowing that I can tinker with the system and recover should something go wrong. But probably my favorite part of Guix is the fantastic community I've seen grow around the project. It exemplifies the sort of caring, kind, supportive group I wish many other projects had. Together I know we'll be able to make advances on many fronts. In particular, I'd like to see further work on peer-to-peer substitutes delivery, a native build daemon, additional tools for managing relocatable pack collections, and continued leadership in bootstrapping.

Florian Pelz (pelzflorian)

GNU Guix to me is a group that cares about its software and about the people involved. I’ve got to know Guix by reading discussions on how to do sandboxing properly. But actually, Guix convinced me with its clarity and approachability and principles. Guix opened my eyes on how parts of GNU fit together. Thanks to all who give us this freedom to understand and decide. By contributing to the translation, I hope to make it reach more users and developers.

Guillaume Le Vaillant

Before I started using Guix in 2019, I was using Gentoo because I liked how I could easily package software and make package variants with some custom patches. However one day an upgrade didn't go well and many packages ended in a bad state. I realized I would have to reinstall the whole system to get things to work again. Before recompiling the whole system, I tried Nix and Guix, because I had read somewhere that they used functional package management, which gives the possibility to roll back to a working state when an upgrade causes problems. I chose Guix because I thought it was going in the right direction by using only free software and trying to get reproducible builds. The fact that package definitions use the Scheme language was a bonus point as I like Lisp languages. And there was even a build system for Common Lisp packages, which is rarely the case in the GNU/Linux distributions I tried over time. So I started using Guix, and packaging software I wanted that were not in Guix yet. One day someone asked me if I would be interested in having commit access, and I accepted. I also found a way to improve the build system for Common Lisp packages that simplified package definitions. In the future, I think it would be nice to add an importer fetching information from Quicklisp, as it would make packaging Common Lisp software even easier.

Hartmut Goebel

Christian Grothoff (GNU Taler) pointed me to Guix early 2016, saying “This will become the new Debian!” and asking me to look at it for GNU Taler. Well, quickly I was attracted by the ideas of reproducible build and the ease of packaging software. I also love the one-time usage of programs without littering my system.

Curiously, even as I'm a Python developer, my first contributions have been about Java packaging. And I spend quite some time trying to build maven. This challenge I gave up after two (or three? can't remember) attempts. Glad Julien Lepiller continued the endeavor and created the Maven build system.

Nowadays I still use Guix on a foreign distro only, as KDE desktop and some of my main applications are still not here. Guix keeps my main system tidy, while I can have different development environments without dependency conflicts.

As you can imagine, I'd like to see KDE desktop in Guix as well as some guix compose for managing compound containers.

Jan (janneke) Nieuwenhuizen

At FOSDEM 2016 there were seven talks about GNU Guix: A talk about the Hurd by Manolis Ragkousis, about functional package management by Ricardo Wurmus and that was just what I needed to hear: Finally a viable promise for the GNU System and much more innovative than I could have hoped for. At the time I also worked on a project where building binary releases was becoming more unmanageable with every release because of conflicting requirements. We were slowly migrating away from C++ to GNU Guile, so while not directly applicable the talk “Your distro is a Scheme library” by Ludovic Courtès also made me feel: Go Guix!

Using Guix, my intricate dependency problems building binary packages quickly and easily disappeared. That gave me the confidence that I needed and I wanted to get involved. My first contributions where a programming game called Laby and its dependencies and a few more packages that I missed. After running Guix on top of Debian GNU/Linux for three months I switched to what we now call Guix System. Guix did not have log rotation yet in those days, so I created a package.

This is how I found how amazingly helpful and friendly the community was. I created the MinGW cross build for Guile 2.0 and then "found out" about the bootstrap binaries: The only packages in Guix that are not built from source. Something just did not feel right. The manual said: “These big chunks of binary code are practically non-auditable which breaks the source to binary transparency that we get in the rest of the package dependency graph.” So, I wrote GNU Mes and started working on solving this problem. Twice we halved the size of the bootstrap binaries and the work is still ongoing.

What possibly started somewhat as a April fools joke in 2020 about the Hurd—this is still unclear—was (mis?)taken by some as a real project and led to a fun hacking frenzy of several months finally producing the "Childhurd": A Guix Shepherd service that gives access to the GNU/Hurd in a VM. My wish for the near future would be see an up-to-date Hurd including the Debian rumpkernel patches that may finally enable running the Hurd on real hardware again.

John Kehayias

All I wanted to do was to try out a new status bar, but the author only officially supported Nix for building. That started me to finally look at Nix after hearing about it in passing before. I was intrigued by the focus on reproducible and declarative builds. The language, not so much. My brother mentioned another project in the same vein but built on a Lisp. As a lover of all things Lisp, that was basically enough for me to dive right in. Beyond the banner features of the powerful package and system management, reproducible builds, system configuration, and, of course, Guile, I quickly found perhaps the biggest and most important: the GNU Guix community. They have been nothing short of amazing: helpful, intelligent, supportive, and fun to hang out with on the #guix channel. In less than a year, my journey so far has taken me through the (is it infamous yet?) recent big core-updates branch and merge, submitting patches for random libraries and key desktop features I use, and participating in the motivating Guix Days 2022. Looking to the future, I hope we can better harness the energy and resources of the growing Guix community. It is already a great project to get involved with and make your own, but with better and quicker patch review, further building out and using our development tools and ecosystem, and continuing to smooth out the rough edges for new users/contributors, I'm sure the next 10 years of GNU Guix will be very bright indeed.

Konrad Hinsen

In my work as a computational scientist, my first encounter with reproducibility issues happened in 1995, when a quantum chemistry package produced different results on two almost identical Silicon Graphics workstations. This was the beginning of a long quest for better computational reproducibility, in the course of which I discovered in 2014 Nix and Guix as two implementations of the same promising idea: the fully automated construction of a complete reproducible software stack. Of the two, Guix was more aligned with my lispy past, and already had a burgeoning computational science user community. I started playing with Guix in 2016, in a virtual machine under macOS, but only fully adopted Guix for my everyday work in 2021, when I left macOS for Linux. During those five years, I also learned to appreciate the Guix community, which is friendly, competent, and refreshingly low-ceremony in spite of continuous growth. That makes for an easy transition from newbie to contributor (mostly contributing packages, but also the time-machine command that matters for reproducibility). The anniversary is a good occasion to express my thanks to all those who answered my many questions, ranging from conceptual to technical, and to the maintainer team that does an important but not very visible work by critically examining all submitted packages and code enhancements. My main wish for the future is a lower barrier to adoption for my colleagues in computational science, and I hope to contribute to making this happen.

Lars-Dominik Braun

Around the end of 2019 we were looking for a way to provide reproducible software environments to researchers in(?) psychology and I was researching software to accomplish that. Binder/repo2docker was the obvious and most mature solution at that time and a colleague of mine had set up a proof of concept server already. But it could only handle public projects out-of-the-box and setting up an entire Kubernetes cluster didn’t seem particularly appealing at that time, because no other project was moving in that direction yet. So I set out to look for alternatives. Another idea was based around OpenStack and one virtual machine per project with preinstalled software, which we would keep for eternity. Also not ideal and OpenStack is very hard to master too. So I looked further at Nix, which – at that time – lacked an obvious way to spawn ad-hoc environments with a certain set of packages. Thankfully I stumbled upon GNU Guix by mere accident, which had exactly that feature. And so in December 2019 my first code contribution was merged.

Prior to that I had never written a single line of Scheme or Lisp and even now it’s still a steep hill. GNU Guix still powers our project and allows us to easily share software environments while providing excellent application startup times. I also started contributing software that I run on my own machines, but I’m not running Guix System, because compared to systemd, Shepherd is quite limited on the desktop and Guix’ lack of first-class support for non-free drivers/firmware, which I need to even boot my machine.

Ludovic Courtès

It all started as a geeky itch-scratching experiment: a tiny bit of Guile code to make remote procedure calls (RPCs) to the Nix build daemon. Why? As I was involved in and excited about Guile and Nix, it felt natural to try and bridge them. Guile had just had its 2.0.0 release, which broadened its scope, and I wanted to take advantage of it. Whether to go beyond the mere experiment is a decision I made sometime after a presentation at the 2012 GNU Hackers Meeting.

It was far from obvious that this would lead us anywhere—did the world really need another package manager? The decisive turn of event, for me, was to see that, at the time Guix officially became part of GNU in November 2012, it had already become a group effort; there was, it seems, a shared vision of why such a crazy-looking project made sense not just technically but also socially—for GNU, for user freedom. I remember Nikita Karetnikov as the first heroic contributor at a time when Guix could barely install packages.

One of my “ah ha!” moments was when I built the first bootable image a year later. G-expressions, the service framework, and checkout authentication are among my favorite hacks. What’s mind-blowing to me though is what others have achieved over the years: the incredible bootstrapping work, Disarchive, Emacs-Guix, the installer, Hurd support, Guix Home, supporting tools like Cuirass, the Data Service, and mumi. There’s also the less visible but crucial work: Outreachy and GSoC mentoring, support on IRC and the mailing lists, build farm administration, translation, dealing with the occasional incident on communication channels, organizing events such as the Guix Days or FOSDEM, and more.

As much as I love hacking the good hack, I think Guix’s main asset is its community: a friendly, productive, and creative group with a sense of attention to the other. I started clueless about what it means “to build a community” and learned a lot from everyone met on the way. We did it, we built this! Thumbs up, Guix!

Luis Felipe

When I found that Guix existed, I saw it could make it easier for GNU to release its Operating System and reach a wider audience. I intended to propose some graphic designs related to this, and sent a message to GNU in order to test the waters. Things didn't go as I expected, so, instead, I decided to direct my contributions towards GNU Guix and its distribution of GNU.

Since then, I've contributed with graphics (Guix and Guile logos and website designs, desktop backgrounds, release and promotional artwork), testing, bug reporting, packaging, and Spanish translations.

It's been about 8 years of Guix for me (the heck!). I started using the package manager on Debian, gradually switched the provenance of my software from Debian to Guix, and, once GNOME became available, I moved to Guix’s distribution of the GNU operating system, which I've been using as my main system for about 3 years now (and I don't see that changing anytime soon).

Right now, I'm enjoying developing software using Guix's reproducible environments and containers, and using one single package manager for every dependency.

I hope this system reaches a wider audience and brings science to home computing along the way. Homes should be capable of producing scientific work too.

Manolis Ragkousis

When I think how I started with Guix, I use one word to describe it, luck! It was early 2014 when I encountered Guix by luck, while I was still a student at Crete, Greece. I remember there was a strike during that time and I had plenty of free time for a week, so I decided that I will try to start working on this. Then an idea came in mind, why not try porting Guix to GNU/Hurd and build a system with it? One thing led to another and it also became a GSoC project in 2015 and 2016. In 2016 I also gave a FOSDEM talk about this, which somehow ended up being the start of me helping out with the GNU Guile devroom in 2017 and 2018, and then what became the Minimalistic Languages until today. When I am thinking about Guix is like thinking about the story of me growing up and the people I met through all these years I consider family! Guix is a big part of my life, I use it everywhere and even though I am not able to help much nowadays I am following the project as much as I can. Here's hoping to another 10 year!

Marius Bakke

I originally got interested in Guix after facing shortcomings in traditional configuration management tools. A fully declarative and immutable system which cleans out old user accounts and packages, that also offers reproducibility, rollbacks, and the ability to generate virtual machines and containers from the same code. Where do I sign up?

It turns out, signing up was easy, and I soon found myself contributing the pieces I needed to make it a daily driver. Watching the community grow from a handful of contributors to 100 monthly has been astonishing. I have learned a lot from this community and am proud to be a part of it. Can't wait to see what the next decade brings. Happy birthday Guix!

Mathieu Othacehe

I was introduced to GNU Guix by a colleague, Clément Lassieur in 2016. At first I found the concept overwhelming. Writing Guile wrappers for each and every Linux service out there and keeping them up to date seemed like impossible. However, I quickly fell in love with the package manager, the distribution and the community behind. A few months later, GNU Guix was running on all my machines and I started hacking on the continuous integration tool: Cuirass.

Since then GNU Guix has been an important part of my life. I wrote most of the Guix System installer while traveling by bike to China in 2018. During the 2020 lockdown, I worked with janneke on the new image API and the Hurd port. At that time, I was proposed a co-maintainer position of the project. In 2021, thanks to an NGI sponsorship, I dedicated 6 months to improving our continuous integration process and overall substitutes coverage.

Recently it has been harder to dedicate as much efforts on the project but I'm sure this is a transient phase. I can't wait to start working again with the incredibly talented people making this piece of software so special to me.

Paul Garlick

I began using and contributing to the Guix project in 2016. I had been searching for a way to preserve software environments that are used for numerical simulation. The applications that run in these environments often comprise a combination of specialised code and building blocks drawn from an underlying framework. There are many moving parts and changes to a low-level library can block the operation of the high- level application. How much better things would be if one could specify the exact components of the environment and re-create it whenever it is needed. I discovered that Guix provides the machinery to do just that. Scheme was new to me then so I had some learning to do before contributing. This included a detour via Vonnegut/Cat's Cradle, of course, to discover the meaning of ice-9. Suitably informed I returned to add a number of finite volume and finite element frameworks to the Guix package collection. Keeping these packages up- to-date and welcoming new simulation-related packages is the next target. Looking ahead to the next ten years, an important task is to popularise the use of the Guix tools. Many more engineers and scientists stand to benefit from the use of the dependable software environments that are now made possible.

Ricardo Wurmus

In 2014 I became responsible for building and installing scientific software at the Max Delbrück Centre, a research institute in Berlin. We used CentOS, so I built custom RPMs, installing applications to ad-hoc prefix directories. After a few weeks I took a minute to consider the horrific implications of maintaining a growing collection of custom software with RPM. As I tried to remember what life choices had led me to this moment, I recalled an announcement email of a quirky GNU package manager written in Scheme. A short web search later I was playing around with Guix.

After an encouraging chat on IRC I realized that I could probably replace our custom RPM repository and build different variants of scientific software on much more solid ground—all the while contributing to a project that felt like a new and exciting take on GNU. We're building the GNU system!

Guix only had very few of the packages I needed, so I got busy. I packaged and bootstrapped the JDK because I was under the mistaken assumption that I would need it for R (turns out Java is optional). Many more foolish adventures followed, and some of them have actually been useful for others.

I had found my tribe of fellow hackers who cared about the vision of the GNU system, encouraged playful experimentation, and were rooting for each other to succeed in building a better system that made software freedom a practical reality, blurring the lines between developers and users. In the decades to come I hope many more people will get to experience what I did and end up calling this community their home.

Simon Tournier

Back in 2014, I watched the video “Growing a GNU with Guix” at FOSDEM but the real revelation had been in 2015 with “GNU Guix: The Emacs of Distros”, again at FOSDEM. Then, I was following the development but not using Guix yet. 2016, new job where I was spending my time to fight against dependencies and Modulefiles. Then I have totally jumped in Guix in December 2018. My first interaction with the project — and not yet running Guix — was a in-person event in Paris before the Reproducible Builds workshop. Back to home, I proofread cover to cover the French manual — my first contribution — and installed Guix on the top of my Debian GNU/Linux system. So amazing! Guix fixes many issues I had at work — and introduce new ones^W challenges. Plus, thanks to people around, I am learning a lot, both about technical details and about inter-personal interactions. My wish for the near future is a community more structured: more events and meetups, more process for smoothing the contributions (“teams” for improving the reviewing by sharing the load, RFC for discussing new features, regular releases, etc.), and more materials for using Guix in various configurations.

In scientific context, transparency — being able to audit the whole computational environment from the source codes to the production of binaries — is one of the keys for a true reproducible research. Since Guix is transparent by design, it appears to me one part for a solution in tackling the computational side of the replication crisis. For the near future, I wish more scientific practitioners will employ Guix.

Thiago Jung Bauermann

I learned about Guix when I was looking for alternative, safe ways of installing an up-to-date Rust toolchain on my machine (at the time rustup didn't verify signatures of downloaded binaries, and it still doesn't do the full job). Guix is a great way to have the latest and greatest software on top of your slower-moving Linux distribution. I love how easy it makes to create instant, ad hoc environments with the packages you need for a specific task. Or to temporarily try out some new app or tool, leaving Guix to garbage-collect it and its dependencies. The Guix community is amazing as well! It's a pleasure to participate on the mailing lists. And I've been enjoying learning Scheme! For the future, I hope Guix can get even better test coverage so that every update of the master branch is guaranteed to not introduce regressions. And that the project gets more committers, to help with the constant influx of patches.

raingloom

There are multiple reasons I started using Guix. On the tech side, I'd been playing around with 9front for a while at that time, but kept running into issues where the imperative structure of namespaces was getting in my way. I like Haskell a lot and heard about the many benefits of a pure functional approach to build systems, et cetera. I ran Guix on top of Arch for a while and liked it a lot. Using package transformations still feels magical. But yall already know about this cool stuff from Ambrevar's blog post. On the social side, I saw that one of my favorite compsci people — Christine Lemmer Webber — was involved with the project, so I knew it probably has a nice community, which turned out to be very true. This is one of the best communities centered around a piece of tech that I've been in and yall inspire me to be better with each interaction. Huge thank you for that. My favorite Guix memory is when someone CC'd me in a patch for the egg importer, which was built on top of the Chicken Scheme build system I contributed. Seeing others build on top of my work is an amazing feeling. For the future, I hope the service management improvements will keep coming, but what I'd like to see the most is Guix running on old and slow devices. There is a lot of work to be done to make it more bandwidth and space efficient and to support development on systems with little RAM. If I could use it instead of PostmarketOS/Alpine, I'd be elated. On the human side, I hope we can keep contributors from burnout, while increasing the software's quality. I think the way Blender development is structured could be a source of inspiration. On that note, using Guix for reproducible art workflows would be rad. Okay that's it, bye yall lovely people.

Vagrant Cascadian

I think I first heard of Guix in 2016, triggering a late-night session trying to wrap my head around the crazy symlink farms at the heart of Guix. By late 2017 I was filing bug reports and eventually patches!

I am deeply fascinated that Guix has Reproducible Builds built right in, with normalized, containerized build environments and the "guix challenge" tool to verify reproducibility. I had heard of Nix as an interesting model, but valued the strong commitment to Free Software with Guix.

Eventually I even grew crazy enough to package Guix in Debian... which indirectly lead to one of my most creative contributions to a Free Software project, a typo poem embedded in!

I really appreciate the community around Guix and the process, values and thoughtfulness that work proactively to maintain a healthy community, even in the face of inevitable and occasional conflict. Guix balances formal and informal in a way that works for me.

I look forward to the day when Guix has a full source bootstrap!

10 Years of Guix artwork by Luis Felipe.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

by Guix Hackers at Monday, April 18, 2022

Mark Damon Hughes

A Schemer in Common Lisp Land

I got my Land of Lisp tshirt (shop is now closed), and thought for my weekend goof-off I'd read thru the book again and try actually using the Common Lisp examples; before I'd just converted it on the fly to Scheme, as I usually do with any Lisp stuff. But doing this exposed me to a fairly annoying environment, so I've been writing up my notes, function equivalents, workarounds. And I'll keep updating this page as I do more with it:

I had thought going in, maybe CL would be usable for some projects. But now, I know there's no way I would try to use this in production.

by mdhughes at Monday, April 18, 2022

Wednesday, April 6, 2022

Scheme Requests for Implementation

SRFI 232: Flexible curried procedures

SRFI 232 is now in final status.

Scheme lacks a flexible way to create and apply curried procedures. This SRFI describes curried, a variant of lambda that creates true curried procedures which also behave just like ordinary Scheme procedures. They can be applied to their arguments one by one, all at once, or anywhere in between, without any novel syntax. curried also supports nullary and variadic procedures, and procedures created with it have predictable behavior when applied to surplus arguments.

by Wolfgang Corcoran-Mathe at Wednesday, April 6, 2022