Planet Scheme

Thursday, November 19, 2026

spritely.institute

Lisp Game Jam devlog: Polydraw

This year's Autumn Lisp Game Jam has just wrapped up! Spritely's contribution to the jam was Polydraw, a peer-to-peer party game that is easy to pick up and play with friends. In Polydraw, players take turns being the artist who draws an image based on a prompt. The catch is that the artist only has 60 seconds and can only draw using triangles. All of the other players are guessers who try to correctly guess the prompt. We're quite pleased with how the game turned out!

The rest of this post is a daily devlog of how we made Polydraw so you can see what the process look likes to make a game with Goblins and Hoot under a tight deadline.

Day 1 (10/30)

It's go time! First, we created a Git repository on Codeberg and copied the Hoot game jam template into it.

Since the core activity in Polydraw is drawing triangles, the first thing Dave implemented was an interface for doing that on a canvas. This first version was crude and didn't include a color picker.

Day 2 and 3 (10/31 and 11/1)

We took the weekend off to prepare for a busy week ahead.

Day 4 (11/02)

This is the day that we really got to work on the game jam.

The first version of the triangle editor stored pixel coordinates of the local canvas, but that wasn't going to be good enough to display on other machines with different sized canvases. So, Dave simply normalized triangle vertices into the [0,1] range so each peer could scale them as needed.

Dave also implemented a basic scene system, a common game programming pattern, to encapsulate the coarse game states such as the title screen, the lobby, and the main game.

A color palette was added along with some additional mockup UI around the canvas.

Day 5 (11/03)

With a good enough game engine in place, we turned our attention to the game protocol. Since we are using Goblins, all interactions are modeled using actors. Dave came up with the following actors:

  • Player: Used for player identity within the game and contains the player's self-proposed name.

  • Lobby: The place where players gather before the game starts. The player that creates the lobby is the host and all other players are guests.

  • Game: Holds all of the game state like the player list, whose turn it is, and the scoreboard.

  • Controller: A player's virtual game controller. Used to speak to the game actor, but all actions are scoped to the individual player.

A test script was written to simulate creating a game, joining the lobby, and playing a round. At this stage, everything was happening in a single process. Goblins makes it easy to start with a single process and then distribute the actors to multiple machines later.

Day 6 (11/04)

Now it was time to get the actors onto the network and hook them up to the game UI. For networking we used OCapN, naturally. Jessica started up an instance of our new end-to-end encrypted relay netlayer that we set as the default relay so the game would "just work" when we uploaded it to itch.io. In fact, dogfooding the relay was a large part of the motivation behind participating in the game jam in the first place!

Dave got the actors online with OCapN and connected the game UI to the actors, then confirmed that a three player game works. Three players is the minimum number of players needed to trigger OCapN's third-party handoff feature. Handoffs are what enable distributed networking in OCapN: Alice can give Bob a reference to Carol, which causes Bob and Carol to establish a new OCapN session with each other. This is the first application we've made that does handoffs over the relay netlayer and it was nice to see them working as intended.

Unfortunately, we discovered an issue with how Goblins handles two peers connecting to each other concurrently, a problem known as "crossed hellos". OCapN mandates that only one connection may be open between any two peers, so we have to pick which connection wins. The bug is that any promises that were created over the losing connection never resolve. This was observed in the game as an empty lobby player list that should be populated. It didn't prevent the rest of the game from working, though, so we didn't try to fix it (yet).

Day 7 (11/05)

On this day, Dave discovered that, despite his best efforts, he fell victim to the "identity footgun". We could write a whole blog post series about the many subtle issues of object identity (in the sense of Scheme's eq?) in object capability systems, but instead we'll refer you to the The Grant Matcher Puzzle and just mention that using object references as keys in associative data structures is error-prone. A promise that eventually resolves to a reference to Alice is not identical (again, in the eq? sense) to a reference to Alice. So, if you accidentally use a promise as a key somewhere or use eq? directly you will probably have a bad time, which is exactly what happened to us.

To deal with this identity crisis, Dave first detected it at runtime by writing a wrapper around eq? called geq?. This new predicate throws an error when comparing a promise to a non-promise. Then he used the resulting backtraces to find the offending code and change it to resolve all necessary promises before doing identity comparisons.

Christine finally cleared enough other things off her plate where she could start contributing to the game. We thought it would be nice to have the game start with a splash screen of triangle'y animations that would resemble the kinds of art players would make in the game. We decided to use the Blender add-on by Ursula Kurdali named Stop Motion since it gives a great amount of freedom to animate by creating and destroying as much geometry as you like. Christine made the first mockup of a splash screen animation of a "death" type character swinging a scythe.

However, the animation hadn't left Blender yet. We're gonna have to figure out how to get those polygons out and into the game engine!

Day 8 (11/06)

This was the day to finish implementing the core game loop. We wanted the weekend to be all about fixing bugs and making it look and feel as good as we can. At last, the points awarded at the end of each round were displayed in the UI. The player list on the left side of the screen was transformed into a leaderboard that sorted players by score.

While creating the leaderboard it was discovered that Goblins was not correctly parsing pairs received over OCapN. Christine swooped in with a quick fix and Jessica folowed up with a regression test.

We also added a small set of emoji reactions underneath the leaderboard. Reactions were not critical to the core gameplay but they were planned from the very beginning of the project. It's good that we had time to add them because it was a lot of fun to spam emojis at the artist to give them an indication of how good or bad they were doing.

Day 9 (11/07)

With two days left and the core gameplay loop complete, the focus shifted to bug fixing and adding juice. Up until this point the HTML was mostly unstyled (as you can see in the videos above), so Dave added a little CSS to improve the look and feel.

A playtest revealed that the simple PRNG we were using was implemented incorrectly. We noticed because the artist prompt was always "bus" no matter how many rounds you played. Oops! This was fixed.

Remember that crossed hellos bug from day 7? Dave finally implemented a workaround, which was to simply throw out the first pancake and try again, as suggested by Christine. A simple race against a timeout was all that was needed. We'll be thinking about how to fix this in Goblins so this situation cannot happen in the future.

Christine started work on making more animations for the game's intro. The "death" animation was a good "hello world" for testing the exporter, but being all black it was too simple to make sure everything worked right; draw order, etc, didn't matter. Christine made a "square guys" animation as a kind of Acid2 test equivalent. This file had several tricky properties to it, most especially that draw order would matter, colors would be important, et cetera.

Christine also made this triangle'y duck themed piece:

Christine wrote a Python script to export the animations from Blender into a custom format encoded using Syrup. The encoded data was then embedded into the Scheme program as bytevector literals using a little script. Dave then started work on a renderer for these animations, but it wasn't working quite well enough by the end of the day to commit and push it.

Up until this point we only had a small number of frequently-repeated words. Morgan Lemmer-Webber curated a word list for us, starting with an existing public domain word list of 5000 words, and curated down to a 621 words, selecting for words that were more feasible to draw (ranging from easy to challenging, but all trying to be feasible).

The game name was changed from the placeholder "Twenty Triangles" to "Polydraw".

Day 10 (11/08)

The final day.

The code to render animations was finished. Some final minor bug fixes were made. Some CSS was tweaked. Original music and sound effects by encrypted.whispers were added. Additionally, we used a few CC0-licensed sound effects by Kenney.

A few gameplay improvements were made, such as showing guessers what the prompt was when the round ends and how many triangles the artist had remaining. Little improvements can make a big difference and both of these changes were very helpful for players and made the game feel more polished.

There wasn't enough time remaining to do as much work on resiliency in the face of network partitions as we would have liked, but Dave did squeeze in a commit that allowed the game to survive if a guest leaves. Unfortunately, the experience for the disconnected guest isn't very good as they don't see any message that their connection dropped.

We did a final playtest to confirm that there weren't any showstopper bugs. Christine added final splash animations.

Finally, Christine added some code (fixed up by Dave) and animations which would overlay over the drawings indicating that drawing was starting, it was time to guess, or the results.

Pencils down!

We submitted the game two hours before the deadline! Now it was time to catch our breath and show off the game to everyone we could. Dave even played it live with a couple other people on a stream! Here's a screenshot which gives you a good look of the final result:

The response was quite positive! The feedback we received closely matched the things we wanted to improve upon if we had more time: better disconnect handling, a way for the host to skip to the next phase instead of being forced to wait for the timer to run out, and invite URLs (rather than URIs) that auto-join the game when visited.

Polydraw ranked 5th overall in the official game jam results.

Try playing it over on itch.io and leave us a comment!

Thanks to our supporters

Your support makes our work possible! If you like what we do, please consider becoming a Spritely supporter today!

Diamond tier

  • Aeva Palecek
  • David Anderson
  • Holmes Wilson
  • Jonathan Frederickson
  • Lassi Kiuru

Gold tier

  • Alex Sassmannshausen
  • Juan Lizarraga Cubillos
  • Nerine Pekowsky
  • Serra Allgood

Silver tier

  • Austin Robinson
  • Brit Butler
  • Charlie McMackin
  • Dan Connolly
  • Deb Nicholson
  • Diana Thayer
  • Eric Bavier
  • Eric Schultz
  • Evangelo Stavro Prodromou
  • Evgeni Ku
  • Glenn Thompson
  • James Luke
  • Jonathan Smith
  • Jonathan Wright
  • Justin Sheehy
  • Matt Panhans
  • Michel Lind
  • Mike Ledoux
  • Nathan TeBlunthuis
  • Nia Bickford
  • Noah Beasley
  • Steve Sprang
  • Travis Smith

Bronze tier

  • Alan Zimmerman
  • Aria Stewart
  • Ben Hamill
  • Benjamin Grimm-Lebsanft
  • Brooke Vibber
  • Brooklyn Zelenka
  • Carl A
  • Crazypedia No
  • Ellie High
  • Elliot Martin
  • François Joulaud
  • Gabor Udvari
  • Grant Gould
  • Gregory Buhtz
  • Ivan Sagalaev
  • James Smith
  • Jason Wodicka
  • Jeff Forcier
  • Marty McGuire
  • Mason DeVries
  • Michael Orbinpost
  • Neil Brudnak
  • Nelson Pavlosky
  • Nicholas Farley
  • Philipp Nassua
  • Rodion Goritskov
  • Ron Welch
  • Stefan Magdalinski
  • Stephen Herrick
  • Steven De Herdt
  • Tamara Schmitz
  • Thomas Talbot
  • William Murphy
  • Yoshua Wakeham
  • a b
  • chee rabbits
  • r g
  • terra tauri

by Dave Thompson, Christine Lemmer-Webber at Thursday, November 19, 2026

Saturday, November 14, 2026

Scheme Requests for Implementation

SRFI 289: Unordered sets and bags

SRFI 289 is now in draft status.

Sets and bags (also known as multisets) are unordered collections that contain only objects of a type given by a collection’s SRFI 128 comparator. Every element of a set is unique (in the sense of the set’s comparator), but bags can contain any number of duplicate elements. This SRFI is a revision of SRFI 113 that fixes several infelicities and is more consistent with related SRFIs.

by Wolfgang Corcoran-Mathe (text and test suite), Peter McGoron (sample implementation), and John Cowan (author of SRFI 113) at Saturday, November 14, 2026

Sunday, November 1, 2026

Scheme Requests for Implementation

SRFI 272: Pretty Printing

SRFI 272 is now in final status.

This SRFI follows the traditional Scheme model of pretty printing, which treats it as a process distinct from general controlled formatting. While general-purpose formatters often prioritize specialized presentation at the expense of machine-readability, Scheme’s pretty-printers (such as those of SLIB and MIT Scheme) have traditionally treated pretty printing as a variant of write, differing primarily in the insertion of whitespace to make the presentation more palatable to humans. Common Lisp’s pretty-printer, by contrast, fills two roles simultaneously by integrating pretty printing with both its format facility and its generalized write procedures. This unified approach offers great power, but at the cost of complexity that can make it difficult to use effectively. We propose a specialized, layered approach, specifying five libraries of increasing functionality, where all but the first are optional. The libraries are downward-compatible: more powerful libraries satisfy all requirements of the simpler ones while adding new features. Implementors may choose to support a maximum level of functionality appropriate for their systems. Integration with monadic and string-based formatting libraries is supported.

by Sergei Egorov at Sunday, November 1, 2026

Thursday, October 29, 2026

Scheme Requests for Implementation

SRFI 279: In(tro)spection Protocol

SRFI 279 is now in final status.

Interactive REPL-driven systems (that most Schemes are) need a way to get detailed information on a given piece of data. Inspectors, as these are conventionally called. This SRFI defines a basic protocol for inspectors, consisting of three procedures: inspect-properties, inspect-property, and inspect-describe. Some suggestions for standard and popular types’ inspection are also provided.

by Artyom Bologov at Thursday, October 29, 2026

Wednesday, October 28, 2026

spritely.institute

Hoot 0.10.0 released!

We are excited to announce the release of Hoot 0.10.0! Hoot is a Scheme to WebAssembly compiler backend for Guile, as well as a general purpose WebAssembly toolchain. In other words, Scheme in the browser!

This release contains new features and bug fixes and since the 0.9.0 release back in May.

Use Hoot in the upcoming Lisp Game Jam!

On Friday (October 30th 2026), the autumn edition of the Lisp Game Jam will begin! It's a 10-day long game jam where participants make games using their favorite flavor of Lisp.

Does making a small web game in Scheme using Hoot sound appealing to you? Well, then we have just the thing to get you started: the Hoot game jam template! This template project has everything you need to start making an HTML5 game with Hoot quickly.

The template repository includes:

  • Bindings to the necessary web APIs to make an interactive game with HTML5 canvas

  • A Makefile for compiling, running a development web server, and generating a .zip bundle for uploading to itch.io

  • A very simple Breakout-like example game that demonstrates how to put all the pieces together

For more inspiration, here are some games made with Hoot for past jams:

Now, onto the release notes!

Compiler changes

  • Added module name to contextualize duplicate import errors.
  • Added support for library versioning for R6RS compatibility.
  • Fixed default version for define-module libraries.
  • Fixed locate-library not respecting configured load extensions. Previously, we were using Guile's load extension parameter instead of Hoot's own.

Runtime changes

  • Added %struct-ref and %struct-set! primitives for accessing record types defined using the interpreter.
  • Stopped raising errors when importing an undefined symbol in the runtime module system. Guile is lax about this so Hoot being strict caused compatibility issues.
  • Fixed optional argument parsing in primitive-eval.
  • Fixed keyword argument parsing for procedures with both optional and keyword arguments in primitive-eval.
  • Fixed error messages regarding runtime module registry to include the correct CLI flag: -fruntime-modules.

Scheme changes

  • Added partial implementation of SRFI-13: String Libraries. It's just a set of re-exports, very similar to Guile, but Guile's SRFI-13 module is not declarative and is thus rejected by Hoot's compiler.
  • Added partial implementation of SRFI-60: Integers as Bits.
  • Added define-macro, defmacro, delete, list->symbol, and symbol to (guile).
  • Exported define-syntax-parameter and syntax-parameterize, symbol-append, integer-length, and gensym from (guile).
  • Added list->bitvector and bitvector to (hoot bitvectors) and re-exported them in (guile).
  • Implemented get-bytevector-all in (rnrs io ports).
  • Made bytevector=? a publicly exported procedure from (hoot bytevectors) and (rnrs bytevectors).
  • Added define-syntax-parameter and syntax-parameterize primitives.
  • Improved (ice-9 exceptions) exports to more closely match Guile.
  • Added (rnrs) module.
  • Added (rnrs base) module.
  • Added (rnrs syntax-case) module.
  • Added (rnrs r5rs) module.
  • Added (rnrs hashtables) module.
  • Added (rnrs files) module.
  • Added (rnrs enums) module.
  • Added (rnrs exceptions) module.
  • Added (rnrs arithmetic fixnums) module.
  • Added (rnrs records inspection) module.
  • Added (rnrs records procedural) module.
  • Added (rnrs records syntactic) module.
  • Added (rnrs io simple) module.
  • Added (rnrs io ports) module.
  • Added port-has-position? to (hoot ports) for R6RS compatibility.
  • Added stubs for centered division to (hoot numbers) and a stub for integer-length to (hoot bitwise) for R6RS compatibility (eventually).
  • Rewrote (rnrs bytevectors) as an R6RS module with version (6).
  • Fixed number->string to round floating point decimal part to an even digit.
  • Fixed number->string for floating point numbers ±2^N, where 52 <= N <= 1023.
  • Fixed incorrect bytevector copy in number->string floating point conversion algorithm.
  • Fixed min/max macros not working when given zero arguments.
  • Fixed the parameters to the log procedure in (guile).
  • Fixed the implementation of negate in guile. It was doing entirely the wrong thing.

CLI changes

  • Port numbers are now validated to be exact integers in the range [1, 65535].
  • Actually complain about invalid option or unknown subcommand. Previously, the code that was supposed to do this was unreachable.
  • Add --experimental-wasm-exnref feature flag when running hoot compile --run=node for continued compatibility with supported NodeJS versions that don't have the new Wasm exnref feature enabled by default.
  • Fixed build-time disabling of hoot server. In past releases, the hoot CLI script would still attempt to load should-be-optional libraries that ./configure had already determined to be missing. Thanks to mcc for discovering this blunder.

Documentation changes

  • Improved instructions in "Building from source" section of the manual.
  • Dependencies are now mentioned in the "Advanced path" section of the README.
  • Updated project template README to say Guix is not required (though it's still the easiest option).

Browser compatibility

  • Compatible with Safari 26 or later.
  • Compatible with Firefox 121 or later.
  • Compatible with Chrome 119 or later.

Get Hoot

Hoot is available in GNU Guix:

$ guix pull
$ guix install guile guile-hoot

Also, Hoot is now available in Debian, though it will take awhile for this release to make it there.

Otherwise, Hoot can be built from source via our release tarball. See the Hoot homepage for a download link and GPG signature.

Documentation for Hoot 0.10.0, including build instructions, can be found here.

Get in touch

For bug reports, pull requests, or just to follow along with development, check out the Hoot project on Codeberg.

If you build something cool with Hoot, let us know on our community forum!

Thanks to Z572, stdh, Walter Lewis, and Jelle Licht for their contributions to this release.

Until next time, happy hooting! 🦉

Thanks to our supporters

Your support makes our work possible! If you like what we do, please consider becoming a Spritely supporter today!

Diamond tier

  • Aeva Palecek
  • David Anderson
  • Holmes Wilson
  • Jonathan Frederickson
  • Lassi Kiuru

Gold tier

  • Alex Sassmannshausen
  • Juan Lizarraga Cubillos
  • Nerine Pekowsky
  • Serra Allgood

Silver tier

  • Austin Robinson
  • Brit Butler
  • Charlie McMackin
  • Dan Connolly
  • Deb Nicholson
  • Diana Thayer
  • Eric Bavier
  • Eric Schultz
  • Evangelo Stavro Prodromou
  • Evgeni Ku
  • Glenn Thompson
  • James Luke
  • Jonathan Smith
  • Jonathan Wright
  • Justin Sheehy
  • Matt Panhans
  • Michel Lind
  • Mike Ledoux
  • Moto A
  • Nathan TeBlunthuis
  • Nia Bickford
  • Noah Beasley
  • Steve Sprang
  • Travis Smith

Bronze tier

  • Alan Zimmerman
  • Aria Stewart
  • Ben Hamill
  • Benjamin Grimm-Lebsanft
  • Brooke Vibber
  • Brooklyn Zelenka
  • Carl A
  • Crazypedia No
  • Ellie High
  • François Joulaud
  • Gabor Udvari
  • Grant Gould
  • Gregory Buhtz
  • James Smith
  • Jason Wodicka
  • Jeff Forcier
  • Marty McGuire
  • Mason DeVries
  • Michael Orbinpost
  • Neil Brudnak
  • Nelson Pavlosky
  • Philipp Nassua
  • Rodion Goritskov
  • Ron Welch
  • Stefan Magdalinski
  • Stephen Herrick
  • Steven De Herdt
  • Tamara Schmitz
  • Thomas Talbot
  • William Murphy
  • a b
  • chee rabbits
  • r g
  • terra tauri

by Dave Thompson at Wednesday, October 28, 2026

Monday, October 26, 2026

Scheme Requests for Implementation

SRFI 242: The CFG Language

SRFI 242 is now in withdrawn status.

This SRFI defines a language to describe control-flow graphs (CFGs) suitable for formulating iterative and recursive algorithms. Using the notion of a CFG term, this language can be seamlessly embedded in the Scheme language. Complex CFG terms can be composed from simple CFG terms.

by Marc Nieper-Wißkirchen at Monday, October 26, 2026

SRFI 265: The CFG Language

SRFI 265 is now in final status.

This SRFI defines a language to describe control-flow graphs (CFGs) suitable for formulating iterative and recursive algorithms. Using the notion of a CFG term, this language can be seamlessly embedded in the Scheme language. Complex CFG terms can be composed from simple CFG terms.

The language described in this SRFI is not meant to be directly used in programs but by library authors to build abstractions like loop facilities on top of it.

by Marc Nieper-Wißkirchen at Monday, October 26, 2026

Sunday, October 25, 2026

Scheme Requests for Implementation

SRFI 288: Compound Conditions

SRFI 288 is now in draft status.

A condition object system is introduced that attempts to be implementable in and interoperable with many implementation’s condition systems, including the condition system of the R6RS. The condition system includes compound conditions, but does not include subtyping of conditions. Simple conditions can be any implementation-defined value, and are also objects that can be made with the interface in this SRFI. An implementation may add other features, such as introspection and serialization.

by Peter McGoron at Sunday, October 25, 2026

Gwen Weinholt

Akku.scm 1.2.0 released

Akku.scm version 1.2.0, the language package manager for R6RS and R7RS Scheme, is now generally available. It can be downloaded from GitLab. The most important change is a fix to the dependency solver, which was running very slowly due to a bug triggered by the many versions of the chez-srfi package.

Akku is a project-based package manager, where each project has its own set of dependencies. You use .akku/env to set shell variables that let your Scheme implementation know where to find everything.

You can upload your own packages to the archive. They go through a review process to ensure nothing malicious is distributed.

Akku also performs automatic conversion from the R7RS define-library form to the R6RS library form. This lets, e.g., Chez Scheme use R7RS libraries. Packages from the Snow package manager, which has many R7RS packages, are also available via Akku.

Further reading

by weinholt at Sunday, October 25, 2026

Saturday, October 24, 2026

jointhefreeworld

Emacs finally folds to my liking

I’ve been slowly turning my Emacs config into something I actually enjoy using, not just something that works, and folding was the last place where that wasn’t true.

Every mode had its own idea of what folding is: treesit-fold in Tree-sitter modes, Hideshow in the Lisp-ish ones, Outline in everything that resembles text, Org with its own invisible text that no one else’s commands can undo.

So every buffer had different keys, a different look, and worst of all different performance characteristics. Folding a big file sometimes felt like holding my breath.

I’m running Emacs 32.0.50 from master (built from source, as one does), and it’s finally one system.

kirigami is the piece I didn’t know I needed  #

kirigami doesn’t fold anything itself. It dispatches: it keeps a list of folding engines, checks which one is actually active in the buffer, and forwards to it. Tree-sitter, Hideshow, Outline, Org, Vdiff — all behind one interface.

So on top of it I put one set of keys, the same in a Kotlin buffer, YAML, a Scheme or ELisp buffer and an Org file:

Key Does
C-c v c close this fold
C-c v C close every fold
C-c v o open this fold
C-c v O open this fold and everything under it
C-c v r reveal everything
C-c v t toggle the fold at point

That sounds small. It is not small. The number of times a day I used to think “wait, what does TAB do in this mode again” is now zero, and the muscle memory transfers across an entire polyglot config.

One wrinkle worth mentioning: kirigami answers with the first engine in its list whose mode is on, and outline-minor-mode is on in nearly every buffer I have. In the buffers where I deliberately turned on Hideshow — Scheme, for instance — that choice has to win, otherwise C-c v c folds “headings” a Scheme file doesn’t have. So I reorder kirigami-fold-list to put Hideshow first instead of rewriting it; engines kirigami adds later still get used wherever Hideshow is off.

Tree-sitter folding, with guides that are actually lines  #

Most of my programming buffers are *-ts-mode, so treesit-fold does the work there — bash, C, C++, CMake, CSS, Dockerfile, Elixir, Go, HTML, Java, JS/TS/TSX, JSON, Kotlin, Lua, Nix, PHP, Python, Ruby, Rust, Scala, TOML, YAML. Line counts are shown inline, which I find genuinely useful when deciding whether to open a fold.

The indicators took the most fiddling. treesit-fold-indicators draws a block’s guide with a bitmap 23 rows tall, centered in the line beside it. A line in my config is 29 pixels — the font’s 23 plus six of line-spacing — so every row of the guide was a 23-pixel dash with three pixels of nothing above and below it. A folded block looked like a dotted line running down the fringe. Not subtle, just wrong.

There’s no setting for that, so I replaced the bitmaps: a single-row bitmap that define-fringe-bitmap repeats down however tall the line is (no spacing can introduce a gap), plus a +=/-= box and an end-elbow cut to the exact height of a line, so the guide runs from the marker into the row below and turns toward the text where the block ends. It’s an advice on the function that builds the string for each line. It looks like a bracket now, not a perforation.

And the colour: the indicators inherited shadow, which most themes make as bright as a comment. Fine for one marker on one line, wrong for a marker on every line of a block — the grey became a rule drawn through the code. They’re now shadow blended 50% toward the theme’s background, recomputed on every theme and scale change, so it fades correctly on light themes too and no theme needs a special entry.

Hideshow earns its keep  #

Not everything has a grammar, and I’m not going to pretend otherwise:

  • Scheme — my Emacs build has no Scheme grammar, so Scheme buffers use Hideshow with hs-block-start-regexp set to "^(" and hs-forward-sexp-function set to forward-sexp. A Scheme file is a sequence of top-level forms, so each definition is a fold, and collapsing the one you’re inside works from anywhere within it. That’s the behaviour I wanted from the start and Hideshow doesn’t guess it by default.
  • Emacs Lisp — the prog-mode fallback, same keys.
  • XML — folds from start tag to end tag using nxml-scan-element-forward, skipping , comments, DOCTYPE and self-closing elements.
  • YAML (the non-Tree-sitter mode) — indentation-based blocks with a small forward-sexp of my own that walks to the end of the indented block.

Hideshow’s own fringe indicators ( hs-show-indicators + hs-indicator-type 'fringe) are on in all of these, so it doesn’t feel like a downgrade from the Tree-sitter buffers.

The rule I enforce per buffer is exactly one set of indicators: Tree-sitter buffers get treesit-fold and Hideshow is turned off; Hideshow buffers get Hideshow and Outline’s clickable buttons are turned off so the two don’t double up; outline-only buffers (Markdown, diff) keep the buttons. Nobody ends up with two markers per block.

Org, which needed its own everything  #

Org folds with text properties, not overlays, so neither Tree-sitter nor Hideshow nor Outline’s buttons can draw a marker for it. So I wrote its own indicators: a fringe triangle pointing down for an open heading, right for a folded one, clickable with mouse-1 to toggle exactly like TAB. Folding itself is untouched — TAB, S-TAB and the C-c v keys all still work.

The performance part is where the real win is. A naive “redraw indicators after every command” implementation is a disaster in a large file, because it re-searches the buffer and rebuilds every overlay on every pause. Mine is driven by the text instead:

  • if buffer-chars-modified-tick didn’t change, only the indicators that are on screen are redrawn — off-screen headings can wait until they’re scrolled to;
  • if it changed and the change was recorded, only the dirty region is rebuilt;
  • if it changed with no recorded change, only then is the whole buffer rebuilt.

Folding changes text properties rather than characters, so folding alone takes the cheap path. Fold state is read back from Org on each redraw rather than remembered, so external folding can never leave a stale marker.

Two config choices matter just as much: org-fold-core-style is text-properties (so a visibility cycle doesn’t re-parse the file), org-element-use-cache is on, and Org starts with showeverything — which is both what I want (a note is read, not consulted) and the cheaper state, because it skips the visibility passes that walk every heading on open. I also removed a org-restart-font-lock that was re-fontifying the whole buffer a second time on open; JIT-lock handles what’s on screen.

It feels smooth, and that’s the whole point  #

The end result is that folding is boring now. C-c v c in Kotlin, Scheme, Elisp, Org, XML, YAML, Markdown. Guides that look like brackets. Indicators quiet enough to ignore until you need them. No pause on a 3000-line Org file. No two competing markers.

It honestly changes how much I collapse and expand while reading code. I used to avoid folding because it felt fiddly; now it’s free and mouse friendly.

If you want to read the actual code: it’s in dev-folding.el for the engines and indicators, the heks/fold-* wrappers in heks-emacs.el for the keys, and the Org indicators live in org.el. Everything is GPL-3.0, and it’s tested: maak verify asserts a fold actually covers a whole definition and that the documented keys are bound, and maak keys fails on a key that does two jobs or names a command that doesn’t exist.

Happy to answer questions about any of it, especially if you’ve fought the treesit-fold-indicators bitmaps yourself, because that one cost me an evening and I’d love to know if there’s a cleaner way.

Saturday, October 24, 2026

OrgWebAlchemy: Benchmarking, Profiling, and the Long Road to Real Org Support

OrgWebAlchemy is a Guile library that parses Org mode with a Parsing Expression Grammar and renders it to HTML, Markdown, or SXML. I wrote an earlier article about why a PEG is the right tool for Org: the grammar describes the language, the parser builds a tree, the renderers turn that tree into output.

This is the follow-up, and it turned into a harder question than I expected: how do you make a parser fast without making it wrong? I didn’t want something that’s quick on my test files and quietly behaves differently on yours.

So: the features I added, how I measured them, what the profiler said, the two rounds of fixes, and how I checked the language didn’t change. Every number below comes with a way to reproduce it, because a number you can’t check is advertising, not evidence.

TLDR; Here is the whole article in five sentences:

  • Parsing a 58 KB Org document now takes 5.1 ms, where it used to take 32.2 ms, a factor of 6.4.
  • The same parse now allocates 3.9 MB, where it used to allocate 30.3 MB, 7.8 times less garbage.
  • A document consisting of one 80 KB line went from 8.2 seconds to 0.37 milliseconds, because the old path was quadratic in the length of a line and the new one is linear.
  • The library gained a Markdown renderer, figures, named elements, cross-references, a proper table reader and mixed nested lists, and it says plainly what it still doesn’t do.
  • None of this was guessed: there are 155 test definitions and 281 assertions, an equivalence run compared the new parser with the previous one on about 70,000 inputs, and a test sweeps about 390,000 positions to check the scanner’s stop set.
One Org document, rendered by both renderers.

Free software, and why this is a library  #

OrgWebAlchemy is free software under the GNU Lesser General Public License, version 3 or later, with this documentation under the GNU Free Documentation License. Run it, study it, change it, ship your changed versions.

It could have been a service instead: you send your Org file to a server you don’t control, and get HTML back. That can be convenient, and it can even be gratis, but it leaves you helpless in the ways that matter. You can’t study it, can’t fix it, and can’t keep using it when the operator loses interest or changes the terms. A library goes into your program, on your machine, under your control.

The LGPL is a compromise, and I’ll name it as one. The GPL would keep every program that uses this code free too, and that’s the stronger guarantee for users. For libraries the GNU Project has long accepted a weaker license so that more programs can use them, including some that aren’t free, and I’ve followed that tradition. Use it in a free program and you’re helping the community; use it in a nonfree one and I’d rather you made that program free.

There’s a second reason it’s a library. Org mode is a language, and a language is worth as much as the number of programs that can read it. A parser only Emacs can use is a parser for one program. One that any Guile program can call is a parser for a text editor, a static site generator, a wiki, a notebook, a mail archiver, a package’s docs build, or something nobody has written yet. I don’t know what you’ll build with it, and that’s the point.

And a word on what it isn’t: it’s not Emacs and doesn’t try to be. Emacs is where Org lives, where it gets edited, and where its export machinery started. OrgWebAlchemy doesn’t replace that, it just lets other Scheme programs read the same files. When it disagrees with Emacs about some corner of the syntax, Emacs is right and this is a bug.

What the program is now  #

OrgWebAlchemy is still three small modules. What changed is how much of the language each one covers.

Module Lines What it owns
(orgwebalchemy peg) 1,700 the grammar, the AST, the run scanners, link classification, name resolution
(orgwebalchemy html) 720 rendering to HTML and SXML, and every CSS class parameter
(orgwebalchemy markdown) 430 rendering the same AST to Markdown
Test suite 1,670 155 test definitions and 281 assertions, many of them table-driven

The API is still small enough to hold in your head.

(use-modules (orgwebalchemy html)
             (orgwebalchemy markdown)
             (orgwebalchemy peg))

(org->html  "This is ~test~ code.")
 ;;  => "This is test code."

(org->markdown  "This is ~test~ code.")
 ;;  => "This is `test` code."

 ;;  parse once, render as often as you like
( define  ast (org->ast  "* Hello\nA *bold* line.\n"))
(org-ast->html ast)
(org-ast->markdown ast)
copy

The rest of this article is about what happens between those calls.

Teaching the parser more of Org  #

“Supporting Org” hides a lot of work, because Org isn’t a markup language with a tidy grammar. It’s a pile of conventions accumulated over years of a GNU Emacs major mode, some of them deliberately informal, and each one has to be considered on its own terms. Every feature below started as a sentence in the Org manual, became a grammar pattern, then became a test.

Links: decide once, in the parser  #

An Org link’s target says what kind of link it is. A target starting with https:// is external. file:photo.png is a file, and so is a bare notes.org. #custom-id points inside the document, *Heading at a heading, and a ::search suffix says where inside the target you land.

The old code let each renderer read the target and decide for itself, which is exactly how two back ends drift apart. HTML grows a special case for image extensions, Markdown grows a slightly different one, and after a year of small repairs the same document means two things depending on which function you call. A user who notices won’t know which one is right, and will be right not to trust either.

So classification moved into the parser, where it happens once.

(org-link-parts  "https://example.org/a")    ;;  => (https . "//example.org/a")
(org-link-parts  "file:photo.png")           ;;  => (file . "photo.png")
(org-link-parts  "notes.org")                ;;  => (file . "notes.org")
(org-link-parts  "#custom-id")               ;;  => (internal . "custom-id")
(org-link-parts  "*My Section")              ;;  => (heading . "My Section")
(org-link-parts  "notes.org::*Heading")      ;;  => (file . "notes.org")
(org-link-search  "notes.org::*Heading")     ;;  => "*Heading"
(org-image-link?  "photo.jpeg?raw=1")        ;;  => #t
copy

Both renderers then ask the same questions and follow the same answer.

Org HTML Markdown
[[file:photo.png][a photo]] a photo ![a photo](photo.png)
[[images/diagram.svg]] ... ![images/diagram.svg](images/diagram.svg)
[[file:notes.org][notes]] notes [notes](notes.org)
[[#custom-id][there]] there [there](#custom-id)
[[*My Section][up]] up [up](#my-section)
[[https://gnu.org][GNU]] GNU [GNU](https://gnu.org)

Two details deserve a sentence each. The ::search suffix is set aside before the path is used as a file name, so file:notes.org::*Heading doesn’t produce a broken file name. And C:/tmp/x.txt isn’t mistaken for a link with a scheme named c, because a real scheme is lower case. Small stuff, but it’s the kind of small stuff that decides whether a program is pleasant to use on somebody else’s computer.

Figures  #

Org gives the element after a #+CAPTION: or #+NAME: keyword a caption and a name. When that element is a link to an image, the three things are one object: a figure. The parser folds them together, so renderers get a single node instead of three neighbors they have to guess about.

(org->html  "#+NAME: fig:arch\n#+CAPTION: The architecture\n[[file:diagram.png][a diagram]]\n")
 ;;  => "
\"a
The architecture
"
copy

The name the figure carries is the identifier that a [[fig:arch]] link resolves to, so the cross-reference can’t break. Both facts come from one place, and one place can’t disagree with itself.

(org->html  "#+NAME: fig:arch\n#+CAPTION: The arch\n[[file:a.png]]\n\nSee [[fig:arch]].\n")
 ;;  => "
...

See fig:arch.

"
copy

Markdown has no figure element, so it gets the image, the caption as an emphasised line, and an explicit anchor for the cross-reference. What counts as a figure is deliberately narrow: the keywords must sit immediately before a paragraph holding exactly one image link and nothing else. Everything else stays a keyword line. Inventing a caption for an element this parser doesn’t caption is worse than saying nothing, because the reader would never learn that something got silently dropped.

Named elements  #

A figure is the image case of a more general rule. A #+NAME: line names the element after it, and anything that renders as something a link can land on can be named: a table, a source block, a quote, a list, a separator, a paragraph.

(org->html  "#+NAME: tbl:users\n| Users | 1024 |\n\nSee [[tbl:users]].\n")
 ;;  => "...

See tbl:users.

"
copy

HTML puts the identifier on the element itself. Markdown has nothing to hang it on, so it emits an anchor in front, same as for a figure. A bare target that matches the name of an element in the same document resolves as a cross-reference, which is why [[tbl:users]] and [[notes]] work. A target that already says what it is — a scheme, a file: prefix, a leading # or *, or a ::search suffix — keeps the meaning it wrote.

One asymmetry, documented rather than hidden: a #+CAPTION: in front of a named table isn’t folded into it, because captions on tables aren’t rendered yet. The line stays an ordinary keyword line where the writer put it.

Tables, read by their bars  #

Tables looked finished until I looked at the edges. What should | on a line of its own be? What is | a ||? What happens to a row that ends the input with no newline?

The old pattern counted separators and made one cell for each, which meant a row with no cells could raise an error, and a row that ended the input got a phantom empty cell from the newline it didn’t have. The rewrite reads a row the way Org does: by the bars that separate the cells, where a bar closes a row when nothing but the end of the line follows it. That one rule answers all three questions.

  • A line that is only its closing bar ( | on its own) is a row of one empty cell, not a row with no cells.
  • | a || is a row of two cells, the second of them empty, rather than a paragraph.
  • A row that ends the input keeps exactly the cells that were written.

An empty cell also keeps its column, because an empty value is now the empty string instead of the bare tag symbol that an empty (* ...) match leaves behind. The difference sounds academic until it is your table.

Lists that mix and nest  #

Org lets a list change its bullet type part-way through, and it lets a nested list use a different kind from its parent. So the grammar stopped having three separate list productions and grew one.

(define-peg-pattern org-list all
                    (+ ( or desc-item unordered-item ordered-item)))
copy

The renderers group consecutive items of the same kind into the proper element, which is what makes - outer with 1. nested inside it come out as an unordered list containing an ordered one. Checkboxes, description items and arbitrary nesting depth follow from the same rule.

The rest of the surface  #

Between those features and the earlier work, the supported set is now: headings with TODO keywords, priorities and tags; paragraphs; unordered, ordered and description lists in any mixture and at any depth; checkboxes; tables with empty cells; source, example, quote, verse, center, export and comment blocks; keywords and in-buffer settings; comments; fixed-width lines; separators; footnotes, whether references, definitions or inline; bold, italic, underline, strikethrough, code, verbatim, superscript, subscript, escaped markup and forced line breaks; links with and without descriptions, bare URLs, images, figures, named elements and heading anchors.

That’s most of what people actually write. Not all of Org, and the manual has a section listing the gaps instead of hiding them, because a library that lies about its coverage just wastes everyone’s time.

Measuring the work  #

Before optimizing anything I wanted a benchmark that could tell me whether a change helped, and that couldn’t be fooled by the change itself.

The benchmarks live in test/veritas/benchmark/scope.scm. They use the same Veritas framework as the unit suite, selected by test type so they never slow down maak test.

 VERITAS_TEST_TYPE=benchmark  \
guix shell -m manifest.scm -- guile -L ./src -L ./test  \
  -c  '((@ (veritas runner) run-tests))'
copy

The suite measures five workloads and prints three tables — wall clock, allocation, and the collapse pass — formatted as Markdown, ready to paste into the manual. That last part matters: a number you have to retype by hand quietly stops being true.

Workload What it exercises
Parse the feature fixture every construct the parser knows, in 1.8 KB
Parse README.org a real 58 KB document with prose, tables and code blocks
Render the feature fixture to HTML parsing plus the HTML renderer
Render the feature fixture to Markdown parsing plus the Markdown renderer
Parse a 50-clause inline paragraph the inline grammar, under maximum pressure

Five runs, keep the fastest  #

One wall clock reading on a laptop isn’t a measurement, it’s a rumor. The suite runs each workload five times and reports the fastest, on the theory that the other four got interrupted by a browser, a compiler, or the collector arriving at a bad moment.

It’s not statistically pure and I won’t pretend it is. It’s a lower bound on the cost of the work, measured on a machine that’s also doing other things. Allocation is the stable half and stays within a few percent; wall clock moves 10–20% between sessions. Repeat it and get a different number and you’ve found the noise, not an error. Be suspicious of anyone who reports wall clock figures without it.

Allocation, not only time  #

Guile’s collector is good enough that you can write a parser allocating a megabyte per kilobyte of input and still call it fast, right up until it’s in a server serving several documents at once. So the suite measures the heap too. The heap is collected before each workload, and what’s reported is heap-total-allocated: everything the workload asked for, including garbage it dropped immediately. That’s allocation pressure, the thing that triggers collections, not the memory a result keeps.

The last column is Guile’s gc-time-taken, which is collector processor time. Guile sums its marker threads, so one full collection of a big heap can be reported as several times its wall clock cost on a many-core machine. Read it as how much collector work a workload causes, not as a share of the clock. Same note is in the benchmark’s docstring, because an easy-to-misread number will get misread.

Fragments and the collapse pass  #

The last measurement in the suite is the one that eventually explained everything: how many string fragments a parse hands to the pass that merges them, and how long the merging takes.

Take a pattern written as (* (and (not-followed-by eol) peg-any)), meaning “consume characters until the newline”. The compiled pattern produces one parse object and one string fragment per character. A line of n characters costs n one-character substrings, n list cells, and a merge over all of them. On a document of short lines nobody notices; on a 45 KB line it’s the whole program.

The fragment table made that visible with no profiling at all.

Document Fragments handed to the collapse pass Collapse (ms)
Feature fixture (1.8 KB) 82 0.04
README.org (58 KB) 1,264 0.39
Plain prose (32 KB) 400 0.22
One 45 KB line 1 0.005

The last row is the interesting one. Forty-five kilobytes of text arrive as a single fragment, because a run scanner took them in one substring instead of forty-five thousand. The README, 58 KB of input, arrives as about 1,300 fragments — roughly one per line of prose instead of one per character.

What the profile said  #

Veritas profiles every benchmark with Guile’s statprof, and that’s what turned a vague feeling that big files were slow into a statement about one procedure.

I fed the old parser a document made of one 40 KB line and looked at the profile.

%     cumulative   self              procedure
 97.55      1.77      1.77  list?
  0.61    115.98      0.01  ice-9/boot-9.scm:220:5:map1
  0.61      0.03      0.01  orgwebalchemy/peg.scm:339:0
  0.61      0.01      0.01  reverse
  0.00  23786.61      0.00  ice-9/peg/simplify-tree.scm:48:0:string-group
  0.00      3.56      0.00  ice-9/peg/simplify-tree.scm:65:0:string-collapse

97.5% of the time is inside list?. Not the grammar, not the renderers, not any of my own code: one predicate, called by Guile’s own tree simplifier.

My hypothesis was that the PEG engine was slow. The truth was a helper in (ice-9 peg simplify-tree) asking (list? lst) of the rest of the list at every level of its recursion, which walks the list quadratically.

Here is the relevant part of string-group.

( define ( string-group lst)
  ( if (not (list? lst))        ;;  <- O(length) at every level
      lst
      ( if (null? lst)
          '()
          ( let ((next (string-group (cdr lst))))    ;;  non-tail recursion
            ...))))
copy

For a run of n fragments that’s about n²/2 list steps. A line of 40,000 characters means 800 million of them, which is exactly the 1.77 seconds in the profile. Doubling the line quadruples the bill, and the measurements agree: 20 KB took about 420 ms, 40 KB about 1,760 ms.

So there were two problems and only one was mine. My grammar produced one fragment per character, handing the quadratic pass an enormous run to chew on. And the collapse pass was quadratic in the length of a run, so even a modest run cost more than it should. Fix only the first and a slow path waits for anyone who writes a long line; fix only the second and you still allocate megabytes for nothing. Both had to go.

The two rounds of repair  #

First round: fast paths in the grammar  #

The first round was micro-optimization of the grammar. It helped, though not where I expected.

The scheme alternation used to be fifteen whole-string comparisons.

(define-peg-pattern url-scheme all
                    ( or  "http://"  "https://"  "ftp://"  "ftps://"
                         "file://"  "ssh://"  "sftp://"  "ws://"  "wss://"
                         "git://"  "gopher://"  "tel:"  "mailto:"
                         "irc://"  "ircs://"))
copy

Grouping the alternatives by first character means a position that can’t begin a scheme fails after one comparison instead of fifteen.

(define-peg-pattern url-scheme all
                    ( or ( and  "h" ( or  "ttps://"  "ttp://"))
                        ( and  "f" ( or  "tps://"  "tp://"  "ile://"))
                        ( and  "s" ( or  "sh://"  "ftp://"))
                        ( and  "w" ( or  "ss://"  "s://"))
                        ( and  "g" ( or  "opher://"  "it://"))
                         "tel:"
                         "mailto:"
                        ( and  "i" ( or  "rcs://"  "rc://"))))
copy

The alternation of inline constructs was badly ordered too. It tried the multi-character constructs, footnotes, links and the scheme scan before the single-character markers, so a line full of *bold* and /italic/ paid for the expensive branches first. Putting the one-character markers first and the link and URL machinery last costs nothing.

(define-peg-pattern inline-special body
                    ( or inline-bold inline-italic inline-code inline-verbatim
                        inline-underline inline-strike
                        inline-superscript inline-subscript
                        inline-line-break inline-escape
                        inline-footnote-inline inline-footnote-def inline-footnote-ref
                        inline-link-with-desc inline-link-no-desc
                        inline-raw-url))
copy

I also added a hand-written class for plain characters, so ordinary text gets taken with a few range tests instead of a not-followed-by check against the whole inline grammar at every position.

The result: the inline benchmark doubled, the document benchmarks improved by about a third, and the single-line case didn’t move at all. That last bit was the useful part, because it said the remaining cost wasn’t in the grammar.

Second round: run scanners  #

A compiled PEG pattern in Guile is just a procedure of (string length position) returning (end . body) or #f. Nothing stops you writing that procedure by hand and naming it in the grammar like any other pattern. That’s the whole idea of the run scanners: find the end of the run in one C-level scan with string-index and return a single substring.

( define ( run-end str len pos stop-chars stop?)
   "Return the end of the run from POS that stops at the first character
of STOP-CHARS at which STOP? holds, or at LEN."
  ( let  loop ((i pos))
    ( let ((candidate ( or (string-index str stop-chars i len) len)))
      ( if ( or (>= candidate len) (stop? str len candidate))
          candidate
          (loop (1+ candidate))))))

( define ( text-run str len pos stop-chars stop?)
  ( let ((end (run-end str len pos stop-chars stop?)))
    (list end (substring str pos end))))

( define ( line-text-scanner str len pos)
   "Consume the rest of the line as one string."
  (text-run str len pos nl-chars always-stop))

(define-peg-pattern line-content all line-text-scanner)
copy

There are thirteen such scanners now, one for each construct whose body is “text up to a delimiter”: a line, a table cell, a link target, a link description, a footnote body, a keyword value, the body of a source block, a description key, the body of a raw URL.

The delicate part is the stop? predicate. For a line, every newline really does end the run. For inline text, an asterisk ends the run only if inline-special really matches there; otherwise it’s an asterisk in the middle of a sentence. So the scanner stops at every candidate, asks the grammar whether something begins there, and if not, continues past that character. Still one string-index call per candidate, and there’s at most one candidate per markup character in a line.

Thirteen scanners would be a maintenance problem if each had its own idea of where text ends, so the character sets live in one place, each with a comment saying who stops there.

( define  nl-chars (char-set #\newline #\return))
( define  table-stop-chars (char-set #\| #\newline #\return))
( define  inline-stop-chars
   ;;  every character that can begin an inline construct: the markup markers,
   ;;  the letters a raw URL scheme can start with, and the newline
  (char-set #\* #\/ #\~ #\= #\_ #\+ #\^ #\\ #\[
            #\h #\f #\s #\w #\g #\t #\m #\i #\r
            #\newline #\return))
( define  link-stop-chars (char-set #\] #\newline #\return))
copy

Second round, second part: a linear collapse pass  #

The other half was dropping the quadratic string-collapse. The library now has its own collapse-tree, keeping the exact contract of Guile’s procedure: children collapsed before their parent, a maximal run of adjacent strings concatenated, a list of strings not joining the run around it, a one-element result unwrapped. The difference is that grouping happens in one pass with one reverse.

( define ( string-group-linear lst)
  ( let  loop ((nodes lst) (run '()) (out '()))
    ( cond
      ((null? nodes)
       (reverse ( if (null? run) out (cons (reverse run) out))))
      ((string? (car nodes))
       (loop (cdr nodes) (cons (car nodes) run) out))
      ( else
       (loop (cdr nodes) '()
             ( if (null? run)
                 (cons (car nodes) out)
                 (cons (car nodes) (cons (reverse run) out))))))))
copy

Same contract, not merely a similar one, so it produces identical trees. That’s what let me prove the rewrite safe instead of hoping it was. On its own it’s about 530× faster at 32,000 fragments.

The measurements  #

Everything below was measured in the project’s Guix shell with Guile 3.0.9, on the Ryzen 5 7640U laptop I write on, with a browser open. Five runs per workload, fastest reported. Three columns: before any optimization, after the grammar fast paths, and now.

Workload Before (µs) Fast paths (µs) Now (µs) Total speed-up
Parse the feature fixture (1.8 KB) 790 590 320 2.5×
Parse README.org (58 KB) 32,200 23,600 5,070 6.4×
Render the fixture to HTML 3,480 2,360 1,840 1.9×
Render the fixture to Markdown 3,090 2,250 1,690 1.8×
Parse a 50-clause inline paragraph 6,010 3,070 2,270 2.7×

The allocation table tells the same story from the other side.

Workload Before (KiB/op) Fast paths (KiB/op) Now (KiB/op)
Parse the feature fixture 900 620 230
Parse README.org 30,300 19,700 3,890
Render the fixture to HTML 3,310 2,200 1,360
Render the fixture to Markdown 3,140 2,160 1,360
Parse the 50-clause inline paragraph 5,660 2,740 1,690

Parsing the 58 KB README now allocates about 68 bytes per input byte, where it used to allocate roughly 530. That’s the difference between a parse that reliably triggers a collection of its own and one that often doesn’t.

One long line: from eight seconds to a third of a millisecond  #

The clearest way to see the quadratic is to hold everything else constant and grow a single line. Here’s a document whose entire content is one line of plain prose.

Line length Before (ms) Fast paths (ms) Now (ms)
5 KB 26.4 26.3 0.07
10 KB 109 105 0.08
20 KB 423 408 0.12
40 KB 1,760 1,630 0.22
80 KB 8,180 7,110 0.37

Read the first two columns: each doubling of the line multiplies the time by four. That’s a quadratic curve, and the grammar fast paths barely dented it, because per-character fragments and a quadratic merge are exactly what grammar fast paths don’t touch. Now the third column: each doubling roughly doubles the time, because string-index runs at memory speed and there’s one fragment per line instead of one per character.

At 80 KB that’s a factor of about 22,000. That’s the extreme end of the curve, though. The realistic headline is the 6.4× on the README, where the time is spread over a whole document instead of concentrated in one pathological line.

The collapse pass on its own  #

To separate the two effects I benchmarked Guile’s string-collapse against the library’s collapse-tree on plain lists of string fragments, with no parser and no renderers involved.

Fragments string-collapse (ms) collapse-tree (ms)
2,000 3.8 0.09
4,000 15.4 0.17
8,000 58.6 0.37
16,000 241 0.72
32,000 1,000 1.9

Four times per doubling on the left, two on the right. That’s the whole profile in one table.

What the profile looks like now  #

The point of a rewrite is that the profile stops pointing at one thing. Parsing the README two hundred times now gives this.

%     cumulative   self              procedure
 26.32      0.75      0.75  %after-gc-thunk
  8.27      0.41      0.24  string-append
  6.02      0.17      0.17  string=
  4.51      0.24      0.13  orgwebalchemy/peg.scm:99:0
  4.51      0.15      0.13  reverse
  3.76      0.15      0.11  string-group-linear
  3.01      1.43      0.09  orgwebalchemy/peg.scm:375:0
  2.26      1.86      0.06  collapse-tree

The biggest entry now is the collector itself, about a quarter of the time. Then the string work that building a tree implies, then the collapse pass at a few percent, then a long tail of small pattern procedures. No more 97%. Further gains have to come from allocating less, not from finding another bottleneck.

Why you can believe the measurements  #

Speed is easy to claim and easy to fake, because you can make any parser faster by making it wrong. Since the work touched the scanners that decide where text ends, I needed a way to show the language hadn’t changed.

Equivalence with the previous parser  #

The test suites can’t express “the parser still does what it did before”, because they encode what it ought to do. So there’s a separate check that loads the old parser and the new one into one process — the old one under renamed modules pulled out with git show REV:src/orgwebalchemy/peg.scm and its neighbors — and compares six results for every input: the document tree, the inline tree, and the HTML and Markdown each renderer writes.

The scanner rewrite was checked this way on about 70,000 inputs. Those included every pair of interesting characters (printable ASCII, tab, newline, carriage return, the other control characters and two multi-byte characters) as documents, as inline text, and inside a link, a footnote and a description list; the README; the test fixtures; hand-written edge cases such as markup at every position, tables with empty cells and lines without a newline; and about 6,000 randomly generated documents.

The empty-value change was checked again on about 38,000 of those inputs. It reported about 230 differences, every one of them a change I’d made on purpose. More interesting: it found no input that raised an error on the new side, where the old code raised on 21 of them, each a table row with no cells in it. That’s how a bug fix should show up — as a difference you can name and explain.

The table change was checked on about 38,000 inputs and found 16 that parse differently. All 16 were table lines whose last bar is followed by nothing but the end of the line, which is exactly what changed.

The stop set  #

A run scanner may leap over a character only if it’s sure that character can’t begin a construct. Get that set wrong by one character and the scanner silently swallows the beginning of a link or a footnote, and no equivalence test against the previous parser would catch it, because the previous parser had no scanners at all.

So inline-stop-chars is checked against the grammar itself: at every position where inline-special really matches, the character that begins it must be in the set. Sweeping about 390,000 positions of the README, the fixtures, every pair of bytes and some random markup found about 6,000 constructs and no gap.

Three deliberate differences  #

Three differences came out of these runs, all now pinned down by tests.

  1. A run of two or more #+CAPTION: or #+NAME: lines before an element that isn’t a figure used to come out of the folding step in reverse order. It now keeps document order, which is what the docs always claimed and what a reader expects of anything called a sequence.
  2. The text of something empty is now the empty string, instead of the bare tag symbol an empty match leaves behind. This is what keeps an empty cell in its column, keeps a source block without a language rendering as a source block, and keeps an empty keyword, comment, fixed-width line, footnote or block in the document instead of letting it vanish.
  3. A table row is read by its bars, as described above.

Everything else produces identical output, and the naming feature is an addition to the language, so it’s covered by tests rather than equivalence.

A trap when adding tests  #

One trap is worth recording, because I lost an afternoon to it. Veritas collects a test’s assertions by filtering the body for procedures, so assertions wrapped in a let, a when or a parameterize are silently reduced to the last one. Keep every assertion at the top level of the test body, and call a helper when you need a parameterization. The tests for the class parameters show the pattern.

What is still expensive  #

Honesty is part of the design, so here’s what these numbers don’t say.

  • Inline parsing is the most expensive stage per byte. A 50-clause paragraph of inline markup, 4.1 KB, takes 2.3 ms, about 1.8 MB per second, against 11 MB per second for a whole document of prose. Some of that is inherent in a grammar where every position has to try the alternatives. Still, it’s the number I’d attack next.
  • Allocation is still dominated by the tree. About 68 bytes of allocation per input byte, and a quarter of the parse is the collector. A cheaper node representation, or flattening parts of the tree while parsing, would buy real time.
  • There’s no incremental path. A changed document is parsed from the beginning. For an editor or a live preview, that’s the feature that would matter most, and the one I most want to write.
  • The benchmarks measure this implementation on its own. They don’t compare it with other Org implementations, and they don’t measure the incremental case. Both would be welcome additions.

What Org has that this library does not  #

The manual has a “Current limitations” section so this article doesn’t have to pretend. The known gaps:

  • Captions on non-figures. A #+CAPTION: before a table, a source block or a block quote is still an ordinary keyword line, because those captions aren’t rendered yet. Figures are the exception, and they’re done.
  • Bare superscripts and subscripts. Only the braced forms, x^{2} and H_{2}O, are recognized. Bare forms like x^2 are not, deliberately, since they can’t always be told apart from ordinary text.
  • Babel, the agenda, the export dispatcher. OrgWebAlchemy is a library, not an Emacs package. It parses Org; it doesn’t execute code blocks and doesn’t hook into Org’s export machinery.
  • The long tail. Org has years of conventions in it: #+ATTR_HTML, #+RESULTS, column width cookies, archived subtrees, custom TODO sequences read from the document, and more. Each one needs a decision about whether to parse it, render it or leave it as text, and this library would rather leave it as text than guess.

If one of these gets in your way, that’s exactly the kind of report worth sending, ideally with the smallest Org snippet that shows the problem.

Distribution  #

A library nobody can install is a library nobody uses, so packaging caught up too.

  • Guix. A self-contained package recipe is in the manual, pinned at version 1.2.1 with its hash, and the project is on its way into Guix itself as guile-orgwebalchemy.
  • Nix. A flake.nix and a default.nix build the working tree with the same derivation as the nixpkgs package, so a change that breaks one breaks the other for the same reason. They cover x86_64 and aarch64, on GNU/Linux and on Darwin.
  • Continuous integration. Pushes to trunk trigger a Woodpecker pipeline that submits the maak-test and maak-deploy tasks to Byggsteg, my own CI system, itself written in Scheme.
  • Documentation. The manual is the library’s README, and the API reference is generated from the docstrings in the source with guile-documenta: the OrgWebAlchemy API documentation.

Reproduce the measurements  #

Every number here comes from the repository and from commands the manual documents. If you want to check my work:

git clone https://codeberg.org/jjba23/orgwebalchemy.git
 cd orgwebalchemy

 #  the unit suite
guix shell -m manifest.scm -- guile -L ./src -L ./test  \
  -c  '((@ (veritas runner) run-tests))'

 #  the benchmarks (they print the tables above)
 VERITAS_TEST_TYPE=benchmark  \
guix shell -m manifest.scm -- guile -L ./src -L ./test  \
  -c  '((@ (veritas runner) run-tests))'
copy

Run them yourself. If your numbers differ a lot from mine, I’d like to know — a parser that’s fast on one laptop and slow on another has a wrong assumption hidden in it.

What I take from this  #

The lesson I keep relearning here is that correctness and speed aren’t two phases of the work. Writing the scanners forced me to state exactly which characters can begin a construct, and that list is now a specification a test sweeps about 390,000 positions. The equivalence harness that showed the rewrite was safe is also the reason I dared attempt it.

Which brings me to the larger point, the reason I care about parsing Org at all. Org mode is one of the good ideas of our time: plain text people can read and write without special software, meaning that survives the programs, grown inside a free editor where its users could extend it. Every program that can read Org makes the format a bit more useful, and a format more programs can read is one people can trust with years of their writing.

This library is a small contribution to that. It lets a Scheme program read what you wrote in Org, render it as HTML a browser can show without running anyone else’s code, or as Markdown other tools understand — under a free license, on your machine, with source you can read. The web has no shortage of ways to take a document away from its author. I’d like to add one more way to give it back.

Links  #

If you build something with OrgWebAlchemy, send a link — it’s always good to see where the library travels. And if this is worth supporting, you can buy me a cup of coffee ☕ so I can keep at it.

Saturday, October 24, 2026

Saturday, October 17, 2026

Scheme Requests for Implementation

SRFI 278: Supplemental Numerics

SRFI 278 is now in final status.

This SRFI defines procedures on Scheme numbers that aim to improve the portability and usefulness of Scheme arithmetic. These procedures include predicates with extended domains, hyperbolic and inverse hyperbolic functions, the R6RS numeric tower, and additional procedures for rounding. Also included are recommendations and justifications for mixed inexact/exact arithmetic to make complex number arithmetic more predictable.

by Peter McGoron at Saturday, October 17, 2026

Friday, October 16, 2026

Retropikzel's blog

Thursday, October 15, 2026

Scheme Requests for Implementation

SRFI 275: URIs, IRIs, and basic paths

SRFI 275 is now in final status.

This SRFI proposes a programming interface for working with RFC 3986 universal resource identifiers (URIs), as well as RFC 3987’s generalization to internationalized resource identifiers (IRIs). This document defines record types, normalization procedures, and conversion between URIs and IRIs. The proposal also specifies a basic programming interface for working with paths in isolation, as this is a pervasive usage of URIs and IRIs. Finally, we contribute a test suite to specify the behaviour of normalization with respect to relative references, which has in the past been a source of divergence between implementations.

by Duncan Guthrie at Thursday, October 15, 2026

SRFI 287: Graphs

SRFI 287 is now in draft status.

This SRFI proposes a programming interface for working with graphs, based on Martin Erwig’s functional graph library design. The FGL design takes a rather different view of graphs from most graph presentations, in that the data structures are persistent, which is a good fit for Scheme. The interface is generic to any node and edge label data-type, as long as a SRFI 128 comparator can be specified.

by Duncan Guthrie at Thursday, October 15, 2026