The Macrological Fascicle

Editor’s introduction to this fascicle

Note: This introduction will not appear in the final report.

Lisp was the first language to incorporate macros operating on the syntax tree of a program, rather than the text of its source code; Scheme was the first language to add automatic hygiene to such a system. This first fascicle in the development of the seventh revision of the report on Scheme therefore builds on six decades of research and innovation in which Scheme and its predecessors have played an active role throughout. Indeed, the new features added to macros in this fascicle were all pioneered in implementations of Scheme.

Hygienic macros were first mentioned in a Scheme report in an optional appendix to the R4RS. Compared to R4RS, the macro system defined by this fascicle adds only what might be called refinements of the original concept:

• a library-based system of multiple phases of evaluation and expansion makes it possible to use one’s own helper procedures in the definition of macro transformers (added in R6RS, extended by this report);

• the high-level syntax-rules system has been unified with the low-level manipulation of syntax objects by the introduction of the syntax-case system, which makes pattern variables accessible as bindings within Scheme code (added in R6RS);

• macro uses can be uses of identifiers outside the operator position of a combination, like variable names (added in R6RS);

• the high-level pattern matching system used by both syntax-rules and syntax-case can match some more types of patterns that were unavailable in R4RS (added in R6RS and R7RS small);

• identifiers may have properties associated with them, which allows macros to pass around additional information associated with identifiers to one another during expansion (added in this report);

• syntax parameters allow certain types of seemingly unhygienic macro to be implemented without actually breaking hygiene (added in this report).

The R6RS divided its treatment of macros between the main report (which treated the high-level syntax-rules system and the binding of syntax keywords to transformers) and the report on standard libraries (which actually defined what a transformer is, as well as syntax objects and syntax-case). Neither document contained a complete explanation of the core semantics of macros and expansion alone. In contrast, since the R7RS volume on standard libraries (internally called Batteries) is intended to be entirely implementable in terms of the Foundations, this fascicle explains the entire macro system of R7RS Large. The standard libraries report of R6RS also presented the syntax-case system as a high-level system, without providing enough lower-level features to allow it to be implemented as a user library; this fascicle, however, contains sufficient primitives in the sections on syntax objects and syntax transformation to allow the syntax-case form to be implemented as derived syntax. Therefore, it is perhaps better not to speak of the R7RS macro system as being based on syntax-case as R6RS’s was, but rather based on syntax objects, as the low-level system in the appendix to the R4RS was.

Since extensions to Scheme macros beyond the syntax-rules system provided by the small language have been among the most controversial proposals made for changes in the Scheme language in the last two decades, I hope this fascicle will address these controversies by showing how the popular syntax-rules system is built up, from a theoretical hygiene model, through an implementation of the model on syntax objects processed by transformer procedures; and how finally, with the addition of the pattern matcher from syntax-rules, a specification of the entire syntax-case system is reached, with syntax-rules itself being a trivial transformation thereof. The tools provided in the first three chapters of this fascicle are sufficient to implement the high-level systems of R5RS and R6RS specified in chapters 4 and 5. A sample implementation which demonstrates this will be provided in due course.

What implementations need to do to support this fascicle

Implementations which do not support syntax-case as specified by the R6RS will need to adopt it, either by completely replacing their expanders or by adapting their existing ones. Experience from implementations which have already made the switch shows that the former is generally the easier approach in practice. A non-normative appendix to this fascicle shows how macros written for the explicit renaming system, the most common low-level implementation of macros besides syntax-case, can be accommodated by expanders written for syntax-case; alternatively, van Tonder (2006) implements a version of the syntax-case system which includes native support for a similar version of explicit renaming to that shown in the appendix.

This fascicle deprecates the R6RS’s provisions for explicit phasing. Implementations which use explicit phasing and restrict all identifier bindings to the phase at which they were created will need to switch to implicit phasing (ignoring the phase declarations on imports specified by the R6RS) and allow uses of syntax defined in previous phases.

Compared to the versions of syntax-case and syntax-rules in the R6RS, this fascicle allows renaming the ellipsis (section 4.5) as well as using the ellipsis and underscore as pattern literals, as in the small language report’s syntax-rules. Existing implementations of the pattern matcher will need to be extended to support these features.

This fascicle further extends the R6RS macro system with the addition of lexically-scoped identifier properties (section 2.5) and syntax parameters (section 2.3), both of which require support from the expander.

This fascicle also adds a number of procedures and syntax forms from the R4RS low-level macro system for which the R6RS offered no equivalent, including quote-syntax (equivalent to R6RS’s syntax without pattern variable substitution). Some of these can be implemented portably in terms of the R6RS higher-level constructs.

The final division of the R7RS Large Foundations into libraries will be decided at a later stage. The library name (r7rs-drafts macro-fascicle) is assigned to a temporary library containing all bindings specified in this fascicle, intended for experiments only. These bindings may change incompatibly if this fascicle is updated before the final report is issued. Production code should not depend on this library, and implementations should not support it any more once the final R7RS Large specification is ratified.

Changes in this fascicle compared to the source texts

The main source for the contents of this fascicle is the Yellow Ballot on macros and syntactic constructs held between October 2021 and February 2022 under the chairship of John Cowan. That ballot resulted in the adoption into R7RS Large of the R6RS Standard Libraries chapter on syntax-case, the R6RS identifier-syntax transformer specifier, and the SRFIs 139 (syntax parameters), 188 (splicing-let-syntax and splicing-letrec-syntax), and 213 (identifier properties), besides a number of other proposals which will be incorporated into future fascicles.

Compared to R7RS small and the source documents adopted under the Yellow Ballot, the following substantive changes and additions have been made:

• The R6RS hygiene model is expressed in different terms and in more detail.

• Low-level procedures and syntax forms originally from the R4RS appendix have been added to address criticism that, in R6RS, the high-level syntax-case pattern matcher offered the only tool to destructure syntax objects. Using these forms together with identifier properties, syntax-case itself can be implemented as derived syntax. In addition, the predicate symbolic-identifier=? from the R6RS examples is defined, because it is part of the operation of R7RS small cond-expand.

• The behaviour of identifier properties has been nearly completely respecified to be more explicit about how properties are attached to identifiers under export, import, and shadowing. The new specification matches the behaviour of existing implementations.

• An identifier-defined? procedure allows detecting whether a particular identifier is bound. Among other uses, this makes identifier properties more ergonomic to use in some cases, and allows macro transformers to report more useful error messages when they operate on an identifier passed to them which is supposed to already be bound.

• The R6RS provisions for explicit phasing have been dropped because they proved unpopular with implementers and users alike. The behaviour of implicit phasing with regards to the evaluation contexts of macro transformers and the visibility of identifiers in different phases is specified. A future fascicle on the library system will complete the definition of implicit phasing and mark the syntactic provision of the R6RS for explicit phasing as deprecated.

• The expansion process defined by R6RS has been adapted for the new macro features in this fascicle. The current draft also shows what the expansion process would look like if mixing definitions and expressions in any order in any body were allowed, but a final decision on this has not yet been made. In the event this change is not made to bodies other than program and library bodies, reverting to the old semantics is a relatively small revision to this part of the text.

• It has been explicitly specified that, although syntax-case and syntax-rules use free-identifier=? to find instances of literals within their input forms, they use bound-identifier=? to find literals within patterns. This reflects the consensus of the Scheme community and the current practice of implementations, following difficulties with advanced macro-defining macros when free-identifier=? is used for both purposes (Clinger and Wand 2020, sec. 14.2, though note that they incorrectly refer to matching pattern literals in uses, rather than within patterns).

• The rules for when syntax objects are wrapped vs. unwrapped as a result of evaluating a syntax expression have been changed to more accurately reflect both the intention of the original R6RS authors (expressed in SRFI 93 but omitted from the final R6RS document) and what existing implementations actually do. (E.g. under the letter of the rules according to R6RS, an expression such as #'(x ...) was not actually guaranteed to evaluate to a proper list.)

• The form custom-ellipsis has been added, allowing the ellipsis renaming feature of R7RS small syntax-rules to be implemented in terms of syntax-case.

• syntax-rules is now specified mostly in terms of the semantics of syntax-case and syntax. In particular, syntax templates inherit the R6RS feature of allowing multiple ellipses after a single pattern variable, already a common extension to the syntax-rules system of R7RS small.

• A new declarative form erroneous-syntax allows more concisely defining macros which always signal an error when expanded. Syntax keywords with this property are expected to be increasingly common for use as syntax parameters and as the keys of identifier properties, besides their existing uses as auxiliary syntax keywords.

The R6RS condition system has also been implicitly adopted, following informal consensus of WG2 members.