Add lexically-scoped, non-hygienic macros #239

jpellegrini · 2021-07-27T14:18:05Z

This implements global and local lexically-scoped macros for STklos (the global macros are kept in modules and are used in conjunction with the local ones, it was easier to do this than mix with the already existent macro system).

This does not change define-macro. It's a whole new machinery added.

Not ready yet - I thought I had fixed loading of files, but it still doesn't work (macros are not expanded when a file is being loaded, don't know why).

I'm sending the PR so you can take a look.

I'll keep working on it.

Two mechanisms are added:

%define-syntax, which defines global macros that are
only visible in the module where they are created;
%let-syntax, which defines local macros.

This commit doe NOT yet allow one to use internal defines as
if they were %let-syntax. This would probably require us to
change the definition of %define-syntax from a macro (it is
defined with define-macro) to a compiler special form. Not
hard, but not yet done.

The compile function is wrapped in a call to %%macroexpand,
so macros -- local and global -- are expanded before anything
happens.

The changes to the compiler were kept to the minimum required.

There are some tests included.

Below is a more detailed description of the changes.

ENVIRONMENT:

The representation of the environment by the compiler has been
augmented: it used to be a list of lists of symbols, where each
inner list represented one level of lexical scope.
It is now essentially the same, except that instead of a symbol,
we may also store a pair there:

x (a symbol) means x is bound in the lexical level
(x . obj) (a pair) means x is bound to a lexically-scoped
macro, and obj is a <syntax> structure.

So we make symbol-in-env? return elements, and rename it.
Symbol-in-env? (which returned either #t or #f) now is a
procedure that returns:

#f
the symbol searched (if it's not a macro)
a pair containing a macro name and its expander

A question that could arise is: why not represent all identifiers as pairs?
Answering: that would require several changes to the compiler. A less invasive
approach was used: wherever (symbol-in-env? ...) was used, the
new function works without any further adaptation on the compiler.

GLOBAL MACROS

A global macro needs to be DEFINEd first, so it will have been
computed and can be stored in a global variable. So its value will
be a <syntax> structure, containing the expander.
When the compiler finds a global variable of this type, it just
expands and compiles the expanded form.

Scheme implementation usually do not allow SET!-ing a variable that
is bound to a macro transformer, but it may be interesting to be
able to change the transformer of a macro.

When the compiler finds a global macro being used,

(define-syntax f ...)
...
(f ...)   <= f is bound to a <syntax> structure!

then it will call f's expander on that form and call itself
recursively.

The new macros are stored as ordinary variables, so they belong to
modules, and can be exported as module symbols.

LOCAL MACROS

A local macro needs to have its identifier inserted in the same
stack structure that holds the names of variables, so lexical
scope works as intended (identifiers for variables -- used at execution
time -- and macros -- expanded at compile time -- may shadow each other).
Also, local macros defined in outermost code should be available for use
in macro code in innermost code:

(define-syntax h ...)

(let-syntax ((g ...))
  ...
    (let-syntax ((f ... ,(g ...) ...))  <= f may refer to g and h

The local macro name is kept on the environment stack by the compiler
as a pair (not a symbol), so that it's clear it's a macro. The pair
is (name macro-obj), where macro-obj is an instance of <syntax>

When the compiler finds a reference to an identifier,

if it is stored as a symbol, compile reference-to-variable
if it is stored as a pair, run the expander (which is stored
along with the macro name), then recursively run the compiler.

HOW EXPANDERS ARE COMPILED

A macro expander should use the environment in which it was defined.
So, compiled-expander should be compiled as

(eval expander module)

When f makes a reference to a macro not in its local environment,
then the global environment should be used. But since we have modules,
the global environment is not unique, so the expander is compiled in
the module where the macro was defined.

References to other local macros are dealt with when expanding: the
compiler expands the car of a form and starts over, with the same environment.

If a global macro has been defined, then it is available as the value
of a global variable, in a module. So, since a local macro must
have access to that global macro, we computed the expander calling
eval in it, on that module (as mentioned before).

Caveat: it is possible to change the content of the global macro
after the local macro has been defined, but before it has been
used.

HYGIENE/RENAMING

We DO NOT yet deal with those issues here. They can be dealt with
separatelty later, maybe with the help of an ALIAS compiler macro.

REMAINING ISSUES

This doesn't seem to work for compiled files. If we compile a
file that creates and then uses a macro, STklos fails to expand it.

jpellegrini · 2021-07-27T14:20:23Z

The only thing that I didn't understand is why compiling this file with stklos-compiler doesn't work (the macro is not expanded, and STklos tries to call the <syntax> object as if it were a procedure):

(define-module test-hygienic-macros)
(select-module test-hygienic-macros)

(%define-syntax test-syntax-1 (lambda form -1))

(format #t "Here: ~a~%" (test-syntax-1))

jpellegrini · 2021-07-27T14:33:57Z

I'll see about implementing explicitly-renaming macros and syntax-case (then syntax-rules comes for free- I think there is even an implementation of it on top of syntax-case somewhere in STklos :)

lassik · 2021-07-27T17:03:25Z

Hygienic macros are not my area of expertise, but please compare this PR with the catalog of existing Scheme macro systems in SRFI 211. In particular, I recall Marc said that "the low-level macro facility of the R4RS" can be used as primitives on top of which every hygienic macro system can be built.

My impression is that in Scheme define-macro is traditionally non-hygienic.

jpellegrini · 2021-07-27T17:43:38Z

Hello @lassik !

I recall Marc said that "the low-level macro facility of the R4RS" can be used as primitives on top of which every hygienic macro system can be built.

Yes, I have read it, I like that SRFI a lot (while researching what options to consider, I also read the HOPL paper by Will Clinger and Mitchell Wand).
But the R4RS macro facility requires a bit more, like adding procedures dealing with information on each datum etc, and that would mean a bigger change in STklos. I would like to do that, but that would require some more work and would be a bit more invasive.
I think the changes to the compiled ended up being quite small, actually.

With what this PR offers, we'd have explicitly renaming macros and syntax-rules easily (I'm sure about that); and also at least a great portion of syntax-case (maybe all of it with some extra changes to the compiler - but I didn't work on that yet, so not completely sure).

It may be possible to add the R4RS macro system also, but I didn't want to add anything too disruptive for the moment.

As to define-macro not being hygienic, yes. The new thing here is that %define-syntax creates macros in modules (STklos' define-macro creates macros that are universally visible), and the new macros also interact well with the local ones created by %let-sytnax and lexically-scoped variables.

egallesio · 2021-07-27T17:44:48Z

Hello @jpellegrini,

I have just added some comment in PR #55, hoping that it will help you for the aspects of loading compiled files. Unfortunately, I will not be able to have a look at this PR before several days. But I'm really looking forward to working on it.
In the meantime, thanks a lot for this contribution.

egallesio · 2021-07-27T17:50:42Z

As to define-macro not being hygienic, yes. The new thing here is that %define-syntax creates macros in modules (STklos' define-macro creates macros that are universally visible), and the new macros also interact well with the local ones created by %let-sytnax and lexically-scoped variables.

They are universally visible if you do not compile their definition file. If you compile a file and do not export it, it is not visible. This point is really not terrible, since it introduces a difference between a compiled and a source file. Having a coherent behaviour would be really great.!

jpellegrini · 2021-07-27T18:09:30Z

Having a coherent behaviour would be really great.!

Well, having macros in modules also paves the way for having an R7RS-compliant library system :)

jpellegrini · 2021-08-21T21:11:56Z

Hi @egallesio - I'll get these new macro ideas implemented, but it will take a while, because I'm considering different possibilities do as to not be too intrusive in the STklos compiler. So I suppose version 1.70 would be released without this.

egallesio · 2021-08-22T17:54:55Z

Hi @jpellegrini,

I don't really know when I'll release 1.70 (and what it should contain, even, if we have probably enough, or nearly enough, stuff to make it now). Don't worry about that, work at your own pace. After all, we don't have a deadline fixed by the marketing team 😉

jpellegrini · 2022-01-11T02:01:09Z

I'll try to get back to this now...

Two mechanisms are added: * `%define-syntax`, which defines global macros that are only visible in the module where they are created; * `%let-syntax`, which defines local macros. This commit doe NOT yet allow one to use internal defines as if they were `%let-syntax`. This would probably require us to change the definition of `%define-syntax` from a macro (it is defined with `define-macro`) to a compiler special form. Not hard, but not yet done. The `compile` function is wrapped in a call to `%%macroexpand`, so macros -- local and global -- are expanded before anything happens. The changes to the compiler were kept to the minimum required. There are some tests included. Below is a more detailed description of the changes. ENVIRONMENT: The representation of the environment by the compiler has been augmented: it used to be a list of lists of symbols, where each inner list represented one level of lexical scope. It is now essentially the same, except that instead of a symbol, we may also store a pair there: - `x` (a symbol) means `x` is bound in the lexical level - `(x . obj)` (a pair) means `x` is bound to a lexically-scoped macro, and `obj` is a `<syntax>` structure. So we make symbol-in-env? return elements, and rename it. Symbol-in-env? (which returned either #t or #f) now is a procedure that returns: - #f - the symbol searched (if it's not a macro) - a pair containing a macro name and its expander A question that could arise is: why not represent all identifiers as pairs? Answering: that would require several changes to the compiler. A less invasive approach was used: wherever (symbol-in-env? ...) was used, the new function works without any further adaptation on the compiler. GLOBAL MACROS A global macro needs to be `DEFINE`d first, so it will have been computed and can be stored in a global variable. So its value will be a `<syntax>` structure, containing the expander. When the compiler finds a global variable of this type, it just expands and compiles the expanded form. Scheme implementation usually do not allow `SET!`-ing a variable that is bound to a macro transformer, but it may be interesting to be able to change the transformer of a macro. When the compiler finds a global macro being used, ``` (define-syntax f ...) ... (f ...) <= f is bound to a <syntax> structure! ``` then it will call `f`'s expander on that form and call itself recursively. The new macros are stored *as ordinary variables*, so they belong to modules, and can be exported as module symbols. LOCAL MACROS A local macro needs to have its identifier inserted in the same stack structure that holds the names of variables, so lexical scope works as intended (identifiers for variables -- used at execution time -- and macros -- expanded at compile time -- may shadow each other). Also, local macros defined in outermost code should be available for use in macro code in innermost code: ``` (define-syntax h ...) (let-syntax ((g ...)) ... (let-syntax ((f ... ,(g ...) ...)) <= f may refer to g and h ``` The local macro name is kept on the environment stack by the compiler as a pair (not a symbol), so that it's clear it's a macro. The pair is `(name macro-obj)`, where macro-obj is an instance of `<syntax>` When the compiler finds a reference to an identifier, - if it is stored as a symbol, compile reference-to-variable - if it is stored as a pair, run the expander (which is stored along with the macro name), then recursively run the compiler. HOW EXPANDERS ARE COMPILED A macro expander should use the environment in which it was defined. So, `compiled-expander` should be compiled as ``` (eval expander module) ``` When `f` makes a reference to a macro not in its local environment, then the global environment should be used. But since we have modules, the global environment is not unique, so the expander is compiled in the module where the macro was defined. References to other local macros are dealt with when expanding: the compiler expands the `car` of a form and starts over, with the same environment. If a global macro has been defined, then it is available as the value of a global variable, _in a module_. So, since a local macro must have access to that global macro, we computed the expander calling `eval` in it, *on that module* (as mentioned before). Caveat: it is possible to change the content of the global macro after the local macro has been defined, but before it has been used. HYGIENE/RENAMING We _DO NOT_ yet deal with those issues here. They can be dealt with separatelty later, maybe with the help of an `ALIAS` compiler macro. REMAINING ISSUES This doesn't seem to work for compiled files. If we compile a file that creates and then uses a macro, STklos fails to expand it.

egallesio · 2022-04-27T14:31:51Z

Hello @jpellegrini,

I have worked exclusively on macros for the last three weeks and I have now macros that are no more globally exported. That means that a file can define a macro and export it or not. I had a lot of problems to be able to bootstrap the system. However, it is seems to be OK now. Macros can be compiled and importing a file, will eventually rebuild only the macros of the .ostk to macro-expand the expressions which will be compiled in another file (rather than a producing a funcall).

For the moment, macros are global only, but I can use pieces of your PR to integrate local macros. The problem, is that environments are no more lists, buts structs (I used this to bootstrap the system; this is no more necessary now). Going back to the list representation is possible, and probably suitable, but necessitates another bootstrap (easier this time).

As soon as I have something stable, I'll commit it.

jpellegrini · 2022-04-27T14:57:34Z

I have now macros that are no more globally exported

Wow, that's great news! I'm sorry I couldn't help more. I'm still somewhat slow but I plan to get back to contributing to STklos.
So I'll wait for you to finish the macro cleanup so I can work on the hygienic macros.

@jpellegrini

Tests are are an adaptation of the tests from @jpellegrini PR #239

egallesio · 2022-06-22T14:58:08Z

Hi @jpellegrini,

I have finally committed anew implementation for macros.

Macros can be local to a module and may or may-note exported. We also have local macros, similar to the one you implemented. The implementation is quite different of the one you have to allow the production of .ostk file.

In fact, the compiler knows now the module it is compiling with compiler-current-module. Eventually import will create non-instantiated modules which only contains macros. So when you compile (define-module M ...), compiler-current-module can be different from current-module, and if we compile (import M1), the macros of module M1 will be added to the non-instantiated module M.

All that stuff seems to be correct for global macros. For local macros, I have still a problem (and I have spent several days on it without success). For instance,

(let ((x 'outer))
  (%let-syntax ((m (lambda () `x)))
    (let ((x 'inner))
      (cons x (m)))))

produces (inner . inner), whereas it should (probably) produce (inner . outer). I say probably because this is the value returned by the following R7 let-syntax

(let ((x 'outer))
  (let-syntax ((m (syntax-rules () ((a) x))))
    (let ((x 'inner))
      (cons x (m)))))

BTW, I have implemented a simple and quite direct let-syntax which uses %let-syntax, but it fails by returning also (inner . inner) here. I will try later to advance on this point, but not for now, since I want to purge a bit of current PR and Issues.

jpellegrini · 2022-06-22T15:11:22Z

Hi @egallesio !
That is really great news!
I'll get a closer look later!

jpellegrini · 2022-06-25T13:27:00Z

Hi @egallesio !
I see that now STklos supports redefining if, define and other special forms! This is great, and it allows the implementation of some more SRFIs.
But I didn't understand one thing: syntaxes and symbols are treated differently in global environments?

stklos> (define define print)
;; define
stklos> define                           ;; Ok, redefined the symbol
#[closure print]
stklos> (define 1 2 3)                        ;; Why does this one fail?
**** Error:
error: define: bad definition
	(type ",help" for more information)
stklos> (define-macro (define . args) `(print . ,args))
;; define
stklos> (define 1 2 3)             
123                               ;; Ok in CAR of form (application position)
stklos> define
#[syntax define]                  ;; Ok, as standalone symbol
stklos> (write define)
#[syntax define]stklos>           ;; Ok, as argument to procedure

So... Why did define not redefine the symbol completely? Its value was changed "as a standalone symbol", but when it was in application position in (define 1 2 3), it was not used. Macros won't be redefined as ordinary symbols?
I'm still begining to study the new macro code, so forgiv eme if I don't get things right.

jpellegrini · 2022-06-27T01:58:19Z

produces (inner . inner), whereas it should (probably) produce (inner . outer). I say probably because this is the value returned by the following R7 let-syntax

Ah yes -- the macro transformer should run with the environment in which it was defined if we want to have proper lexical scope (so the x seen by m is always the outer one)...
Anyway I was going to work on this part, and implement a couple of syntax systems (syntatic closures, explicitly renaming, and syntax-rules), but if you're working on it I'll wait.

jpellegrini · 2022-06-28T02:21:53Z

Hi @egallesio !
I'll tell you the idea I had...

This is compile-%let-syntax, currently:

(define (compile-%let-syntax e env tail?)
  (let ((len (length e)))
  ...
      ...
          ...
                      (let* ((name      (car new))
                             (expander  (cadr new))
                             (new-macro (%make-syntax
                                            name
                                            expander
                                            (eval expander)
                                            #f))) ; #f =>  non global macro

So using (eval expander) will not use the environment one'd expect, and the generated code would refer to the wrong references to local bindings.

I thought that the expander could hold a reference to the scope where it was defined inside the <syntax> structure, so the eval would use it (maybe transforming it into a big let* wrapping the expander), every time the expander needs to be called.

@jpellegrini

Tests are are an adaptation of the tests from @jpellegrini PR egallesio#239

jpellegrini · 2023-03-25T10:27:19Z

@egallesio as far as I can see you have already implemented lexical scope for macros -- and it's really cool that it works for define-macro. There are some small issues remaining, but I think this PR doesn't make much sense anymore, and I propose to close it -- and to open issues for what's left (the support for nested define-macro, and the problem with macros not remembering their definition environment).
Do you agree?

egallesio · 2023-03-27T14:53:11Z

That's OK for me to close it.

BTW, I have some ideas to help to circumvent the existent troubles, but I need to work a bit before being able to try.

jpellegrini · 2023-03-27T17:32:41Z

, I have some ideas to help to circumvent the existent troubles

I'll open an issue specifically for that...

jpellegrini force-pushed the lex-scope-macros branch from c1d4896 to bd4a24e Compare January 11, 2022 02:00

jpellegrini added 3 commits January 17, 2022 16:02

internal define macro

bb438ce

Typo

3bfc1e4

jpellegrini force-pushed the lex-scope-macros branch from bd4a24e to 3bfc1e4 Compare May 18, 2022 12:27

egallesio pushed a commit that referenced this pull request Jun 22, 2022

Added test on macros.

5c979be

Tests are are an adaptation of the tests from @jpellegrini PR #239

jpellegrini marked this pull request as draft February 13, 2023 15:18

jpellegrini pushed a commit to jpellegrini/STklos that referenced this pull request Feb 21, 2023

Added test on macros.

57ef387

Tests are are an adaptation of the tests from @jpellegrini PR egallesio#239

egallesio closed this Mar 27, 2023

egallesio mentioned this pull request Jan 8, 2025

symbol '%find-macro-clause' unbound with users SRFI-8 #705

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add lexically-scoped, non-hygienic macros #239

Add lexically-scoped, non-hygienic macros #239

jpellegrini commented Jul 27, 2021

jpellegrini commented Jul 27, 2021 •

edited

Loading

jpellegrini commented Jul 27, 2021

lassik commented Jul 27, 2021

jpellegrini commented Jul 27, 2021

egallesio commented Jul 27, 2021

egallesio commented Jul 27, 2021

jpellegrini commented Jul 27, 2021

jpellegrini commented Aug 21, 2021

egallesio commented Aug 22, 2021

jpellegrini commented Jan 11, 2022

egallesio commented Apr 27, 2022

jpellegrini commented Apr 27, 2022

egallesio commented Jun 22, 2022

jpellegrini commented Jun 22, 2022

jpellegrini commented Jun 25, 2022

jpellegrini commented Jun 27, 2022

jpellegrini commented Jun 28, 2022

jpellegrini commented Mar 25, 2023

egallesio commented Mar 27, 2023

jpellegrini commented Mar 27, 2023

Add lexically-scoped, non-hygienic macros #239

Add lexically-scoped, non-hygienic macros #239

Conversation

jpellegrini commented Jul 27, 2021

jpellegrini commented Jul 27, 2021 • edited Loading

jpellegrini commented Jul 27, 2021

lassik commented Jul 27, 2021

jpellegrini commented Jul 27, 2021

egallesio commented Jul 27, 2021

egallesio commented Jul 27, 2021

jpellegrini commented Jul 27, 2021

jpellegrini commented Aug 21, 2021

egallesio commented Aug 22, 2021

jpellegrini commented Jan 11, 2022

egallesio commented Apr 27, 2022

jpellegrini commented Apr 27, 2022

egallesio commented Jun 22, 2022

jpellegrini commented Jun 22, 2022

jpellegrini commented Jun 25, 2022

jpellegrini commented Jun 27, 2022

jpellegrini commented Jun 28, 2022

jpellegrini commented Mar 25, 2023

egallesio commented Mar 27, 2023

jpellegrini commented Mar 27, 2023

jpellegrini commented Jul 27, 2021 •

edited

Loading