I made my use-cases more concrete. I think I have three:

1. downgrading a macro to a function while live-coding

2. defining functions named +,-,*,/

   • for compatibility with other lisps

   • for multi-dispatch when working with a hierarchy of mathematical types

3. so I can name locals independently from whatever might be happening in an outer scope.

I'm interested in doing design work around macro hygiene in Fennel. But, for these use-cases, that path might not be a foregone conclusion.

if the macroexpander encounters a conflict with the definition environment and the call environment, the expanded macro can't "reach over" the call-environment definition to find the original definition. It's already been shadowed by that point.

Is this still true if we take snapshots of the definition environment when encountering macro definitions? If the compiler can save a snapshot of the definition environment, associating it with the macro, wouldn't this give us a way to reach over the call-environment?

Ah, I am familiar with this hygiene problem!

For a quick fix for me, I found commenting out the assert-compile checks for this in check-binding-valid and symbol-to-expression gives me a version of Fennel that passes all tests buts three, appears to work fine, and doesn't stop me from committing these scope crimes.

I want to take a look at how other lisps have dealt with this problem, especially schemes since they don't have namespaces, and read through Fennel's implementation of macros to see what the options are. I'll try to put them together in a table of design trade-offs.

Since we don't have namespaces, but we do have tables, maybe each macro can save the environment where it's defined in a table. (I think this is the idea of syntactic closures)

I think something like the following would keep hygiene, be reasonably simple to implement, and have a Lua-esque flavor

(macro handle [def-env call-env] [x]
  `(def-env.case (. ,x :results)
     [:incomplete lines#] (def-env.print :incomplete (def-env.length lines#))
     [:error msg#] (def-env.let [handlers# (def-env.require :handlers)]
                     (handlers#.error msg#))
     [kind#] (def-env.print "unknown results!" kind#)))

then (handle (calculate-request)) would like

(let [def-env (get-macro-def-env :handle)]
  (def-env.case (. (calculate-request) :results)
     [:incomplete lines_0] (def-env.print :incomplete (def-env.length lines_0))
     [:error msg_1] (def-env.let [handlers_2 (def-env.require :handlers)]
                     (handlers_2.error msg_1))
     [kind_3] (def-env.print "unknown results!" kind_3))))

Of course there's no benefit to writing def-env everywhere, so there should be some automation there. I think a sensible default would be every symbol x in the definition becomes def-env.x and every symbol x in an expression passed to the macro at its call site becomes call-env.x.

Hey thanks for the response (and thanks for Fennel)!

This breaks referential transparency.

More specifically, this breaks equivalence under renaming bound variables. Ideally renaming bound variables would be a safe transformation that doesn't change the meaning of the code. For example, these are two ways of spelling the identity function:

(fn id1 [x] x)

vs

(fn id2 [y] y)

Whether we said x or y is a local implementation detail the rest of the code shouldn't see. And in Fennel minus macros, it is! But if we had a macro

(macro y [x] ...)

then renaming x -> y would break our code:

(fn [x] x) ; ok 
==> (fn [y] y) ; error 

The way this connects to referential transparency, if I'm understanding correctly, is both implementations refer to the identity function, but our choice of this reference is not transparent to the rest of the code.

Fortunately, this errors loudly instead of silently breaking, so I don't think there's any way the current behavior can introduce bugs.

Do you happen to remember which branch or what the hygiene issues were? I'd be interested in taking a look!

As of Fennel 1.4.2, macros cannot be shadowed. The code

(macro square-mac [x]
  `(* x x))

(fn mwe [mac]
  (let [square-mac (* mac mac)]
    (+ square-mac 1)))

Fails to compile with the error

Compile error: unknown:4:8: Compile error: local square-mac was overshadowed by a special form or macro

This breaks referential transparency. It's surprising that a name collision between a macro and a function-local variable causes a compiler error. And it makes code more brittle (if two people are working on the same file, they have to coordinate what names they reserved for macros).