~technomancy/fennel#2: 
Metadata macros

This has been planned for already, but I wanted to get it into an issue so we didn't forget about it before release, and so we could discuss API options.

In order to abstract away direct usage of the fennel.metadata API, we'll want to add some macros for getting and setting metadata. I actually have a userland implementation of these that could be pretty easily incorporated into fennel, we just need to hammer out the macro API for it.

#API

I'm thinking something like:

  • (get-metadata func ?metadata-key) / (set-metadata func ?metadata-key value)
    • When metadata is disabled, either a compiler error or expands to nil (could use feedback on which). Ideally used in a with-metadata block as defined below to prevent this.
    • When metadata is enabled, expands to the code that gets/sets the metadata obect so it can be worked with in runtime.
    • If the optional key is provided (e.g. :fnl/docstring), instead does gets/sets specific key (could use feedback: could this lead to too many bugs?)
  • (with-metadata [func-meta func] ...)
    • Accepts a function and a binding for that function's metadata (like let, but always returns the function, so named with- to disambiguate)
    • When metadata is enabled, expands to a let block that executes the body and returns the function. When metadata is disabled, the whole expands to the function itself.
  • (when-metadata ...)
    • Simply executes the body when metadata is enabled. Other than the lack of a condition, behaves like when. A possible simpler alternative to with-metadata. We could have both, but that might add cognitive debt.

#Implementation

Implementation-wise, I already have most of what I need in these userland macros in meta-macros.fnl.

Other than changing up the API signatures to something better suited for built-ins, the only real difference is replacing the hack I used to detect whether metadata is enabled with a way for it to directly query options.useMetadata in the macro env, since a (get-compiler-opts) function would be useful elsewhere.

(originally from https://github.com/bakpakin/Fennel/issues/271)

Status
REPORTED
Submitter
~technomancy
Assigned to
No-one
Submitted
1 year, 1 month ago
Updated
3 months ago
Labels
enhancement

~andreyorst 10 months ago*

I'm working on Cljlib and experimented for some time with metadata injection at runtime and wrote these two macros as a result: macros/core.fnl#L155-166

(fn with-meta [val meta]
  `(let [val# ,val
         (res# fennel#) (pcall require :fennel)]
     (if res#
         (each [k# v# (pairs ,meta)]
           (fennel#.metadata:set val# k# v#)))
     val#))

(fn meta [v]
  `(let [(res# fennel#) (pcall require :fennel)]
     (if res# (. fennel#.metadata ,v))))

with-meta macros accepts a metadata table, and walks through it's keys calling fennel.metadata:set on each pair. meta accepts symbol and returns its metadata if found. You can see that I'm requiring fennel with pcall so this would work even in compiled Lua code when Fennel is not installed.

Arbitrary way to use with-meta is as follows:

>> (with-meta 10 {:fnl/docstring "ten"})
10 ;; metadata was set successfully but can't be seen
>> (local f (with-meta (fn [...] (let [[a b c] [...]] (+ a b c)))
..                     {:fnl/docstring "sum first three args" :fnl/arglist ["a" "b" "c" "..."]}))
>> (doc f)
(f a b c ...)
  sum first three args
>> (f 1 2 3 4)
6

And meta can get metadata back as a table:

>> (meta f)
{
    :fnl/docstring "sum first three args"
    :fnl/arglist ["a" "b" "c" "..."]
}

I'm currently using this to provide docstrings for symbols defined with def and defonce (macros from cljlib) in the following way:

(defonce {:doc "The speed of light in m/s"} c 299792458)

Which I can see later in the REPL:

>> (doc c)
c
  The speed of light in m/s

Same macro is used to generate documentation for functions in the following way:

>> (fn* plus
..   "sum arbitrary amount of nums"
..   ([] 0)
..   ([x] x)
..   ([x y] (+ x y))
..   ([x y & zs] (apply plus (+ x y) zs)))
>> (doc plus)
(plus 
  [] 
  [x] 
  [x y] 
  [x y & zs])
  sum arbitrary amount of nums

Another example is defmulti:

>> (defmulti fac "compute factorial" (fn [x] x))
>> (doc fac)
fac
  compute factorial
>> (defmethod fac 0 [_] 1)
>> (defmethod fac :default [x] (* x (fac (- x 1))))
>> (fac 4)
24

In this case, defmulti returns a table, with __call methamethod, so the documentation is a good thing.

I'm pretty familiar with these macros in Clojure, so I've ported these into Cljlib mostly as is. I'm not sure why original proposal was for functions only, as my experiments show that there's no reason for limiting metadata usage.

~andreyorst 10 months ago

I guess, that the only concern here is that when we set metadata for arbitrary values we may have collision later, because values are not unique. Tables and functions do not have such limitation, but it also would be great if metadata could have been attached to var itself, not to a value. Documenting variables sometimes is a very good thing.

~technomancy 10 months ago

it also would be great if metadata could have been attached to var itself, not to a value

The problem is that unlike Clojure, a var is not a first-class value in Fennel or Lua. It's a compile-time construct that evaporates at runtime. So there is no storage location that would work to keep metadata.

~andreyorst 10 months ago

Unless we could store this var and it's scope inside the metadata table, and doc and other meta-related facilities would know how to deal with it. Even if it is only compile time, since doc is a special, and meta is a macro it can get compile time representation. I'm not sure if this would work though. Maybe if storing info about the module scope?

Still, supplying metadata for functions and tables is a nice feature. Not sure about strings, even though those are interned this doesn't prevent the overriding issue, as two vars which resolve to the same string may want to have different metadata.

~jaawerth 10 months ago*

The main limitation right now is a static analysis one. If we were able to track every value through function returns, reassignments, stuck to tables and accessed as a multisym, etc, then we could use that system to do compile-time resolution of whatever the doc macro is pointing to. Unfortunately, this requires a pretty high level of static analysis, and while the compiler is inching closer to having such tools built-in (walk-tree for scanning and/or transforming an AST, plugins for hooking into certain operations), at present it would still take a lot of extra code to track a value as things are done to it.

We might get there eventually, at which point compile-time metadata wouldn't be quite as reliable as runtime (due to conditional branching and such), but pretty good, and it may be possible to mix the two. The trick is doing it without adding a let of hard-to-maintain code to the compiler. As we build out the available plugin hooks, that might change.

~andreyorst 10 months ago

the thing is, that I don't think that metadata should be attached to values itself. Yes Fennel doesn't have such thing as var, but I think that this is possible to implement it as a fully compile time construct. I'm trying to build up a prototype for this

~technomancy 3 months ago

I'm still a little fuzzy on the distinction between the proposed macros here and the fennel.metadata table and functions inside it. Doing these with macros instead of functions makes me think that this is somehow about injecting compile-time metadata into runtime code, or something? But I'm having a hard time following the use case that this enables.

~andreyorst 3 months ago

I've actually changed my mind a bit on this topic, and I would rather have a these as a specials in the compiletime scope, rather as an "always available macros". Requiring fennel.metadata to use set and get methods is fine, although those could be a part of public API for easier use, like fennel.setmetadata and fennel.getmetadata.

Instead, I would like these macros as a compiler-scope specials so In compile modules I could easilly do this:

(fn defn [name meta? docstring? ...]
  .....
  (if (table? meta?)
      (with-meta meta? name .....)
      .....))

Which means that I would not need to require fennel at to access fennel.metadata:set. I already do something like this in fennel-cljlib/macros.fnl#L51-55, and then use it as follows. So this is not a big issue for me I guess, as requiring fennel at compile time is OK, as it's always present.


What I had in mind initially however, is some kind of a special that would allow me to attach metadata without needing to embed fennel compiler into my application with --require-as-include option, or depend on fennel compiler being available for require. Similarly to how fn already does this for docstrings.

For example, here I define a macro with-meta, which requires Fennel at runtime and, if meta was enabled, attaches metadata to a symbol. I then use it in the fn* macro, attaching custom arglist and docstring metadata to the function. I have to do that, because I'm generating a non-standard arglist docsting for multi-arity functions, and I wish this process could be easier.

Of course, if fn directly supported metadata in it's syntax, something like (fn {:fnl/arglist "[x & xs]"} foo [...] (let [[x & xs] [...]] (values x xs))) I would not need such macros at all in this exact case, as I could just use fn's capabilities. But I might need this for creating local s as I do for defmulti macro

So my request would be instead to support rich metadata literally in all binding forms, namely local, var, and fn. The bindings in let doesn't really need metadata support I guess, but if we could have that, it would be great to. Unfortunately I don't see any way to do it without introducing a specific syntax for metadata tables, like ^{}. Here are syntax ideas for each first three:

  • fn - (fn {:fnl/arglist "a b"} foo [a b private-arg] .....)
  • local, and var - (local {:fnl/docstring "a table"} t {}), same with var.

If we can't add literal support, then specials, that operate at compile time would be good enough.

Register here or Log in to comment, or comment via email.