This has been planned for already, but I wanted to get it into an issue so we didn't forget about it before release, and so we could discuss API options.
In order to abstract away direct usage of the fennel.metadata
API, we'll want to add some macros for getting and setting metadata. I actually have a userland implementation of these that could be pretty easily incorporated into fennel, we just need to hammer out the macro API for it.
I'm thinking something like:
(get-metadata func ?metadata-key)
/ (set-metadata func ?metadata-key value)
with-metadata
block as defined below to prevent this.:fnl/docstring
), instead does gets/sets specific key (could use feedback: could this lead to too many bugs?)(with-metadata [func-meta func] ...)
let
, but always returns the function, so named with-
to disambiguate)let
block that executes the body and returns the function. When metadata is disabled, the whole expands to the function itself.(when-metadata ...)
when
. A possible simpler alternative to with-metadata
. We could have both, but that might add cognitive debt.Implementation-wise, I already have most of what I need in these userland macros in meta-macros.fnl.
Other than changing up the API signatures to something better suited for built-ins, the only real difference is replacing the hack I used to detect whether metadata is enabled with a way for it to directly query options.useMetadata
in the macro env, since a (get-compiler-opts)
function would be useful elsewhere.
(originally from https://github.com/bakpakin/Fennel/issues/271)
I'm working on Cljlib and experimented for some time with metadata injection at runtime and wrote these two macros as a result: macros/core.fnl#L155-166
(fn with-meta [val meta] `(let [val# ,val (res# fennel#) (pcall require :fennel)] (if res# (each [k# v# (pairs ,meta)] (fennel#.metadata:set val# k# v#))) val#)) (fn meta [v] `(let [(res# fennel#) (pcall require :fennel)] (if res# (. fennel#.metadata ,v))))
with-meta
macros accepts a metadata table, and walks through it's keys callingfennel.metadata:set
on each pair.meta
accepts symbol and returns its metadata if found. You can see that I'm requiringfennel
withpcall
so this would work even in compiled Lua code when Fennel is not installed.Arbitrary way to use
with-meta
is as follows:>> (with-meta 10 {:fnl/docstring "ten"}) 10 ;; metadata was set successfully but can't be seen >> (local f (with-meta (fn [...] (let [[a b c] [...]] (+ a b c))) .. {:fnl/docstring "sum first three args" :fnl/arglist ["a" "b" "c" "..."]})) >> (doc f) (f a b c ...) sum first three args >> (f 1 2 3 4) 6And
meta
can get metadata back as a table:>> (meta f) { :fnl/docstring "sum first three args" :fnl/arglist ["a" "b" "c" "..."] }I'm currently using this to provide docstrings for symbols defined with
def
anddefonce
(macros from cljlib) in the following way:(defonce {:doc "The speed of light in m/s"} c 299792458)Which I can see later in the REPL:
>> (doc c) c The speed of light in m/sSame macro is used to generate documentation for functions in the following way:
>> (fn* plus .. "sum arbitrary amount of nums" .. ([] 0) .. ([x] x) .. ([x y] (+ x y)) .. ([x y & zs] (apply plus (+ x y) zs))) >> (doc plus) (plus [] [x] [x y] [x y & zs]) sum arbitrary amount of numsAnother example is
defmulti
:>> (defmulti fac "compute factorial" (fn [x] x)) >> (doc fac) fac compute factorial >> (defmethod fac 0 [_] 1) >> (defmethod fac :default [x] (* x (fac (- x 1)))) >> (fac 4) 24In this case,
defmulti
returns a table, with__call
methamethod, so the documentation is a good thing.I'm pretty familiar with these macros in Clojure, so I've ported these into Cljlib mostly as is. I'm not sure why original proposal was for functions only, as my experiments show that there's no reason for limiting metadata usage.
I guess, that the only concern here is that when we set metadata for arbitrary values we may have collision later, because values are not unique. Tables and functions do not have such limitation, but it also would be great if metadata could have been attached to var itself, not to a value. Documenting variables sometimes is a very good thing.
it also would be great if metadata could have been attached to var itself, not to a value
The problem is that unlike Clojure, a var is not a first-class value in Fennel or Lua. It's a compile-time construct that evaporates at runtime. So there is no storage location that would work to keep metadata.
Unless we could store this var and it's scope inside the metadata table, and
doc
and other meta-related facilities would know how to deal with it. Even if it is only compile time, since doc is a special, and meta is a macro it can get compile time representation. I'm not sure if this would work though. Maybe if storing info about the module scope?Still, supplying metadata for functions and tables is a nice feature. Not sure about strings, even though those are interned this doesn't prevent the overriding issue, as two vars which resolve to the same string may want to have different metadata.
The main limitation right now is a static analysis one. If we were able to track every value through function returns, reassignments, stuck to tables and accessed as a multisym, etc, then we could use that system to do compile-time resolution of whatever the
doc
macro is pointing to. Unfortunately, this requires a pretty high level of static analysis, and while the compiler is inching closer to having such tools built-in (walk-tree
for scanning and/or transforming an AST, plugins for hooking into certain operations), at present it would still take a lot of extra code to track a value as things are done to it.We might get there eventually, at which point compile-time metadata wouldn't be quite as reliable as runtime (due to conditional branching and such), but pretty good, and it may be possible to mix the two. The trick is doing it without adding a let of hard-to-maintain code to the compiler. As we build out the available plugin hooks, that might change.
the thing is, that I don't think that metadata should be attached to values itself. Yes Fennel doesn't have such thing as
var
, but I think that this is possible to implement it as a fully compile time construct. I'm trying to build up a prototype for this
I'm still a little fuzzy on the distinction between the proposed macros here and the
fennel.metadata
table and functions inside it. Doing these with macros instead of functions makes me think that this is somehow about injecting compile-time metadata into runtime code, or something? But I'm having a hard time following the use case that this enables.
I've actually changed my mind a bit on this topic, and I would rather have a these as a specials in the compiletime scope, rather as an "always available macros". Requiring
fennel.metadata
to useset
andget
methods is fine, although those could be a part of public API for easier use, likefennel.setmetadata
andfennel.getmetadata
.Instead, I would like these macros as a compiler-scope specials so In compile modules I could easilly do this:
(fn defn [name meta? docstring? ...] ..... (if (table? meta?) (with-meta meta? name .....) .....))
Which means that I would not need to require
fennel
at to accessfennel.metadata:set
. I already do something like this in fennel-cljlib/macros.fnl#L51-55, and then use it as follows. So this is not a big issue for me I guess, as requiring fennel at compile time is OK, as it's always present.
What I had in mind initially however, is some kind of a special that would allow me to attach metadata without needing to embed fennel compiler into my application with
--require-as-include
option, or depend on fennel compiler being available forrequire
. Similarly to howfn
already does this for docstrings.For example, here I define a macro
with-meta
, which requires Fennel at runtime and, if meta was enabled, attaches metadata to a symbol. I then use it in thefn*
macro, attaching custom arglist and docstring metadata to the function. I have to do that, because I'm generating a non-standard arglist docsting for multi-arity functions, and I wish this process could be easier.Of course, if
fn
directly supported metadata in it's syntax, something like(fn {:fnl/arglist "[x & xs]"} foo [...] (let [[x & xs] [...]] (values x xs)))
I would not need such macros at all in this exact case, as I could just usefn
's capabilities. But I might need this for creatinglocal
s as I do fordefmulti
macroSo my request would be instead to support rich metadata literally in all binding forms, namely
local
,var
, andfn
. The bindings inlet
doesn't really need metadata support I guess, but if we could have that, it would be great to. Unfortunately I don't see any way to do it without introducing a specific syntax for metadata tables, like^{}
. Here are syntax ideas for each first three:
fn
-(fn {:fnl/arglist "a b"} foo [a b private-arg] .....)
local
, andvar
-(local {:fnl/docstring "a table"} t {})
, same withvar
.If we can't add literal support, then specials, that operate at compile time would be good enough.