Fear of Macros(greghendershott.com)
greghendershott.com
Fear of Macros
http://www.greghendershott.com/fear-of-macros/index.html
7 comments
It turns out that fexprs, which is what Kernel provides, are widely considered a bad idea in the Lisp community, and were abandoned in favor of macros for a reason. The basic problem is that you can never tell whether your function call is really a function call or a macro application, until you run the program, and therefore you can't reason about your code at all. Incidentally, this also means that your compiler can't reason about your code, and thus optimization is basically impossible.
The fexprs provided by Kernel are not the same as the ones in traditional lisps which were abandoned - Shutt has developed a theory for reasoning about them in some ways (the vau-calculus, which is capable of defining the lambda calculus too). He describes pure and impure versions of the vau calculus in his dissertation.
Firstly, operatives are improved over the traditional ones by having lexical scoping - where the original lisps were dynamically scoped. The other main feature of operatives is that they gain access to the dynamic scope from where they were called - as this environment is passed implicitly to the operative when it is evaluated.
You're right that they do not allow us to reason inductively about code in the same way as the lambda calculus due to this, but it's a trade off for incredible flexibility. I think we can build alternative models to reason about them for purposes of optimisation and such. (Or alternatively, one can have operatives behave as compilers - in which you feed in type-tagged expressions, and your compiler-operatives can perform full type checking - then output an applicative which strips this type information). I see the possibilities as endless, because the model is extremely abstract.
Kernel choses not to enforce the static separation of operatives and applicatives for simplcity - instead, a convention based approach is used whereby operatives are prefixed with "$", (such as $vau, $lambda, $define!, $provide!). One can check them during runtime with the operative? and applicative? predicates too.
However, it would be perfectly possible to create a compiler which can know this difference, since the forms `($define! ($vau ...))` and `($define! ($lambda ...))` which distinguish them could be given special forms or special syntax in an alternative representation of this model, although you would not be able to tell whether an applicative uses some operatives internally - that's the idea of encapsulation - implementation information is meant to be hidden. (Even at the cost of performance, which often doesn't matter.)
Even if Kernel is not as widely practical as lambda-calculus based languages - Shutt's dissertation is definitely worth a read, and shouldn't be discarded because of a perception of bad fexprs from 40 years ago.
Firstly, operatives are improved over the traditional ones by having lexical scoping - where the original lisps were dynamically scoped. The other main feature of operatives is that they gain access to the dynamic scope from where they were called - as this environment is passed implicitly to the operative when it is evaluated.
You're right that they do not allow us to reason inductively about code in the same way as the lambda calculus due to this, but it's a trade off for incredible flexibility. I think we can build alternative models to reason about them for purposes of optimisation and such. (Or alternatively, one can have operatives behave as compilers - in which you feed in type-tagged expressions, and your compiler-operatives can perform full type checking - then output an applicative which strips this type information). I see the possibilities as endless, because the model is extremely abstract.
Kernel choses not to enforce the static separation of operatives and applicatives for simplcity - instead, a convention based approach is used whereby operatives are prefixed with "$", (such as $vau, $lambda, $define!, $provide!). One can check them during runtime with the operative? and applicative? predicates too.
However, it would be perfectly possible to create a compiler which can know this difference, since the forms `($define! ($vau ...))` and `($define! ($lambda ...))` which distinguish them could be given special forms or special syntax in an alternative representation of this model, although you would not be able to tell whether an applicative uses some operatives internally - that's the idea of encapsulation - implementation information is meant to be hidden. (Even at the cost of performance, which often doesn't matter.)
Even if Kernel is not as widely practical as lambda-calculus based languages - Shutt's dissertation is definitely worth a read, and shouldn't be discarded because of a perception of bad fexprs from 40 years ago.
EDIT: By browsing to the Github repo, I found the link to the One Big HTML page: http://www.greghendershott.com/fear-of-macros/all.html
Original post: I wish there was a single-page version of this that I could Pocket. Or an epub that I could put on my reading devices.
I don't tend to read multi-page things like this at my desk, not in this era of high-resolution lightweight portable screens.
Original post: I wish there was a single-page version of this that I could Pocket. Or an epub that I could put on my reading devices.
I don't tend to read multi-page things like this at my desk, not in this era of high-resolution lightweight portable screens.
For those who don't know, Hendershott wrote Cakewalk, the most popular music-creation software on the PC. He knows his stuff. It's also an engaging article, esp for us old C/C++ hacks who get nervous around Lispy things.
I saw Greg give a nice talk at Racket Con this year. He was really humble and talked about the challenges of writing a Markdown parser. Didn't know he wrote Cakewalk! Cool!
Don't forget that Cakewalk includes CAL - the Cakewalk Application Language, which is a bit Lisp'y, I have to say, as well as being a bit C'ish, to boot. ;)
Confession: I used s-expressions only because even I could figure out how to parse that. I didn't appreciate Lisp in any meaningful sense.
Fast forward to a few years ago. I wanted to learn a lisp, finally. I gravitated toward Scheme. PLT Scheme seemed like the best choice. Today it's known as Racket.
Fast forward to a few years ago. I wanted to learn a lisp, finally. I gravitated toward Scheme. PLT Scheme seemed like the best choice. Today it's known as Racket.
I remember putting Cakewalk through its paces in the late 80's/early 90's on my old DOS machine in the studio .. it was really a great way to get a lot of music written, and I really appreciate your aesthetics for how you managed that application over time - the arrival of CAL was a really interesting move, and as things moved so fast in the pro audio world in the 90's, it always sort of bothered me that more and more tools were not written with scripting - for the user - in mind.
My friends and I wrote all sorts of CAL scripts to 'funkify' our Drum/Rythmn tracks - I wish I still had a copy of funkify.cal around, it'd be nice to see it again.
I moved on from Cakewalk around the DOS/Windows migration, as I've never been a heavy fan of Windows, nor a real user, so I got set up with all kinds of other tools after we left the DOS machines alone .. mostly hardware sequencers.
What do you think about one day doing a hardware sequencer, Greg? With the way things are going, it seems like you could return to the world of music-composition, albeit with a hardware bent, and make some serious progress in that department .. as we know the music industry/market is heavily cyclic, soon enough there will be a return to hardware as a reliable, dependent way of composing music. Any plans in that regard? Imagine: a hardware MIDI/Audio sequencer .. with onboard SCRIPTING! :)
My friends and I wrote all sorts of CAL scripts to 'funkify' our Drum/Rythmn tracks - I wish I still had a copy of funkify.cal around, it'd be nice to see it again.
I moved on from Cakewalk around the DOS/Windows migration, as I've never been a heavy fan of Windows, nor a real user, so I got set up with all kinds of other tools after we left the DOS machines alone .. mostly hardware sequencers.
What do you think about one day doing a hardware sequencer, Greg? With the way things are going, it seems like you could return to the world of music-composition, albeit with a hardware bent, and make some serious progress in that department .. as we know the music industry/market is heavily cyclic, soon enough there will be a return to hardware as a reliable, dependent way of composing music. Any plans in that regard? Imagine: a hardware MIDI/Audio sequencer .. with onboard SCRIPTING! :)
Oh wow, I remember spending an entire year's worth of music class at school with Cakewalk and a Roland.
I don't know what they authored the documentation, but I quite like what gitbook.io is doing in this space.
Racket has a documentation system called Scribble.
Which is actually one of the most interesting features of Racket -- that it lets you implement a `#lang scribble` which does not even use s-expression syntax, but you still have the full power of Racket if you need it.
The default `#lang racket` is a wonderful modern lisp, and you can use Racket as "only" that. But Racket is also a system for making languages.
Which is actually one of the most interesting features of Racket -- that it lets you implement a `#lang scribble` which does not even use s-expression syntax, but you still have the full power of Racket if you need it.
The default `#lang racket` is a wonderful modern lisp, and you can use Racket as "only" that. But Racket is also a system for making languages.
More on Scribble:
Racketeer Mathew Flatt's video on Scribble's concept and the rationale for making it: http://vimeo.com/6630691
Racketeer Mathew Flatt's video on Scribble's concept and the rationale for making it: http://vimeo.com/6630691
Very interesting. Thanks for posting it.
i don't fear macros, but i am wary of them: when there are only a few macros, it seems as though replacing them with functions makes more sense, even if the syntax is slightly messier. when there are so many macros that you've essentially made an entirely new language, writing a REPL feels like a cleaner separation, even if it is more work.
i would be interested to see links to libraries, particularly clojure libraries, that make heavy but judicious use of macros. it totally makes sense that macros are useful because they capture part of this "code as data" idea that's so central to, esp., lisps. but having more examples of actual problems that they solve would be helpful.
i would be interested to see links to libraries, particularly clojure libraries, that make heavy but judicious use of macros. it totally makes sense that macros are useful because they capture part of this "code as data" idea that's so central to, esp., lisps. but having more examples of actual problems that they solve would be helpful.
A good example, clojure logging.
https://github.com/clojure/tools.logging
They look the same as normal logging function calls, but entirely avoid the issue of "logging guards."
I'll point you at a good link, but basically if you have a lot of calls to logger.debug, logger.trace, etc, and they involve a lot of string conversions, you pay the penalty for those string conversions all the time even when you're logging at warn or higher. This leads many java developers to do the following:
if (LOGGER.isDebugEnabled()) { LOGGER.debug("last node was: " + node); }
... which makes logging significantly heavier visually.
With a macro you are essentially adding syntax, so you can bake the guards into your log statements.
Logging guards: http://www.kdgregory.com/index.php?page=java.logging
They look the same as normal logging function calls, but entirely avoid the issue of "logging guards."
I'll point you at a good link, but basically if you have a lot of calls to logger.debug, logger.trace, etc, and they involve a lot of string conversions, you pay the penalty for those string conversions all the time even when you're logging at warn or higher. This leads many java developers to do the following:
if (LOGGER.isDebugEnabled()) { LOGGER.debug("last node was: " + node); }
... which makes logging significantly heavier visually.
With a macro you are essentially adding syntax, so you can bake the guards into your log statements.
Logging guards: http://www.kdgregory.com/index.php?page=java.logging
Although, for what it's worth, Java programmers solved this quite some time ago by allowing crude lazy evaluation of the string. Logging calls actually look like:
In Scala, you just make the parameter to the call a lazy or call-by-name value, and then there's no need to write an explicit template, the language captures the expression which produces the string and defers its execution until it's needed.
Of course, there are many things macros can do which mechanisms like these can't. The choice in language design is between introducing macros, or introducing enough other mechanisms to make macros unnecessary. Introducing macros seems more parsimonious, but given the utter confusion that macros can be used to wreak, it's not a slam dunk.
And, of course, the Scala designers have introduced all these mechanisms and then added macros anyway, because they're crazy.
LOGGER.debug("last node was: {}", node);
In Scala, you just make the parameter to the call a lazy or call-by-name value, and then there's no need to write an explicit template, the language captures the expression which produces the string and defers its execution until it's needed.
Of course, there are many things macros can do which mechanisms like these can't. The choice in language design is between introducing macros, or introducing enough other mechanisms to make macros unnecessary. Introducing macros seems more parsimonious, but given the utter confusion that macros can be used to wreak, it's not a slam dunk.
And, of course, the Scala designers have introduced all these mechanisms and then added macros anyway, because they're crazy.
[deleted]
My favorite example would have to be CLOS the Common Lisp object system. Although it's a bit hairy in certain places, it essentially transforms a procedural/functional language into on of the most powerful object oriented languages around. Macros play a small, but crucial part of it, subverting the rules of lisp in order to add another dimension to the language.
Another great example might be Postmodern and CLSQL, both of which include SQL DSLs, much better than concatinating strings or muddling around with ORMs. CL's loop macro basically ads an algol like language to lisp(if you dislike loop, there exist lispier alternatives like iterate).
Macros take care of the case between writing a function and writing an interpreter, and with judicious use, they can help build much cleaner abstractions that give a qualitative improvement to a code base.
Another great example might be Postmodern and CLSQL, both of which include SQL DSLs, much better than concatinating strings or muddling around with ORMs. CL's loop macro basically ads an algol like language to lisp(if you dislike loop, there exist lispier alternatives like iterate).
Macros take care of the case between writing a function and writing an interpreter, and with judicious use, they can help build much cleaner abstractions that give a qualitative improvement to a code base.
core.async [1] is basically the canonical example of a Clojure library that relies on macros to do its job, although surprisingly few macros actually need to be exposed in the public API. Most notable is the go macro, which performs deep-walking code transformation on its body, permitting clients to write async code in a straightforward, boilerplate-free way.
There's usually a heavy emphasis in the Clojure community on keeping macros as small as possible – the thinking is that any behavior that can be implemented with ordinary functions should be, and most macros should be lightweight wrappers over these functions. This seems to fit together well with what I've observed: functional programming idioms often circumvent some of the more common legacy uses of macros in practice, relegating macros to a still-essential but less prevalent role.
[1] https://github.com/clojure/core.async
There's usually a heavy emphasis in the Clojure community on keeping macros as small as possible – the thinking is that any behavior that can be implemented with ordinary functions should be, and most macros should be lightweight wrappers over these functions. This seems to fit together well with what I've observed: functional programming idioms often circumvent some of the more common legacy uses of macros in practice, relegating macros to a still-essential but less prevalent role.
[1] https://github.com/clojure/core.async
> the thinking is that any behavior that can be implemented with ordinary functions should be
this is what i was trying to say.
this is what i was trying to say.
All of Racket's `#lang` family of languages are implemented with macros on top of core Racket. For example, Scribble, mentioned in another comment, is the documentation language used to author the article, and the Racket docs[1]). Or Slideshow, Racket's programmatic Powerpoint alternative.
Try to add a static type system to Racket or Clojure. Oh but you don't have the ability to change the runtime system. With macros you can do it. See Typed Racket[2] or Typed Clojure[3].
[1]: http://docs.racket-lang.org/ [2]: https://github.com/plt/racket/tree/master/pkgs/typed-racket-... [3]: https://github.com/clojure/core.typed
Try to add a static type system to Racket or Clojure. Oh but you don't have the ability to change the runtime system. With macros you can do it. See Typed Racket[2] or Typed Clojure[3].
[1]: http://docs.racket-lang.org/ [2]: https://github.com/plt/racket/tree/master/pkgs/typed-racket-... [3]: https://github.com/clojure/core.typed
"particularly clojure libraries, that make heavy but judicious use of macros"
Try the Clojure standard libraries. Many central parts of the language like "->>" are macros.
Try the Clojure standard libraries. Many central parts of the language like "->>" are macros.
oh, yes, indeed, the ->> macro is slick. reminds me of the bind operator in haskell. i don't know why i haven't seen something like it in other lisps. but as you said, that's part of the language.
anyway, should clarify: i'm interested in libraries that define and then use lots of macros in order to solve a specific problem. i'd like to know what problem domains are particularly well suited to the sort of language extension that macros provide.
anyway, should clarify: i'm interested in libraries that define and then use lots of macros in order to solve a specific problem. i'd like to know what problem domains are particularly well suited to the sort of language extension that macros provide.
> but as you said, that's part of the language.
Well no since it's a macro. It's part of the standard library, not a special form part of the language.
Well no since it's a macro. It's part of the standard library, not a special form part of the language.
The point of macros is to blur the distinction between what's "built-in" and what's "user-defined" to the point of irrelevancy.
For example, threading macros in Racket:
https://github.com/greghendershott/rackjure/blob/master/rack...
I don't know Clojure, but here's a couple of good macro examples in Scheme.
Scheme's SRFI-41[0] implements Streams, which are lazily evaluated lists. A few macros are used in order to introduce lazy evaluation to the language. The Streams API wouldn't be very nice at all if no macros were used.
How about something more basic like the `let` macro? It's something you can't live without.
Whenever you need to control the order of evaluation, macros are what you want. Don't be wary of them. They are wonderful.
[0] http://srfi.schemers.org/srfi-41/srfi-41.html
Scheme's SRFI-41[0] implements Streams, which are lazily evaluated lists. A few macros are used in order to introduce lazy evaluation to the language. The Streams API wouldn't be very nice at all if no macros were used.
How about something more basic like the `let` macro? It's something you can't live without.
Whenever you need to control the order of evaluation, macros are what you want. Don't be wary of them. They are wonderful.
[0] http://srfi.schemers.org/srfi-41/srfi-41.html
[deleted]
I have gone from Unix shell script to Perl to PHP to C to C++ to Java to Python. Most of them are a lot of the same Algol-descendant syntax: if/else, for loops, while loops, variable assignment etc.
Racket Lisp is a little more of a leap. Even something like the behavior of a for loop is conceptually different. When most of the languages you have been bouncing around are more Algol-like, Lisp is a bigger leap into a different programming paradigm.
Racket Lisp is a little more of a leap. Even something like the behavior of a for loop is conceptually different. When most of the languages you have been bouncing around are more Algol-like, Lisp is a bigger leap into a different programming paradigm.
I'm not quite sure I agree with the for loop example specifically for Racket.
See: http://docs.racket-lang.org/guide/for.html
At least conceptually, for loops work very much like that in the other languages. You might be referring to Lisp in the general sense. If that's the case, remove the "Racket" parts in your comment, refer specifically to the Lisp family, and then the comment should be ok.
See: http://docs.racket-lang.org/guide/for.html
At least conceptually, for loops work very much like that in the other languages. You might be referring to Lisp in the general sense. If that's the case, remove the "Racket" parts in your comment, refer specifically to the Lisp family, and then the comment should be ok.
before this article i didn't want to touch scheme macros.
he does a great job by introducing macros using basic syntax transformers, every other tutorial failed for me by introducing define-syntax-rule before explaining how it worked.
he does a great job by introducing macros using basic syntax transformers, every other tutorial failed for me by introducing define-syntax-rule before explaining how it worked.
An even more useful concept exists in J.Shutt's Kernel programming language[1] - that of first class operatives. Every entity becomes a first-class object, which can be an operative (like some special forms) or an applicative (a scheme function). operatives do not implicitly evaluate the operands, so you can easily create macro-like behavior by defining your own evaluation model. It's also trivial to convert between applicatives and operatives with wrap and unwrap (which are not special forms, they're defined in terms of vau and eval).
[1]:http://web.cs.wpi.edu/~jshutt/kernel.html