Scala Adding Macros to the Language(infoq.com)
infoq.com
Scala Adding Macros to the Language
http://www.infoq.com/news/2012/07/scala-macros
3 comments
This gives me an all-new appreciation for Lisp. It's nice to use standard list operations in a macro instead of needing to learn AST magic.
> This gives me an all-new appreciation for Lisp.
I'm a Lisper who's also programmed in Scala. (And for the record, I'm not a Lisp purist - in fact, there are a number of things I really like about Scala).
But I'm not sure how I feel about this change. Scala can already implement things like DSLs through creative use of type inference, etc.
> It's nice to use standard list operations in a macro instead of needing to learn AST magic.
The power of Lisp's macros come from it's homoiconicity, which is something I could never have appreciated before learning Lisp - in fact, I didn't even understand its significance until I wrote an implementation of a Lisp compiler for the first time. (For the record, it was a better learning experience than it was a compiler, but that was the whole point).
The best explanation I've seen of homoiconic macros is one written by a well-known Perl programmer. http://lists.warhead.org.uk/pipermail/iwe/2005-July/000130.h... (In a way, it makes sense that a Perl programmer would be the best at explaining what homoiconicity is, given that its extreme heteroiconicity is the underlying cause of many of the common criticisms of Perl to begin with).
That email is mostly accurate, and it includes this gem:
> Source code generation is unreliable and inadvisable in every language except Lisp.
This is where macros really shine. Macros allow you to generate source code reliably - or put another way, they let you refactor your code far more than you ever thought possible. (When you realize how easy it is to write Lisp programs that recursively define other valid Lisp programs, it's not hard to see why it's such a favorite of the AI community![1])
But here's the catch - for the kind of reliability that MJD talks about, you need to have the internal representation of the language be a fundamental data structure in the language, not just the textual representation.
Why's that? It's simple:
> It's nice to use standard list operations in a macro instead of needing to learn AST magic.
In Lisp, the internal representation is a list. The AST is a list. There's no 'instead of' here... macros are simply AST magic made simple....!
[1] http://en.wikipedia.org/wiki/Technological_singularity
I'm a Lisper who's also programmed in Scala. (And for the record, I'm not a Lisp purist - in fact, there are a number of things I really like about Scala).
But I'm not sure how I feel about this change. Scala can already implement things like DSLs through creative use of type inference, etc.
> It's nice to use standard list operations in a macro instead of needing to learn AST magic.
The power of Lisp's macros come from it's homoiconicity, which is something I could never have appreciated before learning Lisp - in fact, I didn't even understand its significance until I wrote an implementation of a Lisp compiler for the first time. (For the record, it was a better learning experience than it was a compiler, but that was the whole point).
The best explanation I've seen of homoiconic macros is one written by a well-known Perl programmer. http://lists.warhead.org.uk/pipermail/iwe/2005-July/000130.h... (In a way, it makes sense that a Perl programmer would be the best at explaining what homoiconicity is, given that its extreme heteroiconicity is the underlying cause of many of the common criticisms of Perl to begin with).
That email is mostly accurate, and it includes this gem:
> Source code generation is unreliable and inadvisable in every language except Lisp.
This is where macros really shine. Macros allow you to generate source code reliably - or put another way, they let you refactor your code far more than you ever thought possible. (When you realize how easy it is to write Lisp programs that recursively define other valid Lisp programs, it's not hard to see why it's such a favorite of the AI community![1])
But here's the catch - for the kind of reliability that MJD talks about, you need to have the internal representation of the language be a fundamental data structure in the language, not just the textual representation.
Why's that? It's simple:
> It's nice to use standard list operations in a macro instead of needing to learn AST magic.
In Lisp, the internal representation is a list. The AST is a list. There's no 'instead of' here... macros are simply AST magic made simple....!
[1] http://en.wikipedia.org/wiki/Technological_singularity
But I'm not sure how I feel about this change.
Scala can already implement things like DSLs through
creative use of type inference, etc.
(-) there are some god-awful DSLs available for Scala ; the presence of a macros system would allow the reuse of syntax instead of coming up with operators that look like random noise
(-) LINQ becomes possible - I don't know how many people here used LINQ, but it's really awesome to express a query in terms of filter, map and flatMap, and depending on the underlying collection to issue a database query or to submit some code to the GPU for processing.
And LINQ is an instance where static typing really shines.
When you realize how easy it is to write Lisp programs that recursively define other valid Lisp programs, it's not hard to see why it's such a favorite of the AI community!
I think Lisp's success in early AI work had as much to do with its support for higher order functions, repl, dynamic dispatch etc as it did with the macro facilities. Lisp was way ahead in those respects.
But you don't see nearly as much use of Lisp in AI now that many other languages have learned all of Lisp's tricks except homoiconicity and that a lot of "AI" in the field these days is really statistics and linear algebra.
I think Lisp's success in early AI work had as much to do with its support for higher order functions, repl, dynamic dispatch etc as it did with the macro facilities. Lisp was way ahead in those respects.
But you don't see nearly as much use of Lisp in AI now that many other languages have learned all of Lisp's tricks except homoiconicity and that a lot of "AI" in the field these days is really statistics and linear algebra.
I very much agree with you. Even in Genetic programming, any language that has Algebraic Types and parser combinators gets only a very small additive constant with respect to lisp.
But if you want the best model for the GP language so that it can easily self modify, you will recreate a lisp (at least such was my case) because it looks like Lisp is the homoiconic language and is complete in the mathematical sense (how the reals are complete). If you look at stuff like decision trees, random forests and genetic programming; while implementation language doesn't matter, for execution they really come into their own in a language like lisp.
This has led me to believe that lisp is so far ahead of its time that it is not the language of humans but instead the natural tongue of AIs built on Turing machines.
But if you want the best model for the GP language so that it can easily self modify, you will recreate a lisp (at least such was my case) because it looks like Lisp is the homoiconic language and is complete in the mathematical sense (how the reals are complete). If you look at stuff like decision trees, random forests and genetic programming; while implementation language doesn't matter, for execution they really come into their own in a language like lisp.
This has led me to believe that lisp is so far ahead of its time that it is not the language of humans but instead the natural tongue of AIs built on Turing machines.
When you realize how easy it is to write Lisp programs that recursively define other valid Lisp programs, it's not hard to see why it's such a favorite of the AI community!
I don't know what counts as "AI" anymore but in the Machine Learning community, it looks like Matlab, R, Python and Java dominate in terms of use and cutting edge packages. This is due to two things: modern ML does not focus on areas where Lisp gives a clear advantage and 2) what everyone else is using and reduced learning overhead are most important in determining favorites.
I don't know what counts as "AI" anymore but in the Machine Learning community, it looks like Matlab, R, Python and Java dominate in terms of use and cutting edge packages. This is due to two things: modern ML does not focus on areas where Lisp gives a clear advantage and 2) what everyone else is using and reduced learning overhead are most important in determining favorites.
The best explanation I've seen of homoiconic macros is one written by a well-known Perl programmer. http://lists.warhead.org.uk/pipermail/iwe/2005-July/000130.h...
Thank you for that - I enjoyed reading it.
Also, is that our btilly?
Thank you for that - I enjoyed reading it.
Also, is that our btilly?
Yes, it is.
+1 on the sentiment though I'm not fully understanding why this would be useful for me...
To me this is like a crutch that the language itself cannot solve.
Ultimately, Scala will be held back because of Java. Scala deserves to live on by itself.
To me this is like a crutch that the language itself cannot solve.
Ultimately, Scala will be held back because of Java. Scala deserves to live on by itself.
What I'm saying is that in Lisp it feels like a natural part of the language. There's not a big leap from:
Whereas in Scala the macro facility looks different enough from vanilla code to have to be learned separately.
(defun add-two (x) (+ 2 x))
(defmacro add-two (x) `(+ 2 ,x))
Whereas in Scala the macro facility looks different enough from vanilla code to have to be learned separately.
Even in Lisp it's kind of interesting that s-expressions alone don't give you a convenient enough notation for macros. You really need backquote-comma-splice as well. I'd like to know more about when and how macros became popular in Lisps. I suspect it's closely related to this notation.
Backquote-comma-splice is so useful for building symbolic trees that it's an example of what Whitehead famously celebrated:
http://introtologic.info/AboutLogicsite/whitehead%20Good%20N...
Off topic, but if you read that piece it's interesting how much of what he says applies also to programming, yet how controversial it remains in the realm of programming languages.
Backquote-comma-splice is so useful for building symbolic trees that it's an example of what Whitehead famously celebrated:
http://introtologic.info/AboutLogicsite/whitehead%20Good%20N...
Off topic, but if you read that piece it's interesting how much of what he says applies also to programming, yet how controversial it remains in the realm of programming languages.
S-expressions don't excel at anything--the claim to fame is that they're good enough for everything. If we wanted to put macros into 1956-era Lisp, it'd just be something along the lines of:
So yes, I really like the `,@-style notation, but also think the alternative is tolerable.
(It bugs me that I can't think of a simple example that must be a macro (and not a function), but you get the idea.)
(defmacro add-two (x) (quote ((quote +) 2 x)))
So yes, I really like the `,@-style notation, but also think the alternative is tolerable.
(It bugs me that I can't think of a simple example that must be a macro (and not a function), but you get the idea.)
Oh, I disagree. It looks ok in toy examples (something that is true of many languages' macro facility) but all the calls to CONS and LIST and APPEND and whatever else you need quickly become unwieldy. They obscure the structure of the expression you're trying to build. This is exactly the problem that backquote-comma-splice (on top of s-exprs) solves. And I suppose we need to throw in single-quote too.
It's interesting, though, that this is a usability question rather than a technical one, i.e. you can build the same expressions without that notation, just not as easily. But that's what's great about that Whitehead piece I linked to: he shows how fundamental this is. It seems like a surface matter but isn't, because it influences what other things one can use one's limited mental capacity to do. It makes subsequent thoughts possible that wouldn't otherwise be. (Edit: this is why programmers who talk about how language doesn't matter are deeply wrong. And also why programming languages have surrounding cultures. But I go off topic again.)
By the way, your macro example is off, since there's nothing to cause X to be evaluated; it's forever embalmed in that outer QUOTE.
It's interesting, though, that this is a usability question rather than a technical one, i.e. you can build the same expressions without that notation, just not as easily. But that's what's great about that Whitehead piece I linked to: he shows how fundamental this is. It seems like a surface matter but isn't, because it influences what other things one can use one's limited mental capacity to do. It makes subsequent thoughts possible that wouldn't otherwise be. (Edit: this is why programmers who talk about how language doesn't matter are deeply wrong. And also why programming languages have surrounding cultures. But I go off topic again.)
By the way, your macro example is off, since there's nothing to cause X to be evaluated; it's forever embalmed in that outer QUOTE.
What's great about lisps that support modifications to the Reader is that you can go even beyond the standard sugar and use your own characters for some special purpose. (e.g. adding map-syntax with curly braces...)
> (It bugs me that I can't think of a simple example that must be a macro (and not a function), but you get the idea.)
My main three rules: 1) delayed, conditional, repeated, etc. evaluation of arguments 2) compile-time computation for massive optimization potential 3) running or inserting code before or after any of the argument forms (useful for tests--init clean env, run test body, destroy env) or changing the argument forms themselves.
> (It bugs me that I can't think of a simple example that must be a macro (and not a function), but you get the idea.)
My main three rules: 1) delayed, conditional, repeated, etc. evaluation of arguments 2) compile-time computation for massive optimization potential 3) running or inserting code before or after any of the argument forms (useful for tests--init clean env, run test body, destroy env) or changing the argument forms themselves.
> I can't think of a simple example that must be a macro (and not a function)
Implementing IF in terms of COND or vice-versa.
Implementing IF in terms of COND or vice-versa.
If nothing changes until release it should look like this in Scala:
# without macro
def addTwo(x: Int) = x + 2
# with macro
def addTwo(x: Int) = macro addTwoImpl
def addTwoImpl(c: Context)(x: c.Expr[Int]): c.Expr[Int] = c.reify(x + 2)
macro def addTwo(c: Context)(x: c.Expr[Int]): c.Expr[Int] = c.reify(x + 2)
macro def addTwo(x: Expr[Int]): Expr[Int] = reify(x + 2)
The Perl6 spec also remains quite natural as well:
sub add-two ($x) { 2 + $x }
macro add-two ($x) { quasi { 2 + {{{$x}}} } }Java is the reason that Scala has had any significant interest at all.
I agree and that would seem so, but why then all the hate from Java developers no less?
I'm a Java developer and find working with Scala a pleasure. I will admit I probably wouldn't have given it a serious look if it wasn't for JVM support.
Yeah, it's the reason why Template Haskell (macros) are awkward. The ASTs are just too complicated.
Well, it is because you write AST in Lisp :)
I am suspicious that this is an attempt to make Scala more Clojure-like...
It's not. If I remember right the did some big refactoring of the compiler and found some places where this macro facility made that work less verbose.
http://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2012/Ref...
It's been a while since I watched this talk, so I may have the details wrong. It's a great talk, but at least read the abstract:
Reflection and compilers do tantalizing similar things. Yet, in mainstream, statically typed languages the two have been only loosely coupled, and generally share very little code. In this talk I explore what happens if one sets out to overcome their separation.
The first half of the talk addresses the challenge how reflection libraries can share core data structures and algorithms with the language's compiler without having compiler internals leaking into the standard library API. It turns out that a component system based on abstract types and path-dependent types is a good tool to solve this challenge. I'll explain how the "multiple cake pattern" can be fruitfully applied to expose the right kind of information.
The second half of the talk explores what one can do when strong, mirror-based reflection is a standard tool. In particular, the compiler itself can use reflection, leading to a particular system of low-level macros that rewrite syntax trees. One core property of these macros is that they can express staging, by rewriting a tree at one stage to code that produces the same tree at the next stage. Staging lets us implement type reification and general LINQ-like functionality. What's more, staging can also be applied to the macro system itself, with the consequence that a simple low-level macro system can produce a high-level hygienic one, without any extra effort from the language or compiler.
http://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2012/Ref...
It's been a while since I watched this talk, so I may have the details wrong. It's a great talk, but at least read the abstract:
Reflection and compilers do tantalizing similar things. Yet, in mainstream, statically typed languages the two have been only loosely coupled, and generally share very little code. In this talk I explore what happens if one sets out to overcome their separation.
The first half of the talk addresses the challenge how reflection libraries can share core data structures and algorithms with the language's compiler without having compiler internals leaking into the standard library API. It turns out that a component system based on abstract types and path-dependent types is a good tool to solve this challenge. I'll explain how the "multiple cake pattern" can be fruitfully applied to expose the right kind of information.
The second half of the talk explores what one can do when strong, mirror-based reflection is a standard tool. In particular, the compiler itself can use reflection, leading to a particular system of low-level macros that rewrite syntax trees. One core property of these macros is that they can express staging, by rewriting a tree at one stage to code that produces the same tree at the next stage. Staging lets us implement type reification and general LINQ-like functionality. What's more, staging can also be applied to the macro system itself, with the consequence that a simple low-level macro system can produce a high-level hygienic one, without any extra effort from the language or compiler.
I hear they're adding dynamic typing to Haskell: http://www.infoq.com/news/2011/04/erlang-copied-jvm-and-scal...