Utterly sensible HTML templating in Common Lisp(trapm.com)
trapm.com
Utterly sensible HTML templating in Common Lisp
http://trapm.com/vana-templating-an-utterly-sensible-templatin
14 comments
The problem I always see with practically all templating systems is the lack of proper escaping of input data.
For example, the function 'tag-name' really needs to HTML-escape the input parameter content. The simple example (tag-name "div" "Hello world") breaks down when "Hello World" is replaced by a variable reference.
Fortunately this is an relatively easy problem to solve. A better definition for tag-name is something like:
(defun tag-name (name content) (html (format nil "<~A>~A</~A>" name (to-html content) name))
The function to-html will return the HTML-escaped version of content. The function html returns an object/string that marked as already being HTML-escaped. They work together so that (to-html (html x)) just returns x.
A similar type system can be set up to handle url escaping rules.
If the templating system doesn't keep track of what is escaped and what isn't then that burden is left to the programmer with all of the associated risks of producing malformed output and creating opportunities for injection attacks.
For example, the function 'tag-name' really needs to HTML-escape the input parameter content. The simple example (tag-name "div" "Hello world") breaks down when "Hello World" is replaced by a variable reference.
Fortunately this is an relatively easy problem to solve. A better definition for tag-name is something like:
(defun tag-name (name content) (html (format nil "<~A>~A</~A>" name (to-html content) name))
The function to-html will return the HTML-escaped version of content. The function html returns an object/string that marked as already being HTML-escaped. They work together so that (to-html (html x)) just returns x.
A similar type system can be set up to handle url escaping rules.
If the templating system doesn't keep track of what is escaped and what isn't then that burden is left to the programmer with all of the associated risks of producing malformed output and creating opportunities for injection attacks.
Anything that requires you to write HTML in something that isn't HTML isn't sensible. Sure, go ahead, add tokens to the HTML that can be parsed out and rendered differently, stored in a cache or something. But as soon as you're forced to write in another language to get HTML, you've lost.
Edit: To elaborate, as some have asked: I'm speaking in terms of HTML templating. Having the software render your HTML form for you, or rending out a table full of data is fine. But by and large, the HTML should remain HTML. Granted, in this case, your not writing HTML, your writing code to generate a form. Not sitting there adding HTML attributes, which is handled for you.
As to why I disagree with this approach? At it's core, portability. You can now no longer take an HTML file and use it. You have to rewrite everything into your strange, personal, customized, non-standard language and hope everything comes out alright. Their is also the learning curve in bringing anyone else on board. You also suddenly lack the ability to use any of the tools associated with HTML. Abstraction for the sack of abstraction is silly.
If you have multiple functions for every single tag, your essentially taking HTML and just making the syntax your own. This "(ul '(id "navbar" class "horizontal_list")" is no better than HTML, but it isn't special or unique. It's cumbersome. Your also limited by the framework (which you hope will add new tags as they appear, and support the customization HTML offers).
I'm not against HTML generating code. I'm against rewriting HTML as code. Maybe someone can explain how
(ul '(id "navbar" class "horizontal_list")...
is better than
<ul id="navbar" class="horizontal_list">...
Edit 2: Also, it should be fairly obviously, but all of this is of course just my opinion. =)
Edit: To elaborate, as some have asked: I'm speaking in terms of HTML templating. Having the software render your HTML form for you, or rending out a table full of data is fine. But by and large, the HTML should remain HTML. Granted, in this case, your not writing HTML, your writing code to generate a form. Not sitting there adding HTML attributes, which is handled for you.
As to why I disagree with this approach? At it's core, portability. You can now no longer take an HTML file and use it. You have to rewrite everything into your strange, personal, customized, non-standard language and hope everything comes out alright. Their is also the learning curve in bringing anyone else on board. You also suddenly lack the ability to use any of the tools associated with HTML. Abstraction for the sack of abstraction is silly.
If you have multiple functions for every single tag, your essentially taking HTML and just making the syntax your own. This "(ul '(id "navbar" class "horizontal_list")" is no better than HTML, but it isn't special or unique. It's cumbersome. Your also limited by the framework (which you hope will add new tags as they appear, and support the customization HTML offers).
I'm not against HTML generating code. I'm against rewriting HTML as code. Maybe someone can explain how
(ul '(id "navbar" class "horizontal_list")...
is better than
<ul id="navbar" class="horizontal_list">...
Edit 2: Also, it should be fairly obviously, but all of this is of course just my opinion. =)
At it's core, portability. You can now no longer take an HTML file and use it. You have to rewrite everything into your strange, personal, customized, non-standard language and hope everything comes out alright.
This thing generates plain, standards-compliant html. Just like the rails app I'm porting over to it. If you can't easily take html and port it in/out of whatever tool you're working in, you're doing it wrong. Even with all the abstractions, partials, specialized rails functions, I was able to port the majority of the view layer over in a lazy Sunday afternoon.
Maybe someone can explain how (ul '(id "navbar" class "horizontal_list")... is better than <ul id="navbar" class="horizontal_list">...
It's not any better. Well, there is the part where if it's invalid, there'll be an error instead of some mangled html that browser tries to fix on its own, and you don't have to remember to close the tag, but those are really somewhat minor.
The principle of lowest friction is that this templating system must, at the very minimum, match, if not exceed the ease of HTML. So the above is not very different, but when it grows larger, you can abstract away the repeated calls, make your own tags (as I show with the image and link-to "tags") that make the html more readable and vastly more DRY.
And again, it all goes out to nice html. Take that and import it into whatever framework you're headed off to.
This thing generates plain, standards-compliant html. Just like the rails app I'm porting over to it. If you can't easily take html and port it in/out of whatever tool you're working in, you're doing it wrong. Even with all the abstractions, partials, specialized rails functions, I was able to port the majority of the view layer over in a lazy Sunday afternoon.
Maybe someone can explain how (ul '(id "navbar" class "horizontal_list")... is better than <ul id="navbar" class="horizontal_list">...
It's not any better. Well, there is the part where if it's invalid, there'll be an error instead of some mangled html that browser tries to fix on its own, and you don't have to remember to close the tag, but those are really somewhat minor.
The principle of lowest friction is that this templating system must, at the very minimum, match, if not exceed the ease of HTML. So the above is not very different, but when it grows larger, you can abstract away the repeated calls, make your own tags (as I show with the image and link-to "tags") that make the html more readable and vastly more DRY.
And again, it all goes out to nice html. Take that and import it into whatever framework you're headed off to.
I'm going to take the stance of: "I don't see the usefulness of it, but that doesn't mean it can't be useful." Maybe it's working within the limitations of a particular language that doing something like this is the best way. Maybe I'm completely missing some clear and obvious sign.
> The principle of lowest friction is that this templating system must, at the very minimum, match, if not exceed the ease of HTML.
Realizing also you mentioned as it grows larger, abstraction occurs, this is where I don't see the value. For anyone to use this templating system, they need to know HTML and then they need to know this templating system (which amounts to learning a programming language). Mistakes in the templating system will result in HTML errors. Maybe I've just had too many bad experience with systems like this where fixing the HTML would have been easy, except I had to go through a templating system that abstracted away critical parts.
I'll assume I'm just missing something, or I've had this problem your solving before, and solved it in some other way. Like I said, I don't see how this method benefits anything (and I've done my fair share of work on the web), but I'll chalk that up to a failing on my part to grasp some concept. =)
> The principle of lowest friction is that this templating system must, at the very minimum, match, if not exceed the ease of HTML.
Realizing also you mentioned as it grows larger, abstraction occurs, this is where I don't see the value. For anyone to use this templating system, they need to know HTML and then they need to know this templating system (which amounts to learning a programming language). Mistakes in the templating system will result in HTML errors. Maybe I've just had too many bad experience with systems like this where fixing the HTML would have been easy, except I had to go through a templating system that abstracted away critical parts.
I'll assume I'm just missing something, or I've had this problem your solving before, and solved it in some other way. Like I said, I don't see how this method benefits anything (and I've done my fair share of work on the web), but I'll chalk that up to a failing on my part to grasp some concept. =)
How have you lost? I write my HTML in a very similar way to the linked article, and I don't think I lose at all. In fact, I much prefer it.
It is more consistent. All of my code is now (in my case) Scheme. This makes it easier to change things as you don't have to go between two different languages. And it is simpler.
My favorite reason is the fact that I can use S-Exps. I no longer have to type the ending tag. I remove the possibility for spelling it wrong, or forgetting the slash. And it takes less space and is easier to get the placement correct, as you just utilize paren matching.
It is more consistent. All of my code is now (in my case) Scheme. This makes it easier to change things as you don't have to go between two different languages. And it is simpler.
My favorite reason is the fact that I can use S-Exps. I no longer have to type the ending tag. I remove the possibility for spelling it wrong, or forgetting the slash. And it takes less space and is easier to get the placement correct, as you just utilize paren matching.
The paren matching really is key. I was blown away by how intuitive it is. Previously I had preferred haml, but without paren highlighting it can be hard to tell the depth you're working at relative to another tag.
This could be solved by some editor placing arrows showing which tags are parents to your current position, but I don't think that's available (at least not in emacs, sadly), and paren highlighting is.
Also, never missing a closed tag, mispellings, abstractions, etc. All very nice. When loading a broken page in chrome, it's crazy how much the html will be transformed, so that it's hard to tell why the page renders a certain way. A bit easier with this approach.
This could be solved by some editor placing arrows showing which tags are parents to your current position, but I don't think that's available (at least not in emacs, sadly), and paren highlighting is.
Also, never missing a closed tag, mispellings, abstractions, etc. All very nice. When loading a broken page in chrome, it's crazy how much the html will be transformed, so that it's hard to tell why the page renders a certain way. A bit easier with this approach.
I agree. It looks really clean. I'm not about to throw out haml for it yet, but I'm definitely intrigued.
If you do it this way paredit works for html too :)
Which Scheme libraries/modules are you using?
I use Chicken Scheme with the Awful web framework. I also use the the Chicken Eggs: html-tags (sexp based HTML generation) and html-utils, along with many other supporting modules.
Care to elaborate at all? I'm not sure I understand. Would haml/erb fall under the same umbrella statement?
HTML is great, but there are a few areas where it's tough to do things, like abstracting repetitive elements. There are plenty of ways around this using other languages, and perhaps the best solution for someone like you would be writing html snippets and using javascript to replicate them, thus writing purely in html?
HTML is great, but there are a few areas where it's tough to do things, like abstracting repetitive elements. There are plenty of ways around this using other languages, and perhaps the best solution for someone like you would be writing html snippets and using javascript to replicate them, thus writing purely in html?
I edited the original post to explain, since I got several requests for that. But I did want to "reply" just to say I replied above. =)
Not true, the are advantages to programatically generated HTML using a mini language that simply can't be ignored. See http://seaside.st
I hope you're homophone trolling.
%ul#navbar.horizontal_list
For those who didn't know already, cl-who (http://weitz.de/cl-who/) is a well known sexp-to-html converter. This one works a little differently from cl-who (uses functions instead of macros), but the input is similar.
I am spoiled by nodejs and its live server side DOM rendering. Having one layer of template code reusable on the client and server is too powerful. You would think people would be sick of writing duplicate server templates and ajax templates. To me anything else is a step in the wrong direction.
I think you're pointing out Dave Glass' technique over at yahoo, right? This one http://www.yuiblog.com/blog/2010/04/09/node-js-yui-3-dom-man... ?
The ideas there are definitely very powerful... and just plain sensible. I'd like to find a way to work this in, but it will take time. Do you see any fundamental reason a lisp equivalent of this technique couldn't be reproduced, using something like parenscript?
The ideas there are definitely very powerful... and just plain sensible. I'd like to find a way to work this in, but it will take time. Do you see any fundamental reason a lisp equivalent of this technique couldn't be reproduced, using something like parenscript?
It's been done already: http://code.google.com/p/cl-closure-template/
That's extremely different from the technique that Glass is demoing - in fact, it's the antithesis. It takes care of unifying the server/client templates (by introducing an unnecessary layer, imo), but it's very obviously string-based output rather than live DOM manipulation like Enlive or YUI on node.js
I still think the new gold standard for functional and extensible HTML templating is Enlive - people should look into this before coming up with variations of the same old wheel - https://github.com/cgrand/enlive
I think there's a difference: Enlive is meant to manipulate the DOM using familiar CSS/jQuery like selectors. The article however, is a way to _write_ the HTML in the first place.
I've myself been using a declarative style HTML output engine that I wrote in Python. It's simple, easier, outputs correct HTML and it merges well with my workflow. I don't have to write HTML, nor JavaScript to programmatically generate what I need. It may sound naive, but I find myself comfortable indenting code in Python than write matching HTML tags. If there are errors, the Python code wouldn't run, but HTML code would still render, though it wouldn't be what I want.
Relevant: http://code.google.com/p/zen-coding/
I've myself been using a declarative style HTML output engine that I wrote in Python. It's simple, easier, outputs correct HTML and it merges well with my workflow. I don't have to write HTML, nor JavaScript to programmatically generate what I need. It may sound naive, but I find myself comfortable indenting code in Python than write matching HTML tags. If there are errors, the Python code wouldn't run, but HTML code would still render, though it wouldn't be what I want.
Relevant: http://code.google.com/p/zen-coding/
Enlive is awesome, I used it while working in clojure quite a bit. I definitely prefer it, personally, to something like Clojure's hiccup (which vana-templating is very similar to). The tradeoff is that new-comers who are already familiar with front-end design are going to find it tough to transition - it's a fundamental problem: do you optimize for the newcomer, or for the experienced developer?
Enlive has a lot in common with Vekz' suggestion elsewhere in the comments, and I feel like there's some unifying theme underlying all of this, that should make it easy for newcomers, but allow experienced developers to transform the DOM programmatically.
I'm searching for it now, and I'm sure there'll be a lot of false-starts and dead-ends, but the only sure way not to get there is to not try.
Enlive has a lot in common with Vekz' suggestion elsewhere in the comments, and I feel like there's some unifying theme underlying all of this, that should make it easy for newcomers, but allow experienced developers to transform the DOM programmatically.
I'm searching for it now, and I'm sure there'll be a lot of false-starts and dead-ends, but the only sure way not to get there is to not try.
I agree completely. Enlive has really blown me away.
You should learn about WITH-OUTPUT-TO-STRING.
(string-downcase (string thing)) should be written (string-downcase thing)
The fdefinition trick is not a good one. Define the functions at compile-time, not at runtime.
(string-downcase (string thing)) should be written (string-downcase thing)
The fdefinition trick is not a good one. Define the functions at compile-time, not at runtime.
> (string-downcase (string thing)) should be written (string-downcase thing)
I learnt this trick only after working on CL for a couple of years!
I learnt this trick only after working on CL for a couple of years!
Maybe http://l1sp.org/search/designator will teach you a few more tricks and save you some time...
Nice. Today I learnt about the file position designator :)
I was originally using macros, but I prefer to use functions instead. I'll give it some thought, you may be right in this case.
And thanks for the (string-downcase thing) trick, I'll update that right now :)
And thanks for the (string-downcase thing) trick, I'll update that right now :)
Functions are good. Defining them at runtime with (setf (fdefinition ...) ...) is rarely good.
This looks to me like an HTML generation library, not an HTML templating tool. Common Lisp has a number of both, and I'm wondering what advantage this would have over cl-who, for example?
Can also be done, in runtime including, more cleanly and without parenthesis with coffeescript: http://news.ycombinator.com/item?id=1786002
Not trying to pick on this particular project because it seems cool enough, but,
I don't get all this html templating. With ajax and websockets on the way, why are we still generating html in the application? This all seems like a solution to a problem we should be leaving behind.
I don't get all this html templating. With ajax and websockets on the way, why are we still generating html in the application? This all seems like a solution to a problem we should be leaving behind.
Because you have to hit the server initially. While you could totally serve some JS, JSON data and a small HTML wrapper to kickstart the client-side code, just generating the page is a lot simpler and typically more efficient.
Then you have all the problems of a purely client-side application (dealing with deep-linking, bookmarks, back buttons, users with exotic browsers, search engines, etc). That stuff can all be dealt with, but all dramatically increases the complexity. Your "Hello World" quickly becomes complex and unwieldy.
I personally think the right way is probably a templating engine that can work either client- or server-side so you can generate whatever you need wherever you are.
Then you have all the problems of a purely client-side application (dealing with deep-linking, bookmarks, back buttons, users with exotic browsers, search engines, etc). That stuff can all be dealt with, but all dramatically increases the complexity. Your "Hello World" quickly becomes complex and unwieldy.
I personally think the right way is probably a templating engine that can work either client- or server-side so you can generate whatever you need wherever you are.
I guess I don't see a reason not to generate it on the server. I have a section of my website with some essays, for example, which are formatted from Markdown source plus a bit of HTML templating. The essays don't frequently change, and neither does the template. Why would I want to regenerate the page on the client side every time someone reads it instead of just generating it once and serving up static HTML?
on the way
You said it - websockets aren't here yet.
Plus I think there are still plenty of good applications for static HTML. For starters, even full AJAX apps should work without JS enabled if you care about accessibility.
You said it - websockets aren't here yet.
Plus I think there are still plenty of good applications for static HTML. For starters, even full AJAX apps should work without JS enabled if you care about accessibility.
For starters, even full AJAX apps should work without JS enabled if you care about accessibility.
This should be just about possible using Node.js these days: You can run your entire client-side templating system—and even the <canvas>—on the server, and then server static HTML and PNGs to the browser. Unfortunately, quite a bit of elbow grease is still required.
This should be just about possible using Node.js these days: You can run your entire client-side templating system—and even the <canvas>—on the server, and then server static HTML and PNGs to the browser. Unfortunately, quite a bit of elbow grease is still required.
But my point is - you're still templating. The only thing you're gaining is using the same language on the client and the server. I'm not saying that's a small thing, but the OP could have achieved the same using parenscript.
The GP seemed to be implying that we should only generate our HTML programatically on the client. Thinking about it more, I don't agree with this. I don't write webapps these days, but in our platform we do a lot of code generation. We use a programmatic method to do this because we have to manipulate the intermediate representation a lot - add or remove fields added by earlier phases in the generation etc. This is much more flexible but it's much harder to maintain, and looking at the code generation code it's very hard to see what the end result will be. If we were simply generating code in a single pass I'd definitely use templating, it's just much simpler, easier to develop, and easier to see what the end result will be.
Add in the need with HTML to work with designers and I think templating will be with us for some time to come.
The GP seemed to be implying that we should only generate our HTML programatically on the client. Thinking about it more, I don't agree with this. I don't write webapps these days, but in our platform we do a lot of code generation. We use a programmatic method to do this because we have to manipulate the intermediate representation a lot - add or remove fields added by earlier phases in the generation etc. This is much more flexible but it's much harder to maintain, and looking at the code generation code it's very hard to see what the end result will be. If we were simply generating code in a single pass I'd definitely use templating, it's just much simpler, easier to develop, and easier to see what the end result will be.
Add in the need with HTML to work with designers and I think templating will be with us for some time to come.
So the AJAX request goes out, and what comes back? HTML or data? It will have to be turned into HTML either by the server or the client. And you need some logic for that. So there's really no escaping this.
You're definitely right, actually. I've been working a lot in clojure, node.js, and html5 <canvas> recently on various projects (more posts later), and there's a definitely feeling of reinventing the wheel, and doing it poorly.
That said, there are a ton of different approaches - At the oak.js meetup, I saw a sammy.js app embedded into couchdb. Definitely not my style, but a cool idea anyway.
I'm very eager to get this into several html5 apps that are currently node.js or rails backends and for particular reasons would be well suited for a lisp backend. The goal is to tie in very closely with the html5 landscape, and drop a lot of the legacy stuff that other frameworks care about.
That said, there are a ton of different approaches - At the oak.js meetup, I saw a sammy.js app embedded into couchdb. Definitely not my style, but a cool idea anyway.
I'm very eager to get this into several html5 apps that are currently node.js or rails backends and for particular reasons would be well suited for a lisp backend. The goal is to tie in very closely with the html5 landscape, and drop a lot of the legacy stuff that other frameworks care about.
Am I missing something or is this code defining "global scope" functions whose names include things like a, b, i, p, command, etc...?
What are the chances that such global names will collide with something else?
What are the chances that such global names will collide with something else?
You could easily put them into a separate package if you wish:
http://gigamonkeys.com/book/programming-in-the-large-package...
http://gigamonkeys.com/book/programming-in-the-large-package...
One a little like it for emacs: https://github.com/philjackson/xmlgen
Are there any big diffrence to clojure's hiccup?
Sensible html templating exists already. It's called HAML.
http://haml-lang.com/
http://haml-lang.com/
Actually, while developing this I was really struck by the similarities. With the automatic indenting in emacs for lisp (really quite good), it looks very similar, and it's impossible to leave a tag unclosed, but there's a further gain for using parentheses in that with paren-highlighting, you can tell exactly which tag your in. While I converted several dozen pages to this system from straight html, I was saved again and again by this.
HAML's great, and I disagree with the other "I hate haml" post on the frontpage today, but this approach actually alleviates the issues pointed out with haml's whitespace-is-meaningful approach.
Anyway, it's all in good fun.
HAML's great, and I disagree with the other "I hate haml" post on the frontpage today, but this approach actually alleviates the issues pointed out with haml's whitespace-is-meaningful approach.
Anyway, it's all in good fun.
Yup I'm a huge haml fan, and saw the similarities. I thought your post was great. Very clean, and you can keep optimizing it. I'm learning CL right now and this added some fuel to the motivation fire ;)
https://github.com/cgrand/enlive
This has some really nice properties:
1. The only thing your devs and designers need to agree on is some basic semantic markup and tag IDs. Then they can make a full comp (complete with sample data) for their purposes. You simply rewrite that data.
2. The approach is very declarative, which makes it a lot easier to reason about what is actually being shown and what is actually changing.
3. It's amenable to all the same functional properties of the sexp->xml stuff that you see here.
Enlive's approach is a nearly 100% win for doing site templating over these sorts of sexp->xml libraries.