Type-safely embed DSLs directly into Java(github.com)

43 pointsby mckinney6 anni fa25 comments

Type-safely embed DSLs directly into Java

https://github.com/manifold-systems/manifold/tree/master/manifold-core-parent/manifold#embedding-with-fragments-experimental

6 comments

rhacker6 anni fa

The graphql example is not the greatest, but I would personally use GQL Code Generator to create a little strongy typed client for all your .graphql files:

https://graphql-code-generator.com/docs/plugins/java

Not sure if there's a strong use-case in general here. Even Groovy code, which is a popular choice for JVM DSL languages, has pretty excellent stub generation.

mckinney6 anni fa

But the main idea with this is to avoid the pitfalls of conventional code generation such as GQL Code Generator. See "The Big Picture"[1] in the core docs.

[1] https://github.com/manifold-systems/manifold/tree/master/man...

sreque6 anni fa

When I see "no code on disk" I see that as a downside. What happens when this generated code has a bug? How do I set a break point in this code? How do I see the code to know what it is doing?

Extending the compiler is great... until it isn't. Imagine a feature in java the language not working correctly due to a compiler bug. Debugging or working around this issue could be very difficult. With compiler extensions, you are now widening the opportunity for this type of bug to occur.

More principled code generation systems like Racket's macro system do in fact let you expand the code to see what's generated while avoiding having a separate build step, and even have a macro debugger for interactive debugging. Mainstream languages have a ways to go before we get this kind of tooling for language extension frameworks. Until then, I think I prefer having a separate build step with on-disk, visible, and debug-able code.

mckinney6 anni fa

You can see the generated code in the IDE via Edit | View Java Source when a resource is selected. And you can debug it as well. Sort of the best of both worlds. What makes Manifold most interesting, however, is the entire dev experience when used in the IDE. This short screencast demonstrates some of that: http://manifold.systems/images/graphql.mp4

abeppu6 anni fa

I definitely agree; the author clearly thought no on-disk produced code was a feature, but it makes me nervous. Similarly, the author then goes on to say how the framework supports circular dependencies between manifolds, or manifolds that each modify one another's types ... and I immediately think that will create debugging nightmares.

mckinney6 anni fa

In most cases, the devs that should be concerned about debugging generated code are the authors of the generator. If there are bugs in released code, consumers of the API should file against them. In any respect, as I mentioned in a previous comment when a resource is selected you can view its generated source via Edit | View Java Source.

mrkeen6 anni fa

> What happens when this generated code has a bug?

That's a bug in the generator and should be treated as such. Patch or throw out the generator. It's not sustainable to review and patch "generated code", because then it's just "code".

The only reason I need code-on-disk is if I need to generate bindings into multiple languages.

abeppu6 anni fa

But looking at the generated code, in the specific context where it was produced, is possibly a key step to understanding the bug in the generator. How else are you going to understand how to patch it, or whether the it's bad enough that you have to "throw out the generator" (if you can afford to do that)?

mrkeen6 anni fa

I misspoke when I said "Patch [...] the generator." I meant to say "Have someone else patch it". I meant to imply your time is too valuable to look at the intermediate source.

I'd drive a car to work to save time. If I have to get out and push the car instead, that's happening no more than once. I ditch the car, because it no longer saves me time. Getting out and pushing the car is not going to become part of my commute, and I'm not going to evaluate future cars on how easy they are to push.

You can afford to throw out the generator because it doesn't claim to give you new functionality. It just cuts down on boilerplate. If it can't do that properly, write the boilerplate.

marktangotango6 anni fa

I'm curious how this works under the hood, looks like it's added as an annotation processor, and the DSL is embedded in specially formatted comments, but not in annotations. This implies to me the annotation processor is processing the comments in a source file? I did not know annotation processors could do this! Also implies comments carry through to the compiled .class files which I did not think they did either (runtime bytedcode weaving)?

mckinney6 anni fa

Hi, you can blame me for this. It's a Java compiler plugin[1], which is similar to an annotation processor, but can hook into the compiler at a much earlier stage, which allows it to contribute to all phases of the compiler, including the Parser phase. The Manifold plugin takes full advantage of this, hence its ability to analyze comments, contribute to bytecode generation, etc.

[1] https://docs.oracle.com/javase/8/docs/jdk/api/javac/tree/com...

teslalang6 anni fa

Amazing! Reading the posted link this looks much bigger than embedding, which is crazy impressive btw. Do you anticipate building support for more resources? I notice you have GraphQL support... what about SQL?

eatonphil6 anni fa

So you have a javascript frontend hooked into the java compiler? Or how does your javascript integration work?

mckinney6 anni fa

At compile-time manifold-js[1] parses JS and generates Java types (stubs), which forward execution to rhino at runtime. Basically, JS seamlessly mapped onto Java's type system.

[1] https://github.com/manifold-systems/manifold/tree/master/man...

lmilcin6 anni fa

Except it is not really embedded in Java, only in comments.

I sincerely hope this dies a quick death.

Java with its jars and infrastructure already makes it plenty convenient to work with resources without them having to be in your face when you are editing the file. Good IDE lets you move between the resource file and the code with a single click.

mckinney6 anni fa

> Good IDE lets you move between the resource file and the code with a single click.

Well, not exactly. Most IDEs do NOT support clicking through a generated method call to the corresponding element in a resource file. Manifold is all about that. See it in action: http://manifold.systems/images/graphql.mp4

The Manifold fragments discussed here address only a narrow band of use-cases where embedding the resource improves the dev experience, like with queries and such. It's not for everyone, though.

teslalang6 anni fa

> Except it is not really embedded in Java, only in comments.

Java comments are Java. Embedding = in Java source, which requires some kind of delimiter, so the author conveniently repurposed multiline comments for that which is fine. Also, if you read the post, it clearly states you embed fragments in either comments or strings, if the fragments is a declaration vs a value.

> I sincerely hope this dies a quick death.

Why the hostility? If you have something constructive to offer, great, otherwise this is the kind of comment that gives HN a bad rep.

ulkesh6 anni fa

Yeah, I'm not sure I get it either. More power to the author for solving a problem they may have, but I have yet, in my 20+ years experience with Java, had a need to do this kind of code embedding in the source code (whether in comments or not). Sure, it's nice to be able to run other languages at run-time (e.g. GraalVM, etc), but is there really a need to have this kind of language interoperability at compile time?

mckinney6 anni fa

> but is there really a need to have this kind of language interoperability at compile time?

Well, if you want to leverage Java's static type system (and why not?), the answer is, yes. I imagine you'd want type and member references to the other language to resolve statically using the compiler, right? Similarly, why not have the same functionality in your IDE? Plus code completion, usage searching, refactoring?

Now, as I mentioned in an earlier comment, the embedding part of this addresses just a small segment of use-cases e.g., scoped query editing. The vast majority of other cases work directly against resource files, type-safely. Read more about that here:

https://github.com/manifold-systems/manifold

ulkesh6 anni fa

I’m not at all contesting your reasons, which I expect were plentiful enough to build this system. I’m simply saying that if I want to use Java’s static type system, I’ll just write in Java. Perhaps I’m an old codger these days in my ripe early 40s.

tobyonecanoby6 anni fa

jnellis6 anni fa

You can write other languages like this in Groovy using Intellij. Intellij allows you to annotate a multiline string as another language, then it will parse and syntax color, autoformat for you. The use case would be when you are templating frontend code snippets on the server side. I'm not sure if its just Groovy though, I'm betting Intellij can probably do this with other languages that have multiline strings.

mckinney6 anni fa

Right, but that is just syntax highlighting. This is resolving the content of the string type-safely, at compile-time.

tofflos6 anni fa

Thank you for making this.

Have you considered an approach based on annotations and multi-line strings? I'm not sure string constants are valid targets for annotations but maybe that could be added to the language?

    @Language(name="Javascript")
    """
        function callBark(aBarker) {
            aBarker.bark();
        }
    """

mckinney6 anni fa

Thank you!

Yes, Manifold already supports Java 15 multi-line strings (aka text blocks) like this:

    var myValue = """
    [>.js<]
      function foo() {
        return "hi";
      }  

      foo();
    """;

You can embed a resource fragment as either a declaration or a value. You use a string to embed a value fragment as with the JS example above. Note the [>.js<] header indicates the resource type for the string, similar to your annotation. As you surmised, an expression such as a string literal cannot be annotated.

qisyou6 anni fa

This is cooool!