fiddlosopher

5 miesięcy temudiscuss

The web app can be found at https://pandoc.org/app . This has almost the full power of the command-line app, subject to the limitations imposed by the WASM sandbox. Thus:

- the app cannot fetch resources using HTTP

- JSON filters are not supported, as they involve external programs

- Lua filters are supported, but only if they don't try to do system IO operations or run external programs

- Output to PDF is supported only via Typst (using the WASM version of Typst).

fiddlosopher

7 miesięcy temudiscuss

Pandoc does know how to expand LaTeX macros. For example, given the LaTeX

  \newcommand{\pair}[2]{\langle #1, #2\rangle}
  $$\pair{a^2}{\frac{\pi}{2}}$$

pandoc will give you the Typst

  $ chevron.l a^2 \, pi / 2 chevron.r $

which is correct. Tylax, on the other hand, seems to have problems with this example, producing

  $ angle.l^()frac(pi,)angle.r  $

which does not compile with typst. Going the other direction, pandoc also understands typst scripting. For example, from

  #let count = 8
  #let nums = range(1, count + 1)
  #let fib(n) = (
    if n <= 2 { 1 }
    else { fib(n - 1) + fib(n - 2) }
  )

  The first #count numbers of the sequence are:

  #align(center, table(
    columns: count,
    ..nums.map(n => $F_#n$),
    ..nums.map(n => str(fib(n))),
  ))

pandoc produces this LaTeX:

  The first 8 numbers of the sequence are:

  {\def\LTcaptype{none} % do not increment counter
  \begin{longtable}[]{@{}llllllll@{}}
  \toprule\noalign{}
  \endhead
  \bottomrule\noalign{}
  \endlastfoot
  \(F_{1}\) & \(F_{2}\) & \(F_{3}\) & \(F_{4}\) & \(F_{5}\) & \(F_{6}\) &
  \(F_{7}\) & \(F_{8}\) \\
  1 & 1 & 2 & 3 & 5 & 8 & 13 & 21 \\
  \end{longtable}
  }

With the same input, Tylax produces:

  The first 8 numbers of the sequence are:

  \begin{center}

  \begin{tabular}{|c|}
  \hline
  \hline
  \end{tabular}\end{center}

which is just an empty table.

fiddlosopher

w zeszłym rokudiscuss

Pandoc developer here. Two comments: (1) to convert equations pandoc uses the texmath library, which I also wrote (https://github.com/jgm/texmath). Compiling texmath with `-fexecutable` will give you a standalone executable that just converts the equation (and doesn't add the `<p>` element or anything extraneous). Compiling it with `-fserver` will give you a webserver that converts equations. (2) Regarding the bug, note that it's not an empty `<mo></mo>`. There's an invisible U+2061 character ("function application") inside. We don't want to take that out, but it looks like putting the `<mi>` and the `<mo>` together in an `<mrow>` will also solve the problem. I'll fix this.

fiddlosopher

2 lata temudiscuss

An even older one is my gitit, started in 2008!

https://hackage.haskell.org/package/gitit-0.15.1.2

It doesn't limit itself to markdown, nor to git (you can use darcs, hg, or even sqlite). A bit long in the tooth, though -- I stopped working on it once spam started to make self-hosted public wikis untenable.

fiddlosopher

3 lata temudiscuss

This was my first computer, too. I have fond memories of programming it in 6502 machine language and saving programs on cassette tape. I still have it, along with the power supply my grandfather built from the specs they provided in the user manual. And it still works!

fiddlosopher

3 lata temudiscuss

As I explained, if you don't want raw HTML, use the second form:

  pandoc -f html -t gfm-raw_html https://www.fsf.org | grep div

gives no output. (If you get different results, it is possible that you are using an earlier pandoc version and something changed in this regard.)

fiddlosopher

3 lata temudiscuss

If you don't want pandoc's fenced divs, just choose a markdown dialect that doesn't support them: e.g.,

    pandoc -f html -t gfm https://www.fsf.org

Or, to get rid of the divs and spans altogether, disable raw HTML as well:

    pandoc -f html -t gfm-raw_html https://www.fsf.org

fiddlosopher

3 lata temudiscuss

In fact, both claims are misleading. There are two implementations, and they implement distinct flavors with syntactic differences: https://consolelog.gitee.io/docs-asciidoctor/asciidoc-asciid... Markdown has a smaller flavors/implementations ratio!

fiddlosopher

4 lata temudiscuss

Oh, but there is!

    pandoc --shift-heading-level-by=-1 input.md -o output.docx

This will promote level-2 headings to level-1, and promote a level-1 heading at the top of the document to the document's title.

fiddlosopher

4 lata temudiscuss

The motivation for this choice is not "convenient parsing" but deeper considerations of language design. As explained at https://github.com/jgm/djot#rationale , this choice follows from two desiderata: (1) "The syntax should compose uniformly, in the following sense: if a sequence of lines has a certain meaning outside a list item or block quote, it should have the same meaning inside it." (2) "The syntax should be friendly to hard-wrapping: hard-wrapping a paragraph should not lead to different interpretations, e.g. when a number followed by a period ends up at the beginning of a line." The document explains the compromise we made in commonmark to avoid the need for blank lines. Djot tries to be more principled.

fiddlosopher

4 lata temudiscuss

> Markup language which completely falls over this is Markdown. There’s no way to express generic tree structure, conversion to HTML with specific browser tags is hard-coded.

This isn't really a fair criticism. True, the original Markdown.pl did not produce a generic tree structure, but that's a fact about the program, not the syntax it parses. Many Markdown and Commonmark implementations do support creation of an abstract syntax tree. Pandoc has done this for the last 17 years. It also provides nestable, generic containers as a syntax extension.

> It feels like there’s a smaller, simpler language somewhere

Here's my attempt: <https://djot.net>.

fiddlosopher

4 lata temudiscuss

Note also that TeX math can contain \text{..} which can itself contain $-delimited TeX math, e.g. $x = \text{my $y$}$. This currently breaks the GitHub implementation.

fiddlosopher

5 lat temudiscuss

In pandoc's extended Markdown you can just write

    # Usage

    See [Usage].

https://pandoc.org/MANUAL.html#extension-implicit_header_ref...

fiddlosopher

6 lat temudiscuss

In fact, pandoc now does have an ipynb reader and writer. You can convert between ipynb and any of the formats pandoc supports, and you can manipulate the AST using filters.

fiddlosopher

11 lat temudiscuss

See https://github.com/jgm/pandoc/issues/1456.

fiddlosopher

11 lat temudiscuss

AsciiDoc is a nice project, but I think that pandoc's variant of Markdown has a lot of advantages over AsciiDoc for academic writing. Let's just compare support for math and citations, for example. Pandoc's citation support is output-format independent: you can write the citations, specify a CSL stylesheet and bibliography, and you'll get the same output in every format pandoc supports. In AsciiDoc, as I understand it, if you want automatic citation support you need to use LaTeX, and then you're limited to output in LaTeX and PDF. Similarly for math: pandoc actually converts your LaTeX math to MathML (for formats that like that) or to native Word equations (for docx); in AsciiDoc, as I understand it, your LaTeX math will work if you target LaTeX, but if you want MathML you need to put MathML in the source. If you want to target both, then maybe you need two different source documents?

I don't have much experience with AsciiDoc, but I've encountered other limitations in writing pandoc's AsciiDoc renderer. For example, I didn't see any way to include multiple paragraphs (or other block content) in an AsciiDoc footnote. That's a deal-breaker for many academics.

I also want to emphasize something that is often not mentioned in comparisons of Markdown and AsciiDoc. As John Gruber emphasizes in his Markdown documentation, Markdown emphasizes ease of reading in source format. AsciiDoc has different priorities, and it sacrifices the readability of the source document to get them. Here's an example involving nested lists, from the AsciiDoc manual:

    1. List item one.
    +
    List item one continued with a second paragraph followed by an
    Indented block.
    +
    .................
    $ ls *.sh
    $ mv *.sh ~/tmp
    .................
    +
    List item continued with a third paragraph.
    
    2. List item two continued with an open block.
    +
    --
    This paragraph is part of the preceding list item.
    
    a. This list is nested and does not require explicit item continuation.
    +
    This paragraph is part of the preceding list item.
    
    b. List item b.
    
    This paragraph belongs to item two of the outer list.
    --

Markdown equivalent:

    1.  List item one.
    
        List item one continued with a second paragraph followed by an
        Indented block.
    
            $ ls *.sh
            $ mv *.sh ~/tmp
    
        List item continued with a third paragraph.
    
    2.  List item two continued with an open block.
    
        This paragraph is part of the preceding list item.
    
        1.  This list is nested and does not require explicit item continuation.
    
            This paragraph is part of the preceding list item.
    
        2. List item b.
    
        This paragraph belongs to item two of the outer list.

Note that the AsciiDoc is at least as easy to write, perhaps easier, because you don't need to worry about indentation. But the Markdown source is more readable; the indentation makes clear the structure of the list in a way that mirrors how it would be displayed in a browser or on the page.

fiddlosopher

11 lat temudiscuss

"For example, they do not plan to add syntax highlighting blocks (the ```some code``` on GitHub) to their implementation, because they believe that it is outside the scope of markdown."

To correct the record, fenced code blocks have been there from the beginning: http://spec.commonmark.org/0.18/#fenced-code-blocks.

fiddlosopher

11 lat temudiscuss

Although I'm involved in both these enterprises, let's not confuse their goals.

CommonMark is currently focused on the task of giving a decent spec for the core syntax and robust, efficient implementations; extensions will wait til that project is done, but certainly aren't ruled out.

Pandoc has always been in the game of extending the feature set. Here are just some of the Markdown extensions pandoc supports: LaTeX math (which can be rendered in a variety of formats, including native Word and MathML), LaTeX macros, inline LaTeX, automatically numbered examples and cross-references to these, automatically generated citations (using CSL styles), super and subscripts, strikeout, figures, YAML metadata, definition lists, several styles of tables, fenced code blocks with syntax highlighting, header identifiers, and footnotes. scholdoc just adds a few things on top of all this (and many of them could be implemented in pandoc filters). As noted in one of the other comments on this thread, most of the features scholdoc adds are under active discussion in pandoc as well. So it's not that pandoc and scholdoc have different aims; pandoc just moves more slowly, because it has to worry about how features are implemented in many more output formats, and it operates under some other constraints that scholdoc rejects (e.g. trying to avoid the use of English words like "Figure" for syntax cues).

fiddlosopher

12 lat temudiscuss

Your comments (coming from someone who has actually tackled this surprisingly difficult task) are some of the most valuable we've received; having them on the Discourse forum would be great.

We considered writing the spec in the state machine vein, but I advocated for the declarative style. It may be worth rethinking that and rewriting it, essentially spelling out the parsing algorithm. As you suggest, a parallel document could be created for writers.

I'll need to study your spec further to see what the substantive differences are.

fiddlosopher

12 lat temudiscuss

If you think there are ambiguities, please comment at http://talk.standardmarkdown.com. This is meant to be a provisional spec, up for comment. There is undoubtedly room for improvement.

The C and javascript implementations use a parsing algorithm that we could have simply translated into English and called a spec. (That's the sort of spec vfmd gives.) But it seemed to us that there was value in giving a declarative specification of the syntax, one that was closer to the way a human reader or writer would think, as opposed to a computer.

Re (3): we have an asterisk which can open emphasis. So, to see if we have emphasis, the rules say to parse inlines sequentially until an asterisk that can close emphasis is reached. The first inline we come to is [b*](url), which is a link. There's no closing asterisk, so we don't have emphasis, but a literal asterisk followed by a link.

Re (1): I believe you are right that the case of a referenc e definition before a setext header line should be clarified. However, the other case seems clear enough. ~~~ starts a fenced code block, which ends with a closing string of tildes or the end of the enclosing container. The underline would be included in that code block either way.

Re (2): I believe the talk of precedence may be misleading here (I thought it would be useful heuristically). The basic principle of inline parsing is to go left to right, consuming inlines that match the specs. This resolves all of these cases. Perhaps the talk of precedence should be removed.

I am no stranger to formal specifications. I wrote what I think was the first PEG grammar for markdown (peg-markdown, which came to be used as the basis for multimarkdown and several other implementations). PEG isn't a good fit, especially for block-level parsing. It almost works for inline-level parsing, but there are some constructs (like code spans) that can't be done in PEGs. It might be worth specifying inline parsing in a pseudo-PEG format to avoid worries like those you've expressed.

fiddlosopher

Submissions

Beyond Markdown

Beyond Markdown

comments