Split StringTemplate from HTML

[Nixinova]

#4789 (comment): StringTemplate was added in #1117 as a raw HTML variant, but per #4979 HTML variants should be split.

It seems to be HTML but with control statements and variables inside $dollars$.

Default to using default HTML grammar, which is not great as it doesn't match the dollar syntax at all, but couldn't find an actual one that works.
Found a grammar: dangmarm/stringtemplate-lang
UPDATE (Alhadis): Added grammar for StringTemplates that use either <…> delimiters or $…$ delimiters.

Color #3fb34f from logo.

Closes #4789

[Nixinova]

Ah, found a grammar: dangmarm/stringtemplate-lang

[Nixinova]

Can't CTRL+F the logs but it's the same issue as #5194's failed run I assume.

[Nixinova]

Lightshow doesn't seem to work..

[lildude]

Lightshow doesn't seem to work..

Looks to be a problem with the grammar and not Lightshow. Picking another grammar source works.

[Nixinova]

Should I just revert the grammar changes then? Can't find a better one.

[Alhadis]

[StringTemplate] seems to be HTML but with control statements

From a quick glance at their format's documentation, it seems that StringTemplate files can generate any sort of output, similar to Moustache (even if it's most commonly used to generate HTML).

Ergo, a dedicated StringTemplate grammar would be more appropriate than Atom's HTML grammar, and it shouldn't take me too long to smash one together. I'll hold off on it until @lildude's finished deploying v7.13.0, just in case an update to Alhadis/language-etc screws with the procedure somehow. 😉

[lildude]

All done @Alhadis. Go wild.

[Alhadis]

Go wild.

Roger. Will push an update to @Nixinova's branch that contains the required grammar update.

I'm wondering if there should be an undocumented script/bump-grammar utility to streamline this procedure… 🤔

[Nixinova]

it seems that StringTemplate files can generate any sort of output

Should I improve the StringTemplate heuristic then?

[Alhadis]

Should I improve the StringTemplate heuristic then?

What we have now should be sufficient. The delimiter characters are configurable, so <word> is no less valid than $word$, |word|, [word] or ^word^. StringTemplate's most common use appears to be preparing HTML, and the docs recommend $…$ for HTML output:

This documentation uses <...> to delimit expressions, but you can use any single start and stop character. For HTML, $..$ is a much better choice obviously.

The only thing I'd suggest adding is StringTemplate's <! comment !> syntax:

pattern: >-
    (?x-m)

    # Symmetrical delimiter: $…$ '…' #…# ~…~
    (.)! .+? !\1

    # Mirrored delimiters: <…> […] {…} “…” «…»
    |  <! .+? !>
    | \[! .+? !\]
    | \{! .+? !\}

Skippable sidenotes that you should read anyway

Now, I'd like to take a moment to explain a small but critical nuance about the behaviour of . in Ruby regexes. See, Ruby/Oniguruma's multi-line mode changes the semantics of . so it matches newline characters. This differs from other regex engines, which generally use a separate modifier to enable dotall mode (/s in Perl/PCRE and JavaScript). As a result, this has severe performance implications when processing large files with .+—especially when it's used repeatedly in an expression.

Now, while possessive matching (.++, (?>.+)) helps, an even better solution is to double-check if . should match newlines in the first place. If not, then [^\r\n] or (?-m:.) can have substantial performance improvements. This is mostly relevant to Ruby/Oniguruma, because the default behaviour of . in other engines is not to match newline characters.

Examples of when . should match newlines

Good

<zalgo>.*?</zalgo>

Bad

async function .+\(.*\)

Slightly less bad

async function .+?\(.*?\)

Why only slightly? .+? only affects the performance of successful matches. A failed match means the rest of the input is scanned, which is as slow as if .+ were used.
Solution: use possessive quantifiers (.++) or atomic groups ((?>.+)) to fail faster.

Okay, enough of my rambling bullshit. ^D

[Nixinova]

Is what I added sufficient?

[Alhadis]

@Nixinova Sorry I took so long to write that StringTemplate grammar. The language turned out to be more complex than I assumed it was.

I've pushed an update to your branch to use the new grammar, so, no actions are required on your part. 👍