Token highlighting issues

[shawna-p]

As of v2.0.16 of the original extension, there are numerous issues with token highlighting, particularly with screens, styles, and text tags. Here are several comparisons: the first image is the highlighting from 2.0.16, and the second is the highlighting from 2.4.0 (current version as of this moment). Then I'll post the actual text used for the examples for convenience.

Dialogue & Script

define e = Character("Eileen")
default variable = True
label start:

    scene bg room
    show eileen happy
    e "Here is an example {b}of some text tags{/b}."
    e "You should go to {a=https://www.renpy.org}the Ren'Py website{/a}."
    e "Here is a variable: [variable]."
    e "You can also use arguments." (window_xpos=300)
    e "Style text tags {=my_style}should also work properly.{/=my_style}"
    e "{size=+20}{font=DejaVuSans.ttf}{color=#00F}Tags{/color}{/font}{/size}"
    return

style my_style:
    color "#F0F"
    bold True
    italic True
    outlines [(1, "#000000", 0, 0)]

Issues:

• text tag braces are the same colour as dialogue
• Some text tag content is the same colour as dialogue (e.g. the colour code and font name)
• Closing style text tags are not coloured at all
• Dialogue arguments have no highlighting

Screens

screen history():

    tag menu 

    predict False

    use game_menu(_("History"), scroll=("viewport" if not gui.history_height else "vpgrid"),
        yinitial=1.0, spacing=gui.history_spacing):

        style_prefix "history"

        for h in _history_list:

            window:

                fixed:
                    yfit True

                    if h.who:

                        label h.who:
                            style "history_name"
                            substitute False

                            ## Take the color of the who text from the 
                            ## Character, if set.
                            if "color" in h.who_args:
                                text_color h.who_args["color"]

                    $ what = renpy.filter_text_tags(h.what, allow=gui.history_allow_tags)
                    text what:
                        substitute False

        if not _history_list:
            label _("The dialogue history is empty.")

Issues:

• Missing highlighting for several words (predict, substitute, fixed, yfit, text_color
• Issue regarding the original, however - label has inconsistent highlighting (sometimes it appears to be recognized as though it is a script label)

screen test_screen(arg1, arg2=None):

    if not viewing_guest:
        # This is the overlay for Story Mode
        # that shows the Auto/Skip/Log buttons
        hbox:
            if persistent.vn_window_alpha < 0.1:
                yoffset -275+20
            imagebutton:
                idle Text("Auto", style="vn_button_hover")
                hover Text("Auto", style="vn_button")
                selected_idle Text("Auto", style="vn_button")
                selected_hover Text("Auto", style="vn_button_hover")
                action Preference("auto-forward", "toggle")

        textbutton "Example button":
            pos (30, 400) anchor (0.5, 0.5)
            text_outlines [(1, "#000000", 1, 1)]
            action Return()
            text_color "#FFFFCD"

        text arg1
        text "Some sample text [arg2]":
            outlines [(3, "#FF0")]
            color "#000"

        viewport:
            yinitial 1.0
            scrollbars "vertical"
            mousewheel True
            draggable True
            side_yfill True
            ysize 900
            yalign 1.0

            vbox:
                style_prefix "history"
                spacing 20
                null height 5
                for h in _history_list:
                    if h.who is not None:
                        text h.who
                    text h.what

Issues:

• Many more missing highlights: all the imagebutton properties except action, all text properties regardless of whether they're prefixed with text_ or not, yinitial, side_ prefixed properties like side_yfill, null height 5
• Secondary issue: screen language displayables like vbox, textbutton, viewport should not be the same colour as screen language properties (pos, yoffset, mousewheel etc.). That colour is already shared by many other grammar tokens like if, not, and, etc. and it becomes oversaturated

Python

init python:
    
    class AdjustTrackPos(Action):
        """
        A custom action which uses partial playback to start playback at the
        provided seconds argument. musicroom should be a Music Room object or
        the channel name the music is being played on.
        """
        def __init__(self, seconds, musicroom):
            self.seconds = seconds
            self.musicroom = musicroom
            try:
                self.channel = musicroom.channel
            except:
                self.channel = musicroom
        def get_sensitive(self):
            return (renpy.music.is_playing(channel=self.channel)
                and not preferences.get_mute(self.channel)
                and preferences.get_volume(self.channel))
        def __call__(self):
            adjust_track_pos(self.seconds, self.musicroom)
            renpy.restart_interaction()

Issues

• This one is fine I think but I'm including it for comparison

Conclusion

Personally, I find the inconsistency and reused token colour introduces enough visual confusion that I can't use the extension in its current state. The commit 2f508c0 (found here) begins to introduce the token highlighting problem. It may be worthwhile to revert back to the commit where the tokens were working properly and work forwards to introduce any later features while checking for highlighting inconsistencies. I suggest these examples be worked into the test code files as well - there seems to be a lack of representation for screen language and text tags in particular in testing.

[duckdoom4]

Hi, thanks a lot for the very detailed bug report!

I'll be fixing many of these.

• Dialog arguments should have highlights
• I'll fix closing style tag not being highlighted
• I'll add the missing keyword 'fixed'
• I'll add the missing style properties

The text tags will need to wait. The previous version had a major flaw that I was not able to fix at the time. I might give it another go, but it might not be possible yet.

All of that said, the theme you are using is hiding some of the colorization of certain tokens. Eg. The style properties should be a different color if you use the default theme. And the string/text tags should have a different color for the brackets.

It might be possible to add some additional tokens to support other themes as well, so might take a look at that to fix this issue for your theme.
I see the one used in the string interpolation tags does work for your theme, so I'll add that token to the string tags brackets as well.

[shawna-p]

Understood about the theme - I should have mentioned probably, but the theme in the screenshots is Monokai. That said, it'd be great if it could be updated to use some other style set which didn't conflict with the other common tokens.

Would you be able to elaborate on the issues you are running into with the text tags? I have very little familiarity with language extensions for VS Code, but I'm good with regex and I've been coding for over a decade, so I'm considering putting some legwork in to learning the ins and outs to be able to pitch in, at least to add new keywords as they get added to the engine.

[duckdoom4]

Yeah, the issue with the string tags is related to this issue #255. Basically, I used to test for begin to end, where end would test for a closing tag. But it turns out it is allowed to not close the tags as well, in which case it doesn't end until the string end. (I removed it in this commit #330)

IIRC, the issue was that I wasn't able to code that behavior into the regex because I would need to keep track of the number of opens and closes. However, I recently discovered a new method that allows me to do something like this.
In short, have a look at the uses of 'block-tester' syntax rules. It shows how you could include a rule and 'extend' the parent pattern end.
Edit:
So I tried it again, but I'm still unable to do this with syntax highlighting. The issue is that I can't define an end pattern since the closing tag is optional. And as far as I know, there's no way to describe 'end of parent match'. So this will have to be coded using the parser.
/Edit

You can find the syntax highlighting rules here https://github.com/renpy/vscode-language-renpy/tree/develop/syntaxes.

I'm also working on a parser on a different branch to support semantic highlighting. (Eg. highlighting of variables)

[duckdoom4]

To give you a small example for what it looks like with the default theme (Dark Modern)

[duckdoom4]

I've added most of the fixes I mentioned before with #432.
If you could try out the new version locally and let me know if there are any more issues, that would be awesome ! (See this section if you wanna know how).

Also feel free to contribute any changes you'd like to introduce. I'd be very happy if I could get some help with this project :)