Don't match keywords if they are preceded by a dot and add the 'sequence' keyword

[matteocoder]

I added a negative lookbehind to the regexes of most control keywords, so that they do not match when they are an object property or method. I added a few more relevant tests as well.

Also, I added the 'sequence' keyword, and 'process' is now considered a block keyword.

This PR fixes a couple more tests in the syntax test, and should fix #105.

[michaelblyons]

Is this compatible with ST's "fast" regex engine? (As opposed to the "slow" Oniguruma one?) You can test with the regex compatibility build variant.

[matteocoder]

@michaelblyons I just checked and negative lookbehind is compatible just with Oniguruma. Is it still okay, or is there a better regex construct that would achieve the same result but allow compatibility with the new engine?

[michaelblyons]

You need to make sure that the possibly-keyword is already consumed by a match on the ..

Is there somewhere that is not correctly pushing the members context, or perhaps an inclusion in members that should not be there?

[matteocoder]

@michaelblyons Perhaps it's the fact that the members context is lower on the syntax file, and so the syntax engine sees the keyword matching first and disregards the members context? I'm a newbie at this, so I apologise if I'm saying something wrong.

[michaelblyons]

No, patterns are matched in the order in which they appear in the current set of matches in the tip.of the context stack.

foo:
  # matches first
  - include: bar
  # matches after, even though context is defined first
  - include: any

any:
 - match: \w+
   scope: any

bar:
 - match: \bbar\b
   scope: bar

For a friendly intro to TextMate-like engines, see this post I wrote for a VS Code question. (There are probably ST guides, too, but I don't know links to those off the top of my head.)

[matteocoder]

@michaelblyons Thank you! the post was really informative. Looking at the syntax test, there's really nothing to push the context:

Maybe I could create a typename scope which matches the argument of "-typename", which will then push the members context?

[michaelblyons]

I think that's more specific than you want to be, here. At most, I think we'd want to identify a command use, and then push into a context for its arguments. Then pop when we find a {terminator, pipe, unescaped newline, etc.} and identify --prefixed items versus other items.

This is not necessarily an easy thing to do. I intentionally did not attempt it when I fixed scopes in 4.0. You are welcome to try, but it may be difficult. The Bash syntax shipped with ST does it this way.

[matteocoder]

diff --git a/PowerShell.sublime-syntax b/PowerShell.sublime-syntax
index 8d1941b..fab5aad 100644
--- a/PowerShell.sublime-syntax
+++ b/PowerShell.sublime-syntax
@@ -75,6 +75,9 @@ contexts:
         - include: main
     - match: \b[\w.-]+\.(?i:exe|com|cmd|bat)\b
       scope: variable.function.powershell
+    - match: \b[\w<>_]+(?=\.)
+      scope: storage.type.powershell
+      push: members
     # Exceptions
     - match: \b(?i:throw){{kebab_break}}
       scope: keyword.control.exception.raise.powershell

Indeed, I'm finding this quite tricky. How about the solution outlined above? It consumes the dot before the keywords are matched, and pushes the members context correctly, while also being compatible with the new engine. The existing tests pass too. The only problem I see with this solution is that file names are incorrectly recognised as storage.type. It's frustrating because to PowerShell foo.baz could be a type name or a file name.

If you're unsatisfied with this solution, I could make another PR with only the changes you're comfortable with integrating into the codebase.

[michaelblyons]

New Bash just scopes arguments like that as strings since a syntax engine can't really know what they are. And for the purposes of normal use, they're unquoted strings.

Alternatively, you could just remove the scope and consume without applying a scope.

[matteocoder]

Wonderful! I thought scoping was always necessary. I removed the scope setting and the lookbehind isn't necessary anymore, so I removed that as well.

diff --git a/PowerShell.sublime-syntax b/PowerShell.sublime-syntax
index 8d1941b..6645f88 100644
--- a/PowerShell.sublime-syntax
+++ b/PowerShell.sublime-syntax
@@ -75,6 +75,8 @@ contexts:
         - include: main
     - match: \b[\w.-]+\.(?i:exe|com|cmd|bat)\b
       scope: variable.function.powershell
+    - match: \b[\w<>_]+\b(?=\.)
+      push: members
     # Exceptions
     - match: \b(?i:throw){{kebab_break}}
       scope: keyword.control.exception.raise.powershell

Do you think this change is good?

[matteocoder]

@michaelblyons I apologise for double posting, but I decided to alter the regular expression a little by taking into account all the character classes allowed by the C# spec.

diff --git a/PowerShell.sublime-syntax b/PowerShell.sublime-syntax
index 8d1941b..9abc03c 100644
--- a/PowerShell.sublime-syntax
+++ b/PowerShell.sublime-syntax
@@ -23,6 +23,7 @@ variables:
   integer_suffix: (?i:ul?|lu?)
   bytes_unit: (?i:[kmgtp]b)
   kebab_break: (?![\w-])
+  csharp_allowed_chars: \p{Lu}\p{Ll}\p{Lm}\p{Nl}\p{Nd}\p{Pc}\p{Mn}\p{Mc}\p{Cf}
 
 contexts:
 
@@ -75,6 +76,10 @@ contexts:
         - include: main
     - match: \b[\w.-]+\.(?i:exe|com|cmd|bat)\b
       scope: variable.function.powershell
+    # Consume a string with a trailing dot
+    # to prevent members with particular names from being recognized as keywords.
+    - match: \b[{{csharp_allowed_chars}}]+\b(?=\.)
+      push: members
     # Exceptions
     - match: \b(?i:throw){{kebab_break}}
       scope: keyword.control.exception.raise.powershell

If you're okay with these changes, I can push another commit which incorporates your feedback.

By the way, is there something weird going on with the regex matching? I'm testing the pattern [\p{L}\p{N}]+, which in theory should match one or more characters, but Sublime Text is showing me that it is matching just a few:

[michaelblyons]

Looks like ST is trying to match p, n, l, {, and }, no?

[matteocoder]

Ah interesting. I took that regex from the storage.type assertion, and I thought it would match all unicode letters and digits.

[michaelblyons]

ST has 3 different regex engines, iirc, in different places. Maybe Find uses one where character classes aren't valid in a character set?

[matteocoder]

Thank you for you help! Indeed, it seems that find and replace use another library for regex matching.

I've pushed my changes containing your feedback. I'm looking forward to hearing your opinion.

[matteocoder]

@deathaxe @michaelblyons Done!