Python Rules -More flattened flags

[hkandjimi]

@jrfaller I have been playing around with the python rules for a while and tested them local which have worked fine. However, when I include the rules and compile a docker image I keep getting the error :
"PythonTreeSitterNgTreeGeneratorTest > testString() FAILED
org.opentest4j.AssertionFailedError: expected: <12> but was: <8>"

I have inspected the testString() in PythonTreeSitterNgTreeGeneratorTest class but not sure which features were counted to get to the 12.

Here are the new rules for python:
python_rules.txt

Kindly let me know if this would work or if I can change the assert check from 12 to 8 and possibly push the changes

[jrfaller]

Hi @hkandjimi and thanks a lot for the PR.

I tried a diff to look for the new rules and I have this:

python:
  aliased:
    dict: collection_type
    list: collection_type
    set: collection_type
    tuple: collection_type
  flattened:
    - identifier
    - integer
    - collection_type
    - float
    - boolean_operator
    - comparison_operator
    - expression_statement
    - assignment

I am not sure about the rules. Flattening boolean_operator, comparison_operator, expression_statement and assignment does not seems like a good idea since it's usually interesting to diff subexpressions. Integer, identifier and float are already leaf nodes.

Regarding the aliased types (dict, etc...) I am not sure which source snippet would make them appear, could you provide a code snippet?

Regarding the asserts it's a mere sanity check and it's supposed to change each time the rules change so no worries here.

Cheers!

[hkandjimi]

Here is the current rules specification on my local build.

python:
  flattened:
    - string
    - float
    - integer
    - collection_type
    - identifier
    - boolean_operator
    - comparison_operator
    - expression_statement
    - binary_operator
    - definition    
  aliased:
    '==': comparison_operator
    '<=': comparison_operator
    '>=': comparison_operator
    '!=': comparison_operator
    '<': comparison_operator
    '>': comparison_operator
    '<>': comparison_operator
    +=: affectation_operator
    -=: affectation_operator
    '*=': affectation_operator
    /=: affectation_operator
    =: affectation_operator
    '|=': affectation_operator
    '&=': affectation_operator
    '^=': affectation_operator
    '-=': affectation_operator
    '+=': affectation_operator
    'and': logical_operator
    'or': logical_operator
    'not': logical_operator
    '~': bitwise_operator
    '&': bitwise_operator
    '|': bitwise_operator
    '^': bitwise_operator
    '<<': bitwise_operator
    '>>': bitwise_operator
    list: collection_type
    dict: collection_type
    tuple: collection_type
    set: collection_type
    function_definition: definition
    class_definition: definition
    lambda: definition
  ignored:
    - '('
    - ')'
    - '{'
    - '}'
    - '['
    - ']'
    - '.'
    - ':'
    - ','
    - 'default_parameter ='
    - 'import'
    - 'def'
    - 'for'
    - 'in'
    # - 'if'
    - 'with'
    - 'as'
    - 'return'
    # - 'elif'
    # - 'else'
    - 'while'
    - 'try'
    - 'except'
    - 'finally'
  label_ignored: []

And below are the versions of source codes I have been attempting to diff
Version 1:

valid = True

month = input("Enter the name of a month (e.g. January, ..., December):\\n")

startDay = eval(input("Enter the start day (1 for Monday, ..., 7 for Sunday):\\n"))

amountOfDays = 0
counter = 1

month = month.lower()

#block to assign values to the parameters
if month == "january" or month == "march" or month == "may" or month == "july" or month == "august" or month == "october" or month == "december":
    amountOfDays = 31
elif month == "february":
    amountOfDays = 28
elif month == "april" or month == "june" or month == "september" or month == "november":
    amountOfDays = 30
elif startDay >= 1 and startDay <= 7:
    print("Invalid calendar: you have either entered an incorrect month name or start day.")
    valid = False
else:
    print("Invalid calendar: you have either entered an incorrect month name or start day.")
    valid = False

    
if valid:
    print(month.capitalize())
    print("Mo Tu We Th Fr Sa Su\\n")    
    #for loop which adds padding to the beginning of the calendar
    for i in range((startDay - 1) * 3):
        print(" ", end="")

    #main for loop to add the day numbers
    for i in range(startDay, startDay + amountOfDays):
        if i % 7 == 0:
            if counter <= 9 and counter >= 0:
                print(" ", i - startDay + 1, sep="", end="\\n")
            else:
                print(i - startDay + 1, sep="", end="\\n")        
        else:
            if counter <= 9 and counter >= 0:
                print(" ", i - startDay + 1, sep="", end=" ")
            else:
                print(i - startDay + 1, sep="", end=" ")    
            
        counter += 1

Version 2

valid = True

isException = False
     
month = input("Enter the name of a month (e.g. January, ..., December):\\n")

startDay = eval(input("Enter the start day (1 for Monday, ..., 7 for Sunday):\\n"))

amountOfDays = 0
counter = 1 + 5

month = month.lower()

#block to assign values to the parameters
if month == "january" or month == "march" or month == "may" or month == "july"  or month == "october":
    amountOfDays = 31
elif month == "february":
    amountOfDays = 28
elif month == "june" or month == "september" or month == "november":
    amountOfDays = 30
elif month == "april" or month == "august" or month == "december":
    amountOfDays = 31
    isException = True

if startDay <= 1 or startDay >= 7 or amountOfDays == 0:
    print("Invalid calendar: you have either entered an incorrect month name or start day.")
    valid = False

    
if valid:
    print(month.capitalize())
    if isException:
        print("Mo Tu We Th Fr Sa Su")    
    else:
        print("Mo Tu We Th Fr Sa Su\\n")    
    
    #for loop which adds padding to the beginning of the calendar
    for i in range((startDay - 1) * 3):
        print(" ", end="")

    #main for loop to add the day numbers
    for i in range(startDay, startDay + amountOfDays):
        if i % 7 == 0:
            if counter <= 9 and counter >= 0:
                print(" ", i - startDay + 1, sep="", end="\\n")
            else:
                print(i - startDay + 1, sep="", end="\\n")        
        else:
            if counter <= 9 and counter >= 0:
                print(" ", i - startDay + 1, sep="", end=" ")
            else:
                print(i - startDay + 1, sep="", end=" ")    
            
        counter += 1

My current build with these rules works fine, and my focus is to have as many flattened nodes as possible to avoid having to do further analysis.

[jrfaller]

OK, thanks a lot for the test case. I think that the rules you are using are flattened too many things for the default configuration, therefore I will look at them and cherry-pick what I think is interesting.
However, it made me realize that it would be very nice to allow users to provide a custom rule file.
Cheers!

[hkandjimi]

Maybe a possible idea would be to include this as part of the options provided, just like you are able to pick your own parser; one could possibly select a preferred rules file.