Dynamic arrays of non-managed struct and classic arrays of managed struct #2106

fernewelten · 2023-08-24T10:27:05Z

Fixes #1818 and #2080.

The new compiler can now generate code for dynamic arrays of non-managed structs and for classic arrays of managed structs.

Also, you can define dynamic arrays of dynamic arrays and dynamic arrays of classic arrays. These can be nested at any depth.

When there are pointers of entities that have pointers, then RTTI must be enabled.

Dynamic array of non-managed struct

Typical code:

struct Warrior
{
    int Strength;
    int Health;
};

function room_FirstLoad()
{
    Warrior Lemmings[] = new Warrior[10];
    player.Say("That one's strength is %d", Lemmings[1].Strength);
}

Lemmings in room_FirstLoad() points to one chunk of memory where 10 structs are tucked side-by-side. Lemmings[0], Lemmings[1] etc. will access the respective warriors, no need to initialise them individually with new. But the dynamic array itself must be initialised with new[], as shown in the code above.

If you go that way, then the static struct (in this case, struct Warrior) must not contain any dynamic variable components, not even indirectly – unless RTTI is enabled. For instance, the following code will fail to compile, unless RTTI is enabled:

struct Stats
{
    int Stat[];    // ← dynamic component, not allowed unless RTTI is enabled
};    

struct Warrior
{
    Stats WarriorStats;
};

function room_FirstLoad()
{
    Warrior Lemmings[] = new Warrior[10];
}

Classic array of managed struct

Typical code:

managed struct Warrior
{
    int Strength;
    int Health;
};

function room_FirstLoad()
{
    Warrior Warriors[3, 10];
    for (int nationality = 0; nationality < 3; ++nationality)
        for (int citizen = 0; citizen < 10; ++citizen)
            Warriors[nationality, citizen] = new Warrior;
    player.Say("That one's strength is %d", Warriors[1][1].Strength);
}

Warriors in room_FirstLoad()is a chunk of 30 pointers tucked side-by-side. The classic array itself need not be initialised. But to get 30 usable components, you must initialise each of them with new, as shown in the code above.

The managed struct may not contain dynamic variable components unless RTTI is enabled, but that's nothing new.

Dynamic arrays of dynamic arrays

This will only work when RTTI is enabled.

Here's code that defines and uses a dynamic array of a dynamic array of shorts:

short Test[][]; 
int game_start()
{
    // Initialise a 3 by 3 array of arrays
    Test = new short[3][];
    for (int i = 0; i < 3; i++)
        Test[i] = new short[3];
    // Use an element
    Test[2][1] = 4711;
}

Here's code to define and use a dynamic array of a dynamic array of a managed struct:

managed struct Warrior
{
   int Strength;
   char Values[20];
};

int game_start()
{
    Warrior *Army[][];
    Army = new Warrior[3][];
    Army[2] = new Warrior[3];
    Army[1][2] = new Warrior;
    Army[1][2].Strength = 99;
}

You can nest the dimensions as deeply as you want. So int Field[][][][][][]; is fine. You'll have to initialize the structure as follows:

Field = new Field[7][][][][][];
then Field[i] = new Field[7][][][][]; (for all non-negative values of i that are lower than 7)
then Field[i][j] = new Field[3][][][]; (for all the combinations of i and j where i is lower than 7 and j is lower than 3)
then Field[ì][j][k] = new Field[5][][]; …
… and so on until you come to Field[i][j][k][l][m] = new Field[99];

Limitation:
You can't define a classic array of a dynamic array, e.g., int Field[][3, 5]; is forbidden. Once you've started with the dynamic markers [], you can only add further dynamic markers to the right.

On the other hand, you can define a dynamic array of a classic array, e.g., int Field[3, 5][][];.

ivan-mogilko · 2023-08-24T10:47:17Z

This addresses #1818 and #2080.

Please note that you can use keyword "Fixes" or "Resolves" in the ticket description, which will automatically bind this PR to the issue ticket, and also close the issue ticket when the PR is merged.

E.g.:

Fixes #1818, #2080

ivan-mogilko · 2023-08-24T11:41:12Z

Classic array of managed struct

But was not this already supported, since the beginning? Or did something change in their regard?

ivan-mogilko · 2023-11-24T22:00:12Z

@fernewelten I see that you drafted this PR as something that would solve two feature requests. Is one of them is covered by the changes already? Seeing this has stalled, could that make sense to make a PR only for the first feature for now?

fernewelten · 2023-11-24T22:38:28Z

Thanks for the enquiry. I'm at it. I hope to be done in the next few days.

Consistency, especially in the parser Also, denote unsigned literals as such

Also, some var renamings

It is very common that the compiler knows what symbol should be coming and then skips over it blindly, using 'GetNext()'. Modify these places to call the new function 'SkipNextSymbol()' instead. It takes a symbol that is expected and makes sure that it actually comes next.

…peWith()' Modify 'VartypeWithArray()' to make it possible to join adjacent classic array specifiers '[2, 3][5]' → '[2, 3, 5]' Replace 'VartypeWith()' by the functions - 'VartypeWithConst()' - 'VartypeWithDynarray()' - 'VartypeWithDynpointer()' The semantics of the respected cases has become that different that it doesn't make sense to have a common ‘umbrella’ function for them all.

Symboltable: – Modify 'VartypeWithArray()' to process arrays of arrays Parser: – Allow dynamic arrays of non-managed structs – Disallow dynamic arrays of classic arrays because that would make 'new' expressions ambiguous – Grok 'new[…][]' when initialising dynarrays of arrays – Make multiple indexes big-endian – In 'AccessData_ProcessCurrentArrayIndex()' → '…_ProcessArrayIndex()', grok a single index of a sequence of indexes. Clarify comments Googletests – Retracted the test that checks whether dynamic arrays of non-managed structs are disallowed (they are allowed now)

fernewelten · 2023-11-26T01:51:17Z

So here's the code for the PR.

Note that I've force-pushed.

I've modified the very first comment to include all the functional changes.

As for the changes in the code, I've done five commits: I've done some refactoring to the code, and it's clearer to keep the refactoring changes (e.g., renaming of variables) that don't modify any logic separate from the actual work that adds or changes logic.

ivan-mogilko · 2023-11-26T02:59:01Z

Oh, this addresses even more features than before.

Dynamic arrays of dynamic arrays

This means "jagged arrays" is not it? The relevant ticket was #1258.
Then this PR covers 3 tickets: #1258, #1818 and #2080.

I shall test this soon.

ivan-mogilko · 2023-11-26T03:05:11Z

Hmm, could you clarify something, in the PR's description you say that this fixes #2080, which is "regular arrays of dynamic arrays". But further on you say "You can't define a classic array of a dynamic array". I cannot see any example for #2080 in the ticket's description either. So are these supported now, or not, and what is your opinion on the feature proposal itself?

On another hand, in the PRs title and description you mention "Classic array of managed struct", but these were already supported since the beginning of managed pointers. Did anything change in their regard, why this is mentioned here?

fernewelten · 2023-11-26T12:39:16Z

So this PR isn't about new concepts only. I've gone through the different cases of managed/non-managed entities and new/classic arrays that we offer so far and tried to describe them in a consistent system. Behind-the-scenes, I've refactored the relevant code, and these were the cases that I kept in mind that must all work or keep working.

fernewelten · 2023-11-26T13:41:52Z

When we have a declaration such as int foo[3, 5][]; then there's the question in which direction it is read. Is foo a classic 3-by-5 array where each component is a dynamic array, or is foo a dynamic array where each component is a classic 3-by-5 array?

The way I've implemented it, foo is a classic 3-by-5 array where each component is a dynamic array. In an array of an array, the outermost dimension is specified and addressed first. (I explain my reasoning below. – This needn't stay this way; if there's consensus to switch things around, I'm willing to.)

So foo[0, 0] is an element of a classic 3-by-5 array, which is a dynamic array. To initialise it, we'd need code such as foo[0, 0] = new int[100];. To initialise all the elements of the 3-by-5 classic array, we'd need code such as,

int foo[3, 5][];
for (int i = 0; i < 3; i++)
    for (int j = 0; j < 5; j++)
        foo[i, j] = new int[100];

The reason why I chose this way over the other is, what happens with new. We used to have a world where [] cannot be stacked. In this world, whenever a [ follows new TYPE we initialise a dynamic array and the next integer defines the number of elements that must be created.

w = new Weapon*[100];

I wanted to keep that rule in a world where we can have [][]. For instance, consider the type, Weapon*[][]. To initialise this type we'd write new Weapon*[, and the next integer defines the number of elements to be created. So that leads to w = new Weapon[100][];.

So generalising that, this means that in an array of an array, the outermost dimension is written first.

fernewelten · 2023-11-26T14:27:51Z

You can have classic arrays where each component is a dynamic array, but you cannot have dynamic arrays where each component is a classic array.

So currently, int foo[][3, 5] is disallowed: Once you start with [], you can only tack on more [] to the right. For instance, int foo[3, 5][][][]… is allowed, to any nesting depth of the []. That would be a classic 3-by-5 array where each component is a dynamic array whose component is a dynamic array ….

That's an implementation restriction that I might be able to release.

ivan-mogilko · 2023-11-26T21:10:24Z

@fernewelten Thank you for clarification!

When we have a declaration such as int foo[3, 5][]; then there's the question in which direction it is read. Is foo a classic 3-by-5 array where each component is a dynamic array, or is foo a dynamic array where each component is a classic 3-by-5 array?
The way I've implemented it, foo is a classic 3-by-5 array where each component is a dynamic array.

This makes total sense to me indeed.

You can have classic arrays where each component is a dynamic array, but you cannot have dynamic arrays where each component is a classic array.
So currently, int foo[][3, 5] is disallowed
That's an implementation restriction that I might be able to release.

It's hard to tell how much of a priority that is. I've been trying to imagine a reason to do something like this, and the only case that comes to mind is having a dynamic array of fixed-size char arrays, to store "raw strings" of const size in them.
Strictly in terms of memory, is not dynamic array of regular arrays about same thing as dynamic array of structs? as in you have a dyn arr of a sequence of bytes in both cases.
As a workaround: after your PR you may have dynamic arrays of structs which wrap classic arrays in them.

ivan-mogilko · 2023-11-26T21:13:01Z

So, I tested your PR for all 3 new features, and they seem to work!

The script I used is under spoiler

struct NormalStruct
{
	int a, b, c;
	float f1, f2, f3;
};

void Test_DynArraysOfStructs()
{
	NormalStruct dynarr[];
	dynarr = new NormalStruct[10];
	
	for (int i = 0; i < dynarr.Length; ++i)
	{
		dynarr[i].a = 10 + i;
		dynarr[i].f1 = IntToFloat(i) * 3.33;
	}
	
	Display("NormalStruct at index %d is: %d, %f", 5, dynarr[5].a, dynarr[5].f1);
}

void Test_StaticArraysOfDynArrays()
{
	Character* arr[10][];
	
	for (int i = 0; i< 10; ++i)
	{
		arr[i] = new Character[10];
	}
	
	arr[5][6] = player;
	
	Display("Character at static arr %d, dyn arr %d = %s", 5, 6, arr[5][6].Name);
	
        // Copy a pointer to dynamic array at index 5 to another variable
	Character* dynarr[] = arr[5];
	
	Display("Character at dynamic arr %d = %s", 6, dynarr[6].Name);
}

void Test_JaggedArrays()
{
	NormalStruct jagarr[][][];
	
	jagarr = new NormalStruct[10][][];
	jagarr[2] = new NormalStruct[40][];
	jagarr[5] = new NormalStruct[60][];
	jagarr[2][22] = new NormalStruct[300];
	jagarr[5][55] = new NormalStruct[600];
	jagarr[2][22][222].f2 = 0.222;
	jagarr[5][55][555].f2 = 0.555;
	
	Display("jagarr[%d][%d][%d].f2 = %f", 2, 22, 222, jagarr[2][22][222].f2);
	Display("jagarr[%d][%d][%d].f2 = %f", 5, 55, 555, jagarr[5][55][555].f2);
}

struct StructWithArray
{
	int arr[100];
};

void Test_DynArraysOfStatArrs()
{
	//int dynarr[][5];
	//dynarr = new int[5][5];
	// !!! THIS DOES NOT WORK AT THE MOMENT !!!
	
	StructWithArray dynarr[];
	dynarr = new StructWithArray[5];
	dynarr[2].arr[50] = 10;
	
	Display("%d", dynarr[2].arr[50]);
}


function game_start()
{
	Test_DynArraysOfStructs();
	Test_StaticArraysOfDynArrays();
	Test_JaggedArrays();
	Test_DynArraysOfStatArrs();
}

This is going to be a good addition to the AGS script syntax.

ivan-mogilko · 2023-11-26T21:45:29Z

Found couple of errors.

It's possible to assign a jagged (multi-dimensional) dynamic array's pointer to a dynamic array of more dimensions, if you do this through accessing an array's element

        int arr1[][];
	arr1 = new int[10][];
        arr1[5] = new int[20];
	
	int arr2[][];
	arr2 = arr1[5]; // the 5th element is a 1-dimensional array of ints

You may then proceed accessing this second array, writing and reading elements in it. It works, but I suspect that memory contents become incorrect after this.

There's a corruption in one error message again (I remember there have been a similar problem in the past). Use following script:

        int arr[];
	arr[1][2] = 10;

The compiler prints following error message:

NewArrays.asc(76): An array index cannot follow an expression of type 'ЊЗќ'

(the garbage may vary of course)

fernewelten · 2023-11-27T18:53:18Z

int arr1[][];
arr1 = new int[10][];
arr1[5] = new int[20];
int arr2[][];
arr2 = arr1[5];

IMHO this is a type-check bug. I'm looking into it.

NewArrays.asc(76): An array index cannot follow an expression of type 'ЊЗќ'

Darn, I've forgotten a .c_str() again.

Dynamic array types must match exactly. Also: fix small bug by adding '.c_str()' to a std::string expression

fernewelten · 2023-12-04T16:32:03Z

Adding a better typecheck and fixing the 'c_str()' bug.

ivan-mogilko · 2023-12-05T08:10:45Z

I confirm these issues were fixed.

This was referenced Aug 24, 2023

AGS 4 / new compiler: support dynamic arrays of normal structs #1818

Closed

AGS 4 / New compiler: regular arrays of dynamic arrays #2080

Closed

fernewelten marked this pull request as draft August 24, 2023 11:01

fernewelten added 5 commits November 26, 2023 01:56

New compiler: Identifier renaming

782c008

Consistency, especially in the parser Also, denote unsigned literals as such

New compiler: Move SkipTo functions to internallist.h

ddb3870

Also, some var renamings

fernewelten force-pushed the NC_DynArrayOfNonManaged_NonManagedOfDyn branch from 6c00f9e to 3909560 Compare November 26, 2023 01:03

fernewelten marked this pull request as ready for review November 26, 2023 02:01

ivan-mogilko added ags 4 related to the ags4 development context: script compiler labels Nov 26, 2023

New compiler: Proper typechecking for dynamic arrays

1169261

Dynamic array types must match exactly. Also: fix small bug by adding '.c_str()' to a std::string expression

ivan-mogilko merged commit cc52b3e into adventuregamestudio:ags4 Dec 5, 2023
20 checks passed

ivan-mogilko mentioned this pull request Dec 5, 2023

AGS4/New compiler: Feature proposal: Dynamic arrays of dynamic arrays ("jagged" arrays) #1258

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic arrays of non-managed struct and classic arrays of managed struct #2106

Dynamic arrays of non-managed struct and classic arrays of managed struct #2106

fernewelten commented Aug 24, 2023 •

edited

Loading

ivan-mogilko commented Aug 24, 2023 •

edited

Loading

ivan-mogilko commented Aug 24, 2023

ivan-mogilko commented Nov 24, 2023 •

edited

Loading

fernewelten commented Nov 24, 2023

fernewelten commented Nov 26, 2023 •

edited

Loading

ivan-mogilko commented Nov 26, 2023 •

edited

Loading

ivan-mogilko commented Nov 26, 2023 •

edited

Loading

fernewelten commented Nov 26, 2023

fernewelten commented Nov 26, 2023 •

edited

Loading

fernewelten commented Nov 26, 2023 •

edited

Loading

ivan-mogilko commented Nov 26, 2023 •

edited

Loading

ivan-mogilko commented Nov 26, 2023 •

edited

Loading

ivan-mogilko commented Nov 26, 2023 •

edited

Loading

fernewelten commented Nov 27, 2023 •

edited

Loading

fernewelten commented Dec 4, 2023

ivan-mogilko commented Dec 5, 2023

Dynamic arrays of non-managed struct and classic arrays of managed struct #2106

Dynamic arrays of non-managed struct and classic arrays of managed struct #2106

Conversation

fernewelten commented Aug 24, 2023 • edited Loading

Dynamic array of non-managed struct

Classic array of managed struct

Dynamic arrays of dynamic arrays

ivan-mogilko commented Aug 24, 2023 • edited Loading

ivan-mogilko commented Aug 24, 2023

ivan-mogilko commented Nov 24, 2023 • edited Loading

fernewelten commented Nov 24, 2023

fernewelten commented Nov 26, 2023 • edited Loading

ivan-mogilko commented Nov 26, 2023 • edited Loading

ivan-mogilko commented Nov 26, 2023 • edited Loading

fernewelten commented Nov 26, 2023

fernewelten commented Nov 26, 2023 • edited Loading

fernewelten commented Nov 26, 2023 • edited Loading

ivan-mogilko commented Nov 26, 2023 • edited Loading

ivan-mogilko commented Nov 26, 2023 • edited Loading

ivan-mogilko commented Nov 26, 2023 • edited Loading

fernewelten commented Nov 27, 2023 • edited Loading

fernewelten commented Dec 4, 2023

ivan-mogilko commented Dec 5, 2023

fernewelten commented Aug 24, 2023 •

edited

Loading

ivan-mogilko commented Aug 24, 2023 •

edited

Loading

ivan-mogilko commented Nov 24, 2023 •

edited

Loading

fernewelten commented Nov 26, 2023 •

edited

Loading

ivan-mogilko commented Nov 26, 2023 •

edited

Loading

ivan-mogilko commented Nov 26, 2023 •

edited

Loading

fernewelten commented Nov 26, 2023 •

edited

Loading

fernewelten commented Nov 26, 2023 •

edited

Loading

ivan-mogilko commented Nov 26, 2023 •

edited

Loading

ivan-mogilko commented Nov 26, 2023 •

edited

Loading

ivan-mogilko commented Nov 26, 2023 •

edited

Loading

fernewelten commented Nov 27, 2023 •

edited

Loading