Merge branch 'master' into dotnet8

Docs/articles/Kernel/MemoryManagement.md

@@ -35,7 +35,16 @@ The RAT is managed through the `RAT` class. Pages are allocated via `void* RAT.A
 
 The Heap itself is managed by the `Heap` class. It contains the mechanism to allocate (`byte* Heap.Alloc(uint aSize)`), re-allocate ('byte* Heap.Realloc(byte* aPtr, uint newSize)') and free (`void Heap.Free(void* aPtr)`) objects of various sizes. Objects are seperated by size in bytes into Small (Smaller than 1/4 Page), Medium (Smaller than 1 Page) and Large (Larger than 1 Page). Currently Medium and Large objects are managed the same way using the methods in `HeapLarge` which do little more than allocating/freeing the necessary number of pages. Small objects are managed differently in `HeapSmall`. 
 
-Small Objects are managed using the SMT (Size Map Table), which is initalised using `void HeapSmall.InitSMT(uint aMaxItemSize)`. The basic idea of the SMT is to allocate objects of similar sizes on the same page. The SMT grows dynamically as required. The SMT is made up of a series of pages, each of which contains a series of `RootSMTBlock` each of which link to a chain of `SMTBlock`. The `RootSMTBlock` can be thought of as column headers and the `SMTBlock` as the elements stored in the column. The `RootSMTBlock` are a linked list, each containing the maximum object size stored in its pages, the location of the first `SMTBlock` for this size, and the location of the next `RootSMTBlock`. The list is in ascending order of size, so that the smallest large enough `RootSMTBlock` is found first. A `SMTBlock` contains a pointer to the actual page where objects are stored, how much space is left on that page, and a pointer to the next `SMTBlock`. If every `SMTBlock` for a certain size is full, a new `SMTBlock` is allocated. The page linked to by the `SMTBlock` is split into an array of spaces, each large enough to allocate an object of maximum size with header, which can be iterated through via index and fixed size when allocating. Each object allocated on the `HeapSmall` has a header of 2 `ushort`, the first one storing the actual size of the object and the second, the GC status of the object.
+Small Objects are managed using the SMT (Size Map Table), which is initalised using `void HeapSmall.InitSMT(uint aMaxItemSize)`.
+The basic idea of the SMT is to allocate objects of similar sizes on the same page. The SMT grows dynamically as required.
+The SMT is made up of a series of pages, each of which contains a series of `RootSMTBlock` each of which link to a chain of `SMTBlock`.
+The `RootSMTBlock` can be thought of as column headers and the `SMTBlock` as the elements stored in the column.
+The `RootSMTBlock` are a linked list, each containing the maximum object size stored in its pages, the location of the first `SMTBlock` for this size, and the location of the next `RootSMTBlock`.
+The list is in ascending order of size, so that the smallest large enough `RootSMTBlock` is found first.
+A `SMTBlock` contains a pointer to the actual page where objects are stored, how much space is left on that page, and a pointer to the next `SMTBlock`.
+If every `SMTBlock` for a certain size is full, a new `SMTBlock` is allocated.
+The page linked to by the `SMTBlock` is split into an array of spaces, each large enough to allocate an object of maximum size with header, which can be iterated through via index and fixed size when allocating.
+Each object allocated on the `HeapSmall` has a header of 2 `ushort`, the first one storing the actual size of the object and the second, the GC status of the object.
 
 ## Garbage Collection
 
@@ -45,6 +54,10 @@ The garbage collector has to be manually triggerd using the call `int Heap.Colle
 
 Note that the GC does not track objects only pointed to by pointers. To ensure that the GC nevertheless does not incorrectly free objects, you can use `void GCImplementation.IncRootCount(ushort* aPtr)` to manually increase the references of your object by 1. Once you no longer need the object you can use `void GCImplementation.DecRootCount(ushort* aPtr)` to remove the manual reference, which allows the next `Heap.Collect` call to free the object. 
 
+`Heap.Collect` only cleans up the objects which are no longer used but will leave behind empty pages in the SMT.
+These pages can be cleaned up using `HeapSmall.PruneSMT` which will return the number of pages it freed.
+Note that if in future elements are reallocated, this will cause new pages in the SMT to be allocated again, so using this too often may not be useful.
+
 ## Automatically Trigger Garbage Collection
 
 When `RAT.MinFreePages` is set to a positive value and the number of free pages (as tracked by `RAT.FreePageCount`) drops below this value, on page allocation `Heap.Collect` will automatically be called. Each time this happens the value in `RAT.GCTriggered` is incremented by one.

Tests/Kernels/Cosmos.Compiler.Tests.TypeSystem/Kernel.cs

@@ -140,6 +140,18 @@ private unsafe void TestGarbageCollector()
             StaticTestClass.B.FieldA = 10;
             collected = Heap.Collect();
             Assert.AreEqual(0, collected, "Storing elements in static class keeps them referenced");
+
+            for (int i = 0; i < 10_000; i++)
+            {
+                _ = new object();
+            }
+            Heap.Collect();
+            uint heapSmallPages = RAT.GetPageCount((byte)RAT.PageType.HeapSmall);
+            int freed = HeapSmall.PruneSMT();
+            uint afterPrune = RAT.GetPageCount((byte)RAT.PageType.HeapSmall);
+            Assert.IsTrue(heapSmallPages >= afterPrune, "Running PruneSMT does not increase the number of pages in use");
+            Assert.AreEqual(freed, heapSmallPages - afterPrune, "PruneSMT returns the correct number of pages freed");
+
         }
 
         #region Test Methods

source/Cosmos.Build.Builder/packages.lock.json

@@ -0,0 +1,63 @@
+{
+  "version": 1,
+  "dependencies": {
+    ".NETFramework,Version=v4.7.2": {
+      "Microsoft.VisualStudio.Setup.Configuration.Interop": {
+        "type": "Direct",
+        "requested": "[1.16.30, )",
+        "resolved": "1.16.30",
+        "contentHash": "lC6SqNkraWUSY7cyF5GUmXSECoTMwslBc/r1dguChjsi0D0BlF7G6PLsvXD0NFCwnpKlgVzUYrIq7DQakdGerw=="
+      },
+      "Newtonsoft.Json": {
+        "type": "Direct",
+        "requested": "[13.0.1, )",
+        "resolved": "13.0.1",
+        "contentHash": "ppPFpBcvxdsfUonNcvITKqLl3bqxWbDCZIzDWHzjpdAHRFfZe0Dw9HmA0+za13IdyrgJwpkDTDA9fHaxOrt20A=="
+      },
+      "NuGet.Common": {
+        "type": "Direct",
+        "requested": "[5.9.1, )",
+        "resolved": "5.9.1",
+        "contentHash": "BRX0V8k8QXcEWL33V2HcBU1+eu/Qp5LDJlheYFSZ03hKjlHVHFfLquvUUYLgSNO7tiBxCFe7pTSgO4XanyzteQ==",
+        "dependencies": {
+          "NuGet.Frameworks": "5.9.1"
+        }
+      },
+      "NuGet.Configuration": {
+        "type": "Direct",
+        "requested": "[5.9.1, )",
+        "resolved": "5.9.1",
+        "contentHash": "dpjLDYEuhR1J8L3s9c1qVHEBaWqFI2/x3qRmhqVYDLB5B9ETVe1jVkxLkFQreQVgh+N5cVgtZWx1jGuArj7QaQ==",
+        "dependencies": {
+          "NuGet.Common": "5.9.1"
+        }
+      },
+      "System.Runtime.WindowsRuntime": {
+        "type": "Direct",
+        "requested": "[4.6.0, )",
+        "resolved": "4.6.0",
+        "contentHash": "IWrs1TmbxP65ZZjIglNyvDkFNoV5q2Pofg5WO7I8RKQOpLdFprQSh3xesOoClBqR4JHr4nEB1Xk1MqLPW1jPuQ=="
+      },
+      "WPF-UI": {
+        "type": "Direct",
+        "requested": "[2.0.3, )",
+        "resolved": "2.0.3",
+        "contentHash": "1e9Q8pQGfi9YFp3WavKZjXyLYohIjXxk5q0NqEaamofVgIdxRxwqrkWha584FBUISRBBT/qFOrl97YBl5bs30Q==",
+        "dependencies": {
+          "System.Drawing.Common": "6.0.0"
+        }
+      },
+      "NuGet.Frameworks": {
+        "type": "Transitive",
+        "resolved": "5.9.1",
+        "contentHash": "LuJA875MQpPMdik6KUsDUnEDSXWX+T/sExFikA0A5zGFkEW37weP5b6NxljWlrw4UNOWTgmTeumm802Jwz20sw=="
+      },
+      "System.Drawing.Common": {
+        "type": "Transitive",
+        "resolved": "6.0.0",
+        "contentHash": "NfuoKUiP2nUWwKZN6twGqXioIe1zVD0RIj2t976A+czLHr2nY454RwwXs6JU9Htc6mwqL6Dn/nEL3dpVf2jOhg=="
+      }
+    },
+    ".NETFramework,Version=v4.7.2/win7-x86": {}
+  }
+}

source/Cosmos.Core/Memory/HeapSmall.cs

@@ -546,5 +546,81 @@ private static int GetAllocatedObjectCount(SMTPage* aPage, uint aSize)
         }
 
         #endregion
+
+        #region Cleanup
+
+        /// <summary>
+        /// This function will free all pages allocated for small objects which are emnpty
+        /// </summary>
+        /// <returns>Number of pages freed</returns>
+        public static int PruneSMT()
+        {
+            int freed = 0;
+            SMTPage* page = SMT;
+            while (page != null)
+            {
+                freed += PruneSMT(page);
+                page = page->Next;
+            }
+            return freed;
+        }
+
+        /// <summary>
+        /// Prune all empty pages allocated on a certain page
+        /// </summary>
+        /// <param name="aPage"></param>
+        /// <returns></returns>
+        private static int PruneSMT(SMTPage* aPage)
+        {
+            int freed = 0;
+            RootSMTBlock* ptr = (RootSMTBlock*)aPage->First; // since both RootSMTBlock and SMTBlock have the same size (20) it doesnt matter if cast is wrong
+            while(ptr != null)
+            {
+                freed += PruneSMT(ptr, ptr->Size);
+                ptr = ptr->LargerSize;
+            }
+            return freed;
+        }
+
+        /// <summary>
+        /// Prune all empty pages which are linked to root block for a certain size
+        /// The root block or first one following it will not be removed!
+        /// </summary>
+        /// <param name="aBlock"></param>
+        /// <param name="aSize"></param>
+        /// <returns></returns>
+        private static int PruneSMT(RootSMTBlock* aBlock, uint aSize)
+        {
+            int freed = 0;
+            int maxElements = (int)(RAT.PageSize / (aSize + PrefixItemBytes));
+            SMTBlock* prev = aBlock->First;
+            SMTBlock* block = prev->NextBlock;
+            while(block != null)
+            {
+                if (block->SpacesLeft == maxElements)
+                {
+                    // This block is currently empty so free it
+                    prev->NextBlock = block->NextBlock;
+                    RAT.Free(block->PagePtr);
+
+                    uint* toCleanUp = (uint*) block;
+                    block = prev->NextBlock;
+
+                    toCleanUp[0] = 0;
+                    toCleanUp[1] = 0;
+                    toCleanUp[2] = 0;
+
+                    freed++;
+                }
+                else
+                {
+                    prev = block;
+                    block = block->NextBlock;
+                }
+            }
+            return freed;
+        }
+
+        #endregion
     }
 }

source/Cosmos.Core/Memory/RAT.cs

@@ -297,6 +297,11 @@ public static uint GetFirstRATIndex(void* aPtr)
             throw new Exception("Page type not found. Likely RAT is rotten.");
         }
 
+        /// <summary>
+        /// Get the pointer to the start of the page containing the pointer's address
+        /// </summary>
+        /// <param name="aPtr"></param>
+        /// <returns></returns>
         public static byte* GetPagePtr(void* aPtr)
         {
             return (byte*)aPtr - ((byte*)aPtr - RamStart) % PageSize;
@@ -326,7 +331,7 @@ public static void Free(uint aPageIdx)
             byte* p = mRAT + aPageIdx;
             *p = (byte)PageType.Empty;
             FreePageCount++;
-            for (; p < mRAT + TotalPageCount; )
+            for (; p < mRAT + TotalPageCount;)
             {
                 if (*++p != (byte)PageType.Extension)
                 {
@@ -336,5 +341,14 @@ public static void Free(uint aPageIdx)
                 FreePageCount++;
             }
         }
+
+        /// <summary>
+        /// Free the page this pointer points to
+        /// </summary>
+        /// <param name="aPtr"></param>
+        public static void Free(void* aPtr)
+        {
+            Free(GetFirstRATIndex(aPtr));
+        }
     }
 }

source/Cosmos.Deploy.Pixie/Cosmos.Deploy.Pixie.csproj

@@ -1,9 +1,9 @@
 <Project Sdk="Microsoft.NET.Sdk">
 
     <PropertyGroup>
-        <TargetFramework>net8.0-windows</TargetFramework>
-        <RuntimeIdentifier>win-x86</RuntimeIdentifier>
 
+        <TargetFramework>net8.0-windows</TargetFramework>
+        <RuntimeIdentifier>win-x64</RuntimeIdentifier>
         <OutputType>WinExe</OutputType>
         <ApplicationIcon>Cosmos.ico</ApplicationIcon>
         <UseWPF>true</UseWPF>

source/Cosmos.Deploy.Pixie/packages.lock.json

@@ -2,6 +2,6 @@
   "version": 1,
   "dependencies": {
     "net6.0-windows7.0": {},
-    "net6.0-windows7.0/win-x86": {}
+    "net6.0-windows7.0/win-x64": {}
   }
 }

source/Cosmos.Deploy.USB/Cosmos.Deploy.USB.csproj

@@ -1,8 +1,9 @@
 <Project Sdk="Microsoft.NET.Sdk">
 
     <PropertyGroup>
-        <TargetFramework>net8.0-windows</TargetFramework>
-        <RuntimeIdentifier>win-x86</RuntimeIdentifier>
+
+        <TargetFramework>net6.0-windows</TargetFramework>
+        <RuntimeIdentifier>win-x64</RuntimeIdentifier>
 
         <OutputType>WinExe</OutputType>
         <ApplicationIcon>Cosmos.ico</ApplicationIcon>

source/Cosmos.Deploy.USB/packages.lock.json

@@ -37,7 +37,7 @@
         }
       }
     },
-    "net6.0-windows7.0/win-x86": {
+    "net6.0-windows7.0/win-x64": {
       "Microsoft.Win32.Registry": {
         "type": "Transitive",
         "resolved": "5.0.0",

source/Cosmos.VS.DebugEngine/Cosmos.VS.DebugEngine.csproj

@@ -2,7 +2,7 @@
 
     <PropertyGroup>
         <TargetFramework>net48</TargetFramework>
-        <RuntimeIdentifier>win-x86</RuntimeIdentifier>
+        <RuntimeIdentifier>win-x64</RuntimeIdentifier>
     </PropertyGroup>
 
     <PropertyGroup>

source/Cosmos.VS.DebugEngine/packages.lock.json

@@ -1330,7 +1330,7 @@
         }
       }
     },
-    ".NETFramework,Version=v4.8/win-x86": {
+    ".NETFramework,Version=v4.8/win-x64": {
       "Microsoft.Win32.Registry": {
         "type": "Direct",
         "requested": "[5.0.0, )",

source/Cosmos.VS.ProjectSystem/Cosmos.VS.ProjectSystem.csproj

@@ -2,7 +2,7 @@
 
     <PropertyGroup>
         <TargetFramework>net48</TargetFramework>
-        <RuntimeIdentifier>win-x86</RuntimeIdentifier>
+        <RuntimeIdentifier>win-x64</RuntimeIdentifier>
         <RootNamespace>Cosmos.VS.ProjectSystem</RootNamespace>
         <FileVersion>1.0.0.0</FileVersion>
         <AssemblyVersion>1.0.0.0</AssemblyVersion>

source/Cosmos.VS.ProjectSystem/packages.lock.json

@@ -1347,7 +1347,7 @@
         }
       }
     },
-    ".NETFramework,Version=v4.8/win-x86": {
+    ".NETFramework,Version=v4.8/win-x64": {
       "Microsoft.Win32.Registry": {
         "type": "Direct",
         "requested": "[5.0.0, )",

source/Cosmos.VS.Windows/Cosmos.VS.Windows.csproj

@@ -2,7 +2,7 @@
 
     <PropertyGroup>
         <TargetFramework>net48</TargetFramework>
-        <RuntimeIdentifier>win-x86</RuntimeIdentifier>
+        <RuntimeIdentifier>win-x64</RuntimeIdentifier>
         <RootNamespace>Cosmos.VS.Windows</RootNamespace>
         <CreateVsixContainer>False</CreateVsixContainer>
         <DeployExtension>False</DeployExtension>

source/Cosmos.VS.Windows/packages.lock.json

@@ -1136,7 +1136,7 @@
         }
       }
     },
-    ".NETFramework,Version=v4.8/win-x86": {
+    ".NETFramework,Version=v4.8/win-x64": {
       "Microsoft.Win32.Primitives": {
         "type": "Transitive",
         "resolved": "4.3.0",