doc: migration expressions

doc/md/canister-maintenance/compatibility.md

@@ -59,7 +59,7 @@ For example, `v1`'s stable types:
 ``` motoko no-repl file=../examples/count-v1.most
 ```
 
-An upgrade from `v1` to `v2`'s stable types consumes a [`Nat`](../base/Int.md) as an [`Int`](../base/Nat.md), which is valid because `Int <: Nat`.
+An upgrade from `v1` to `v2`'s stable types consumes a [`Nat`](../base/Int.md) as an [`Int`](../base/Nat.md), which is valid because `Nat <: Int`, that is,  `Nat` is a subtype of `Int`.
 
 ``` motoko no-repl file=../examples/count-v2.most
 ```
@@ -113,30 +113,40 @@ Version `v3` with Candid interface `v3.did` and stable type interface `v3.most`:
 
 ## Incompatible upgrade
 
-Let's take a look at another example where the counter's type is again changed, this time from [`Int`](../base/Int.md) to [`Nat`](../base/Float.md):
+Let's take a look at another example where the counter's type is again changed, this time from [`Int`](../base/Int.md) to [`Float`](../base/Float.md):
 
 ``` motoko no-repl file=../examples/count-v4.mo
 ```
 
-This version is neither compatible to the Candid interface nor to the stable type declarations.
-- Since `Float </: Int`, the new type of `state` is not compatible to the old type.
-- The return type change of `read` is also not valid.
+This version is neither compatible to stable type declarations, nor to the Candid interface.
+- Since `Int </: Float`, that is, `Int` is not a subtype of `Float`, the old type of `state`, `Int`, is not compatible with the new type, `Float`.
+  This means that the old value of `state`, an integer, cannot be used to initialize the new `state` field that now requires a float.
+- The change in the return type of `read` is also not safe.
+  If the change were accepted, then existing clients of the `read` method, that still expect to receive integers, would suddenly start receiving incompatible floats.
+
+With [enhanced orthogonal persistence](orthogonal-persistence/enhanced.md), Motoko actively rejects any upgrades that require type-incompatible state changes.
 
-Motoko rejects upgrades with incompatible state changes with [enhanced orthogonal persistence](orthogonal-persistence/enhanced.md).
 This is to guarantee that the stable state is always kept safe.
 
 ```
 Error from Canister ...: Canister called `ic0.trap` with message: RTS error: Memory-incompatible program upgrade.
 ```
 
-In addition to Motoko's check, `dfx` raises a warning message for these incompatible changes, including the breaking Candid change.
+In addition to Motoko's runtime check, `dfx` raises a warning message for these incompatible changes, including the breaking Candid change.
+
+Motoko tolerates Candid interface changes, since these are more likely to be intentional, breaking changes.
 
 :::danger
 Versions of Motoko using [classical orthogonal persistence](orthogonal-persistence/classical.md) will drop the state and reinitialize the counter with `0.0`, if the `dfx` warning is ignored.
+
+For this reason, users should always heed any compatibility warnings issued by `dfx`.
 :::
 
+
+
 ## Explicit migration
 
+### Explicit migration using several upgrades
 There is always a migration path to change structure of stable state, even if a direct type change is not compatible.
 
 For this purpose, a user-instructed migration can be done in three steps:
@@ -158,6 +168,72 @@ For this purpose, a user-instructed migration can be done in three steps:
 
 Alternatively, the type of `state` can be changed to `Any`, also implying that this variable is no longer used.
 
+### Explicit migration using a migration function
+
+The previous approach of using several upgrades to migrate data is both tedious and
+obscure, mingling production with migration code.
+
+To ease data migration, Motoko now supports explicit migration using a separate data migration function.
+The code for the migration function is self-contained and can be placed in its own file.
+
+The migration function takes a record of stable fields as input and produces a record of stable fields as output.
+
+The input fields extend or override the types of any stable fields in the actor's
+stable signature.
+The output fields must be declared in the actor's stable signature, and have types that can be consumed by the corresponding declaration in the stable signature.
+
+* All values for the input fields must
+be present and of compatible type in the old actor, otherwise the
+upgrade traps and rolls back.
+* The fields output by the migration
+function determine the values of the corresponding stable variables in the
+new actor.
+* All other stable variables of the actor, i.e. those neither consumed nor
+produced by the migration function are initialized in the usual way,
+either by transfer from the upgraded actor, if declared in that actor, or, if newly declared,
+by running the initialization expression in the field's declaration.
+* The migration function is only executed on an upgrade and ignored on a fresh installation of the actor in an empty canister.
+
+The migration function, when required, is declared within square brackets following the `actor` keyword in an actor or actor class
+declaration, for example:
+
+``` motoko no-repl file=../examples/count-v7.mo
+```
+
+Employing a migration function offers another advantage: it lets you re-use the name of an
+existing field, even when its type has changed:
+
+``` motoko no-repl file=../examples/count-v8.mo
+```
+
+Here, we've put the migration code in a separate library:
+
+``` motoko no-repl file=../examples/Migration.mo
+```
+
+The migration function can be selective and only consume or produce a subset of the old and new stable variables. Other stable variables can be declared as usual.
+
+For example, here, with the same migration function, we also declare a new stable variable, `lastModified` that records the time of the last update,
+without having to mention that field in the migration function:
+
+``` motoko no-repl file=../examples/count-v9.mo
+```
+
+The stable signature of an actor with a migration function now consists of two ordinary stable signatures, the pre-signature (before the upgrade), and the post-signature (after the upgrade).
+
+
+For example, this is the combined signature of the previous example:
+
+``` motoko no-repl file=../examples/count-v9.most
+```
+
+The second signature is determined solely by the actor's stable variable declarations.
+The first signature contains the field declarations from the migration function's input, together with any distinctly named stable variables declared in the actor.
+
+For compatibility, when performing an upgrade, the (post) signature of the old code must be compatible with the (pre) signature of the new code.
+
+The migration function can be deleted or adjusted on the next upgrade.
+
 ## Upgrade tooling
 
 `dfx` incorporates an upgrade check. For this purpose, it uses the Motoko compiler (`moc`) that supports:
@@ -197,25 +273,12 @@ A common, real-world example of an incompatible upgrade can be found [on the for
 
 In that example, a user was attempting to add a field to the record payload of an array, by upgrading from stable type interface:
 
-``` motoko no-repl
-persistent actor {
-  type Card = {
-    title : Text;
-  };
-  var map : [(Nat32, Card)] = [(0, { title = "TEST"})];
-};
+``` motoko no-repl file=../examples/Card-v0.mo
 ```
 
 to *incompatible* stable type interface:
 
-``` motoko no-repl
-persistent actor {
-  type Card = {
-    title : Text;
-    description : Text;
-  };
-  var map : [(Nat32, Card)] = [];
-};
+``` motoko no-repl file=../examples/Card-v1.mo
 ```
 
 ### Problem
@@ -233,52 +296,32 @@ cannot be consumed at new type
 
 Do you want to proceed? yes/No
 ```
-It is recommended not to continue, as you will lose the state in older versions of Motoko that use [classical orthogonal persistence](orthogonal-persistence/classical.md). 
+It is recommended not to continue, as you will lose the state in older versions of Motoko that use [classical orthogonal persistence](orthogonal-persistence/classical.md).
 Upgrading with [enhanced orthogonal persistence](orthogonal-persistence/enhanced.md) will trap and roll back, keeping the old state.
 
 Adding a new record field to the type of existing stable variable is not supported. The reason is simple: The upgrade would need to supply values for the new field out of thin air. In this example, the upgrade would need to conjure up some value for the `description` field of every existing `card` in `map`. Moreover, allowing adding optional fields is also a problem, as a record can be shared from various variables with different static types, some of them already declaring the added field or adding a same-named optional field with a potentially different type (and/or different semantics).
 
-### Solution
+To resolve this issue, some form of  [explicit data migration](#explicit-migration) is needed.
+
+We present two solutions, the first using a sequence of simple upgrades, and a second, recommended solution, that uses a single upgrade with a migration function.
 
-To resolve this issue, an [explicit](#explicit-migration) is needed:
+### Solution 1 using two plain upgrades
 
 1. You must keep the old variable `map` with the same structural type. However, you are allowed to change type alias name (`Card` to `OldCard`).
 2. You can introduce a new variable `newMap` and copy the old state to the new one, initializing the new field as needed.
 3. Then, upgrade to this new version.
 
-``` motoko no-repl
-import Array "mo:base/Array";
-
-persistent actor {
-  type OldCard = {
-    title : Text;
-  };
-  type NewCard = {
-    title : Text;
-    description : Text;
-  };
-
-  var map : [(Nat32, OldCard)] = [];
-  var newMap : [(Nat32, NewCard)] = Array.map<(Nat32, OldCard), (Nat32, NewCard)>(
-    map,
-    func(key, { title }) { (key, { title; description = "<empty>" }) },
-  );
-};
+``` motoko no-repl file=../examples/Card-v1a.mo
 ```
 
 4. **After** we have successfully upgraded to this new version, we can upgrade once more to a version, that drops the old `map`.
 
-``` motoko no-repl
-persistent actor {
-  type Card = {
-    title : Text;
-    description : Text;
-  };
-  var newMap : [(Nat32, Card)] = [];
-};
+
+``` motoko no-repl file=../examples/Card-v1b.mo
 ```
 
-`dfx` will issue a warning that `map` will be dropped. 
+
+`dfx` will issue a warning that `map` will be dropped.
 
 Make sure, you have previously migrated the old state to `newMap` before applying this final reduced version.
 
@@ -289,4 +332,30 @@ Stable interface compatibility check issued a WARNING for canister ...
  will be discarded. This may cause data loss. Are you sure?
 ```
 
+### Solution 2 using a migration function and single upgrade
+
+Instead of the previous two step solution, we can upgrade in one step using a migration function.
+
+1. Define a migration module and function that transforms the old stable variable, at its current type, into the new stable variable at its new type.
+
+
+``` motoko no-repl file=../examples/CardMigration.mo
+```
+
+2. Specify the migration function as the migration expression of your actor declaration:
+
+
+``` motoko no-repl file=../examples/Card-v1c.mo
+```
+
+**After** we have successfully upgraded to this new version, we can also upgrade once more to a version that drops the migration code.
+
+
+``` motoko no-repl file=../examples/Card-v1d.mo
+```
+
+However, removing or adjusting the migration code can also be delayed to the next, proper upgrade that fixes bugs or extends functionality.
+
+Note that with this solution, there is no need to rename `map` to `newMap` and the migration code is nicely isolated from the main code.
+
 <img src="https://github.com/user-attachments/assets/844ca364-4d71-42b3-aaec-4a6c3509ee2e" alt="Logo" width="150" height="150" />

doc/md/canister-maintenance/upgrades.md

@@ -184,7 +184,7 @@ This is automatically supported when the new program version is stable-compatibl
 
 More precisely, the following changes can be implicitly migrated:
 * Adding or removing actor fields.
-* Changing mutability of the actor field.
+* Changing the mutability of an actor field.
 * Removing record fields.
 * Adding variant fields.
 * Changing `Nat` to `Int`.
@@ -193,14 +193,23 @@ More precisely, the following changes can be implicitly migrated:
 
 ### Explicit migration
 
-Any more complex migration is possible by user-defined functionality.
+More complex migration patterns that require non-trivial data transformations are possible, but require additional coding and care by the user.
+
+One way to replace some stable variables by a new set with different types is to use a sequence of upgrades to transform the state as desired:
 
 For this purpose, a three step approach is taken:
 1. Introduce new variables of the desired types, while keeping the old declarations.
 2. Write logic to copy the state from the old variables to the new variables on upgrade.
 3. Drop the old declarations once all data has been migrated.
 
-For more information, see the [example of explicit migration](compatibility.md#explicit-migration).
+A cleaner, more maintainable solution is to declare an explicit migration expression that is used
+to transform a subset of existing stable variables into a subset of replacement stable variables.
+
+Both of these data migration paths are supported by static and dynamic checks that prevent data loss or corruption.
+Of course, a user may still lose data due to coding errors, so should tread carefully.
+
+For more information, see the [example of explicit migration](compatibility.md#explicit-migration) and the
+reference material on [migration expressions](../reference/language-manual#migration-expressions).
 
 ## Legacy features
 

doc/md/examples/Card-v0.did

@@ -0,0 +1,2 @@
+service : {
+}

doc/md/examples/Card-v0.mo

@@ -0,0 +1,6 @@
+persistent actor {
+  type Card = {
+    title : Text;
+  };
+  var map : [(Nat32, Card)] = [];
+};

doc/md/examples/Card-v0.most

@@ -0,0 +1,5 @@
+// Version: 1.0.0
+type Card = {title : Text};
+actor {
+  stable var map : [(Nat32, Card)]
+};

doc/md/examples/Card-v1.did

@@ -0,0 +1,2 @@
+service : {
+}

doc/md/examples/Card-v1.mo

@@ -0,0 +1,7 @@
+persistent actor {
+  type Card = {
+    title : Text;
+    description : Text;
+  };
+  var map : [(Nat32, Card)] = [];
+};

doc/md/examples/Card-v1.most

@@ -0,0 +1,5 @@
+// Version: 1.0.0
+type Card = {description : Text; title : Text};
+actor {
+  stable var map : [(Nat32, Card)]
+};

doc/md/examples/Card-v1a.did

@@ -0,0 +1,2 @@
+service : {
+}

doc/md/examples/Card-v1a.mo

@@ -0,0 +1,17 @@
+import Array "mo:base/Array";
+
+persistent actor {
+  type OldCard = {
+    title : Text;
+  };
+  type NewCard = {
+    title : Text;
+    description : Text;
+  };
+
+  var map : [(Nat32, OldCard)] = [];
+  var newMap : [(Nat32, NewCard)] = Array.map<(Nat32, OldCard), (Nat32, NewCard)>(
+    map,
+    func(key, { title }) { (key, { title; description = "<empty>" }) },
+  );
+};

doc/md/examples/Card-v1a.most

@@ -0,0 +1,7 @@
+// Version: 1.0.0
+type NewCard = {description : Text; title : Text};
+type OldCard = {title : Text};
+actor {
+  stable var map : [(Nat32, OldCard)];
+  stable var newMap : [(Nat32, NewCard)]
+};

doc/md/examples/Card-v1b.did

@@ -0,0 +1,2 @@
+service : {
+}

doc/md/examples/Card-v1b.mo

@@ -0,0 +1,8 @@
+persistent actor {
+  type Card = {
+    title : Text;
+    description : Text;
+  };
+  var newMap : [(Nat32, Card)] = [];
+};
+

doc/md/examples/Card-v1b.most

@@ -0,0 +1,5 @@
+// Version: 1.0.0
+type Card = {description : Text; title : Text};
+actor {
+  stable var newMap : [(Nat32, Card)]
+};

doc/md/examples/Card-v1c.did

@@ -0,0 +1,2 @@
+service : {
+}

doc/md/examples/Card-v1c.mo

@@ -0,0 +1,13 @@
+import CardMigration "CardMigration";
+
+persistent actor
+  [ CardMigration.migrate ] // Declare the migration function
+  {
+  type Card = {
+    title : Text;
+    description : Text;
+  };
+
+  var map : [(Nat32, Card)] = []; // Initialized by migration on upgrade
+
+};

doc/md/examples/Card-v1c.most

@@ -0,0 +1,8 @@
+// Version: 2.0.0
+type Card = {description : Text; title : Text};
+type OldCard = {title : Text};
+actor ({
+  stable var map : [(Nat32, OldCard)]
+}, {
+  stable var map : [(Nat32, Card)]
+}) ;

doc/md/examples/Card-v1d.did

@@ -0,0 +1,2 @@
+service : {
+}