onlyoffice rework

Documentation/docs/plugins.md

@@ -43,7 +43,7 @@ Links in top right drop down application grid can be configured in the `applicat
 
 Plugins are defined as entries under the `applications.plugins` array, with at least the following properties:
 
-- `id` is mandatory and must be unique to each plugin
+- `id` is mandatory and must be unique to each plugin. It must be rather short, and alphanumeric with `_`s only.
 - `internal_domain` is the internal domain of the plugin's server, it must be the same as the one in the docker-compose.yml file for ex. or resolvable and accessible to the Twake Drive backend server.
 - `external_prefix` is the external URL prefix of the plugin exposed by the backend and proxied to the `internal_domain`.
 - `api.private_key` is the shared secret used by the plugin's server to authentify to the backend
@@ -133,6 +133,119 @@ When a user requests a preview and then possibly to edit the file, an IFrame is
 
 In the case of the OnlyOffice application, these URLs are pointing to the connector plugin, which then proxies back and forth with the OnlyOffice document server.
 
+#### Editing session key
+
+When editing a file, at first, a new session key is created on the file by the
+backend and atomically swapped.
+
+This session key uniquely identifies the user, company, and plugin and it is not encrypted.
+
+An editing session key prevents others from starting, can be updated with new versions, and
+ended with and without a new version.
+
+See operations `beginEditing` and `updateEditing` in `tdrive/backend/node/src/services/documents/services/index.ts` for operations available on that key.
+
+Basic flow is:
+  - Call `beginEditing`
+    - If a key previously was set for that file, uses the `/check` endpoint of the plugin to update the key status first
+    - Twake Drive generates an editing session key and returns it (or returns the existing one).
+  - Optionally call `updateEditing?keepEditing=true` with a file stream to generate intermediary versions
+  - End the session by calling `updateEditing` (without `keepEditing=true`). If a file is provided, a new (and final) FileVersion
+    is created with that content. Otherwise, the editing session is cleared without creating a new version.
+
+
+#### API to expose by the application
+
+Authentication from the Twake Drive backend is a JWT with the property `type` being
+`"tdriveToApplication"` and signed with the shared secret. The application must accept
+multiple parallel requests for the same key gracefully.
+
+- `POST /tdriveApi/1/session/${editing_session_key}/check`
+
+    Sent by Twake Drive backend when beginning a new editing session or investigating
+    stored keys. The application is expected to process the key if possible before responding,
+    and provide a response in JSON. The `error` key of that body should be truthy if the response
+    is not known. Otherwise it should respond:
+
+  - `{ status: 'unknown' }`: the key isn't known and maybe used for a new session
+  - `{ status: 'updated' }`: the key needed updating but is now invalid
+  - `{ status: 'expired' }`: the key was already used in a finished session and can't be used again
+  - `{ status: 'live' }`:    the key is valid and current and should be used again for the same file (if multiple writers are allowed)
+
+#### Example flow of editing session
+
+```mermaid
+sequenceDiagram
+  autonumber
+  actor User as User/Browser
+  participant Front as Frontend
+  box Server side
+    participant Back as Backend
+    participant Conn as onlyoffice<br>connector
+    participant OO as Only Office<br>Edition Server
+  end
+
+  User->>Front: Open in<br>editor
+  Front->>Back: Get plugin config
+  Front->>User: Open plugin config<br>editor.edition_url
+
+  User->>Conn: Open editor_url<br>(proxied by backend)
+  Conn->>Back: beginEditing
+  alt existing key
+  Back->>Conn: checkSessionStatus
+  Conn->>OO: getForgotten
+  note over Conn, OO: recover forgotten<br>process, and new key
+  Conn->>OO: info command
+  note right of Conn: decide status of key<br>live or stale
+  note over Conn, OO: detect ended but<br>not changed keys
+  note over Conn, OO: normal callback processing with<br>update if required
+  OO->>Conn: callback with key status
+  Conn->>Back: key status<br>(live/expired/updated/etc)
+  end
+  activate Back
+  Back->>Conn: editing_session_key
+  Conn->>User: HTML host for Editor with<br>special callback URL
+  User->>OO: Load JS Editor directly from OO server
+  activate User
+  loop User editing
+  User->>User: Furious Document<br>Editing
+  User-->>OO: Periodic saving
+  end
+  deactivate User
+  note left of User: Closes editor
+  note over User,OO: 10 seconds after last user closes their editor
+  OO->>Conn: callback to save the new version<br>or close without changes
+  Conn->>Back: updateEditing?keepEditing=false<br>with URL to new version from OO
+  deactivate Back
+```
+
+#### Batch processing of unknown keys
+
+Periodically, the plugin, and twake drive, should run batch cleanup operations on editing session keys
+to ensure they are live, or removed, as they may block other operations until then.
+
+Here is an example initiated by the plugin:
+
+```mermaid
+sequenceDiagram
+  autonumber
+  actor User as User/Browser
+  participant Front as Frontend
+  box Server side
+    participant Back as Backend
+    participant Conn as onlyoffice<br>connector
+    participant OO as Only Office<br>Edition Server
+  end
+
+  alt Periodic scheduled task
+  Conn->>OO: getForgottenList
+    loop Each forgotten file
+      Conn->>Back: Save new version<br>end editing session
+      Conn->>OO: Delete forgotten file
+    end
+  end
+```
+
 ### Example: OnlyOffice plugin
 
 The [OnlyOffice connector plugin](https://github.com/linagora/twake-drive/tree/main/tdrive/connectors/onlyoffice-connector) is an example of plugin. It's readme includes an example configuration for the backend.

tdrive/backend/node/src/cli/cmds/editing_session.ts

@@ -0,0 +1,16 @@
+import { CommandModule } from "yargs";
+
+const command: CommandModule = {
+  describe: "Editing sessions tools",
+  command: "editing_session",
+  builder: yargs =>
+    yargs.commandDir("editing_session_cmds", {
+      visit: commandModule => commandModule.default,
+    }),
+  // eslint-disable-next-line @typescript-eslint/no-empty-function
+  handler: () => {
+    throw new Error("Missing sub-command");
+  },
+};
+
+export default command;

tdrive/backend/node/src/cli/cmds/editing_session_cmds/list.ts

@@ -0,0 +1,166 @@
+import yargs from "yargs";
+
+import runWithPlatform from "../../lib/run-with-platform";
+import type { TdrivePlatform } from "../../../core/platform/platform";
+import type { DatabaseServiceAPI } from "../../../core/platform/services/database/api";
+import {
+  DriveFile,
+  EditingSessionKeyFormat,
+  TYPE as DriveFile_TYPE,
+} from "../../../services/documents/entities/drive-file";
+import {
+  FileVersion,
+  TYPE as FileVersion_TYPE,
+} from "../../../services/documents/entities/file-version";
+import User, { TYPE as User_TYPE } from "../../../services/user/entities/user";
+import { FindOptions } from "../../../core/platform/services/database/services/orm/repository/repository";
+
+async function makeUserCache(platform: TdrivePlatform) {
+  const usersRepo = await platform
+    .getProvider<DatabaseServiceAPI>("database")
+    .getRepository<User>(User_TYPE, User);
+  const cache: { [id: string]: User } = {};
+  return async (id): Promise<[boolean, User]> => {
+    if (id in cache) return [true, cache[id]];
+    const user = await usersRepo.findOne({ id });
+    if (user) cache[id] = user;
+    return [false, user];
+  };
+}
+
+interface ListArguments {
+  all: boolean;
+  name: string;
+}
+
+async function report(platform: TdrivePlatform, args: ListArguments) {
+  const users = await makeUserCache(platform);
+  async function formatUser(id) {
+    const [wasKnown, user] = await users(id);
+    return user?.email_canonical
+      ? user.email_canonical + (wasKnown ? "" : ` (${id})`)
+      : `${JSON.stringify(id)} (user id not found)`;
+  }
+  const drivesRepo = await platform
+    .getProvider<DatabaseServiceAPI>("database")
+    .getRepository<DriveFile>(DriveFile_TYPE, DriveFile);
+  const versionsRepo = await platform
+    .getProvider<DatabaseServiceAPI>("database")
+    .getRepository<FileVersion>(FileVersion_TYPE, FileVersion);
+  const filter = {
+    is_in_trash: false,
+  };
+  if (!args.all) filter["editing_session_key"] = { $ne: null };
+  const opts: FindOptions = { sort: { name: "asc" } };
+  if (args.name) filter["name"] = args.name;
+  const editedFiles = (await drivesRepo.find(filter, opts)).getEntities();
+  const formatDate = (date: Date) => date.toISOString();
+  const formatTS = (ts: number) => formatDate(new Date(ts));
+  console.error(`DriveFiles${args.all ? "" : " with non-null editing_session_key"}:`);
+  console.error("");
+  for (const dfile of editedFiles) {
+    console.error(`- ${dfile.name} (${dfile.id}) of ${await formatUser(dfile.creator)}`);
+    if (dfile.scope !== "personal") console.error(`  - scope:    ${dfile.scope}`);
+    if (dfile.is_directory) console.error("  - directory !");
+    if (dfile.is_in_trash) console.error("  - in trash !");
+    console.error(`  - modified: ${formatTS(dfile.last_modified)}`);
+    if (dfile.editing_session_key) {
+      const parsed = EditingSessionKeyFormat.parse(dfile.editing_session_key);
+      console.error("  - editing_session_key:");
+      console.error(`    - URL encoded:   ${encodeURIComponent(dfile.editing_session_key)}`);
+      console.error(`    - applicationId: ${parsed.applicationId}`);
+      console.error(`    - companyId:     ${parsed.companyId}`);
+      console.error(`    - instanceId:    ${JSON.stringify(parsed.instanceId)}`);
+      console.error(
+        `    - userId:        ${await formatUser(parsed.userId)} (${
+          parsed.userId === dfile.creator ? "same as creator ID" : "not the creator"
+        })`,
+      );
+      console.error(
+        `    - timestamp:     ${formatDate(parsed.timestamp)} (${Math.floor(
+          (new Date().getTime() - parsed.timestamp.getTime()) / 1000,
+        )}s ago)`,
+      );
+    }
+
+    const versions = (
+      await versionsRepo.find({ drive_item_id: dfile.id }, { sort: { date_added: "asc" } })
+    ).getEntities();
+    let previousSize = 0;
+    let lastVersion: FileVersion;
+    console.error("  - Versions:");
+    for (const version of versions) {
+      console.error(
+        `    - ${formatTS(version.date_added)} by ${await formatUser(version.creator_id)}`,
+      );
+      console.error(`        - id:          ${version.id}`);
+      console.error(
+        `        - size:        ${version.file_metadata.size} (${
+          version.file_metadata.size > previousSize ? "+" : ""
+        }${version.file_metadata.size - previousSize})`,
+      );
+      previousSize = version.file_metadata.size;
+      lastVersion = version;
+      console.error(`        - application: ${JSON.stringify(version.application_id)}`);
+    }
+    if (previousSize != dfile.size)
+      console.error(
+        `  - mismatched sizes: DriveFile.size is ${dfile.size} but last Version.file_metadata is ${previousSize}`,
+      );
+    if (lastVersion) {
+      const lastTimestamp = lastVersion.date_added;
+      if (lastTimestamp != dfile.last_modified)
+        console.error(
+          `  - mismatched FileVersion.date_added (${formatTS(
+            lastTimestamp,
+          )}) != DriveFile.last_modified (${formatTS(dfile.last_modified)}) - delta: ${
+            (lastTimestamp - dfile.last_modified) / 1000
+          }s`,
+        );
+      if (lastTimestamp != dfile.last_version_cache.date_added)
+        console.error(
+          `  - mismatched FileVersion.date_added (${formatTS(
+            lastTimestamp,
+          )}) != DriveFile.last_version_cache.date_added (${formatTS(
+            dfile.last_version_cache.date_added,
+          )}) - delta: ${(lastTimestamp - dfile.last_version_cache.date_added) / 1000}s`,
+        );
+      if (lastVersion.file_size != dfile.size)
+        console.error(
+          `  - mismatched FileVersion.file_size (${lastVersion.file_size}) != DriveFile.dfile.size (${dfile.size})`,
+        );
+    }
+  }
+  if (!editedFiles.length) console.error("  (no matching DriveFiles)");
+}
+
+const command: yargs.CommandModule<unknown, unknown> = {
+  command: "list",
+  describe: `
+    List current DriveFile items that have an editing_session_key set
+  `.trim(),
+
+  builder: {
+    all: {
+      type: "boolean",
+      alias: "a",
+      describe: "Include all DriveFiles (not just the ones with editing_session_keys)",
+      default: false,
+    },
+    name: {
+      type: "string",
+      alias: "n",
+      describe: "Filter DriveFiles by name (must be exact)",
+      default: false,
+    },
+  },
+  handler: async argv => {
+    const args = argv as unknown as ListArguments;
+    await runWithPlatform("editing_session list", async ({ spinner: _spinner, platform }) => {
+      console.error("\n");
+      await report(platform, args);
+      console.error("\n");
+    });
+  },
+};
+export default command;

tdrive/backend/node/src/cli/cmds/editing_session_cmds/parse.ts

@@ -0,0 +1,34 @@
+import yargs from "yargs";
+
+import { NonPlatformCommandYargsBuilder } from "../../utils/non-plaform-command-yargs-builder";
+import { EditingSessionKeyFormat } from "../../../services/documents/entities/drive-file";
+
+interface ParseArguments {
+  editing_session_key: string;
+}
+
+const command: yargs.CommandModule<unknown, unknown> = {
+  command: "parse <editing_session_key>",
+  describe: `
+    Parse the provided editing_session_key and output json data (to stderr)
+  `.trim(),
+
+  builder: {
+    ...NonPlatformCommandYargsBuilder,
+  },
+  handler: async argv => {
+    const args = argv as unknown as ParseArguments;
+    const parsed = EditingSessionKeyFormat.parse(decodeURIComponent("" + args.editing_session_key));
+    console.error(
+      JSON.stringify(
+        {
+          ageH: (new Date().getTime() - parsed.timestamp.getTime()) / (60 * 60 * 1000),
+          ...parsed,
+        },
+        null,
+        2,
+      ),
+    );
+  },
+};
+export default command;

tdrive/backend/node/src/core/platform/services/database/services/orm/connectors/postgres/postgres-query-builder.ts

@@ -47,9 +47,18 @@ export class PostgresQueryBuilder {
               values.push(...inClause);
             }
           } else {
-            const value = `${this.dataTransformer.toDbString(filter, columnsDefinition[key].type)}`;
-            whereClause += `${key} = $${idx++} AND `;
-            values.push(value);
+            const isANotEqualFilter = filter && Object.keys(filter).join("!") === "$ne";
+            if (filter === null || (isANotEqualFilter && filter["$ne"] === null)) {
+              whereClause += `${key} IS${filter === null ? "" : " NOT"} NULL`;
+            } else {
+              const filterValue = isANotEqualFilter ? filter["$ne"] : filter;
+              const value = `${this.dataTransformer.toDbString(
+                filterValue,
+                columnsDefinition[key].type,
+              )}`;
+              whereClause += `${key} ${isANotEqualFilter ? "!=" : "="} $${idx++} AND `;
+              values.push(value);
+            }
           }
         });
     }

tdrive/backend/node/src/core/platform/services/database/services/orm/repository/repository.ts

@@ -43,6 +43,11 @@ export type FindOptions = {
   sort?: SortOption;
 };
 
+export type AtomicCompareAndSetResult<FieldValueType> = {
+  didSet: boolean;
+  currentValue: FieldValueType | null;
+};
+
 /**
  * Repository to work with entities. Each entity type has its own repository instance.
  */
@@ -129,10 +134,7 @@ export default class Repository<EntityType> {
     fieldName: keyof EntityType,
     previousValue: FieldValueType | null,
     newValue: FieldValueType | null,
-  ): Promise<{
-    didSet: boolean;
-    currentValue: FieldValueType | null;
-  }> {
+  ): Promise<AtomicCompareAndSetResult<FieldValueType>> {
     if (previousValue === newValue)
       throw new Error(`Previous and new values are identical: ${JSON.stringify(previousValue)}`);
     return this.connector.atomicCompareAndSet(entity, fieldName, previousValue, newValue);