ServerSync

This package provides server-to-server syncing of collections. It is modeled along the lines of what clients do, too, i.e., collections are locally cached, in case the remote server gets disconnected.

How it Works

This uses DDP.connect to connect to a remote server and subscribe to collections on it, but also caches that collection in the local mongodb. It gracefully handles periods of being disconnected, incl. syncing local changes back to the remote if they happened while disconnected, but not if at the same time the document was changed remotely. This means that the collection is "owned" by the remote server and any changes made there trump any local changes.

Note: make sure to remove the autopublish package on the server. ServerSync only works properly with explicit publications. With autopublish, the entire collection is resynced on each reconnect which is undesirable and will overwrite any and all local changes, even when conflict free with what has (or hasn't) changed on the server.

Note: Also, this package does not like custom _ids. It assumes that document _ids are generated by Meteor or Mongo, and in particular are unique. If you are setting your own document _ids, this package will probably not prevent you from messing up your sync.

The package supports multiple modes, specified in the second argument to the sync function:

• read: means that the client never writes to the server. Changes to the local copy of the collection will be ignored by the server and may be overwritten by the server at any time. In this mode only it is possible to provide an existing local collection (in the 'collection' option) into which all remote documents will be added. This is particularly useful well wanting to merge remote collections from multiple remote hosts into one (for instance for creating a dashboard with data from several application servers).

• write (default): In this mode a client can insert new documents into the shared collection at any time, and even if these insertions happen while disconnected, the new items will get synced to the server on reconnect. Updates and removals can also be done while offline, but if the server makes changes while offline, these changes will overwrite any local changes on reconnect.

  Note that local changes are detected using collection hooks. This means that direct mongodb manipulations (e.g., via mongorestore) will not be detected/synced. It also means that you can avoid syncing changes by using the .direct methods.

New in Version 0.5: Changes from the server are applied in batch, i.e., only once all data is received are changes going to be applied to the local collection. This ensure sync atomicity and prevents data inconstencies due to partial syncs. This can be important when changes in multiple documents or collections need to be done simulatenously in order for the database to be consistent/valid.

Example

Meteor.startup(() => {
  // connect to master:
  a = new ServerSyncClient("http://localhost:3000", {
    onConnect: function() {
      console.log("connected to master");
    },
    onReconnect: function() {
      console.log("reconnected to master");
    },
    beforeSyncDirty: function(count) {
      console.log("beforeSyncDirty", count);
    },
    afterSyncDirty: function(count) {
      console.log("afterSyncDirty", count);
    }
  });
  a.sync('items', {  // sync the "items" collection from the master
    onReady: function() {
      var coll = a.getCollection('items');
      // do something with this collection (e.g., publish it locally)
      console.log("ready", coll.find().count());
    },
    beforeSyncUp: function(type, id, doc) { 
      console.log("beforeSyncUp", type, id, doc);
    },
    beforeSyncDown: function(type, id, doc) { 
      console.log("beforeSyncDown", type, id, doc);
    },
    afterSyncUp: function(type, id, doc) { 
      console.log("afterSyncUp", type, id, doc);
    },
    afterSyncDown: function(type, id, doc) { 
      console.log("afterSyncDown", type, id, doc);
    },
    args: [Date.now()] // arguments to pass to publication function on server
  });
});

See https://github.com/chfritz/serversync-example for a full example.

Detailed Behavior

This section describes the behavior of serversync in more detail. Specifically we consider all possible cases.

Online

This is easy. Any insertion, update, or removal on either side should sync to the other side immediately.

Offline

This is the tricky part. We need to consider all possible combinations on master and slave during the offline period. In the following table we assume that each action in a row regards the same _id.

Master	Slave	Result Master	Result Slave	Comments
insert	-	-	insert

•  | insert | insert        | -            |
  

update | - | - | update |

•  | update | update        | -            |
  

remove | - | - | remove |

•  | remove | remove        | -            |
  

update | update | - | update (undo own change) | We do conflict resolution at the document level. update | remove | - | update (undo remove) | remove | update | - | remove |

•  | insert | n/a           | n/a          | not possible: we disallow non-unique _ids
  

insert | * | n/a | n/a | not possible: we disallow non-unique _ids

Not running

This is different from disconnected because the app may lose it's state or the initial sync may not have happened yet.

The following time lines show the cases I thought of and how they are being handled.

M = Master, S = Slave, - = running (up), _ = not running (down)
(I)nsert, (D)elete, X = anything

M   ___-IX--  _-X----  _--X-__---  _-____-X-
S   _-IX----  ___-X--  _-___---X-  _--X-----

# Cases that require a persistent change set:
M   _____---IX-
S   _-IX-__----