Migration framework for versioning of JS IPFS Repo
$ npm i ipfs-repo-migrationsThis package is inspired by the go-ipfs repo migration tool
As js-ipfs evolves and new technologies, algorithms and data structures are incorporated it is necessary to enable users to transition between versions. Different versions of js-ipfs may expect a different IPFS repo structure or content (see: IPFS repo spec, JS implementation ). So the IPFS repo is versioned, and this package provides a framework to create migrations to transition from one version of IPFS repo to the next/previous version.
This framework:
- Handles locking/unlocking of repository
- Defines migrations API
- Executes and reports migrations in both directions: forward and backward
- Simplifies creation of new migrations
- Works on the browser too!
> npm install ipfs-repo-migrationsimport migrations from 'ipfs-repo-migrations'import migrations from 'ipfs-repo-migrations'Example:
import migrations from 'ipfs-repo-migrations'
const repoPath = 'some/repo/path'
const currentRepoVersion = 7
const latestVersion = migrations.getLatestMigrationVersion()
const repoOptions = {
... // the same storage backend/storage options passed to `ipfs-repo`
}
if(currentRepoVersion < latestVersion){
// Old repo! Lets migrate to latest version!
await migrations.migrate(repoPath, latestVersion, {
repoOptions
})
}To migrate your repository using the CLI, see the how to run migrations tutorial.
Executes a forward migration to a specific version, or to the latest version if a specific version is not specified.
Arguments:
path(string, mandatory) - path to the repo to be migratedrepoOptions(object, mandatory) - options that are passed to migrations, that use them to construct the datastore. (options are the same as for IPFSRepo).toVersion(int, mandatory) - version to which the repo should be migrated.options(object, optional) - options for the migrationoptions.ignoreLock(bool, optional) - if true will not lock the repo when applying migrations. Use with caution.options.onProgress(function, optional) - callback that is called during each migration to report progress.options.isDryRun(bool, optional) - flag that indicates if it is a dry run that should give the same output as running a migration but without making any actual changes.
Signature of the progress callback.
Arguments:
migration(object) - object of migration that just successfully finished running. See Architecture of migrations for details.counter(int) - index of current migration.totalMigrations(int) - total count of migrations that will be run.
Executes backward migration to a specific version.
Arguments:
path(string, mandatory) - path to the repo to be revertedrepoOptions(object, mandatory) - options that are passed to migrations, that use them to construct the datastore. (options are the same as for IPFSRepo).toVersion(int, mandatory) - version to which the repo should be reverted to.options(object, optional) - options for the reversionoptions.ignoreLock(bool, optional) - if true will not lock the repo when applying migrations. Use with caution.options.onProgress(function, optional) - callback that is called during each migration to report progress.options.isDryRun(bool, optional) - flag that indicates if it is a dry run that should give the same output as running a migration but without making any actual changes.
Return the version of the latest migration.
Migrations are one of those things that can be extremely painful on users. At the end of the day, we want users never to have to think about it. The process should be:
- SAFE. No data lost. Ever.
- Revertible. Tools must implement forward and backward (if possible) migrations.
- Tests. Migrations have to be well tested.
- To Spec. The tools must conform to the spec.
If your migration has several parts, it should be fail-proof enough that if one part of migration fails the previous changes are reverted before propagating the error. If possible then the outcome should be consistent repo so it migration could be run again.
All migrations are placed in the /src/migrations folder. Each folder there represents one migration that follows the migration
API.
All migrations are collected in /src/migrations/index.js, which should not be edited manually.
The order of migrations is important and migrations must be sorted in ascending order.
Each migration must follow this API. It must export an object in its index.js that has following properties:
version(int) - Number that represents the version which the repo will migrate to (eg.8will move the repo to version 8).description(string) - Brief description of what the migrations does.migrate(function) - Function that performs the migration (see signature of this function below)revert(function) - If defined then this function will revert the migration to the previous version. Otherwise it is assumed that it is not possible to revert this migration.
Do not confuse this function with the require('ipfs-repo-migrations').migrate() function that drives the whole migration process!
Arguments:
repoPath(string) - absolute path to the root of the reporepoOptions(object, optional) - object containingIPFSRepooptions, that should be used to construct a datastore instance.
Do not confuse this function with the require('ipfs-repo-migrations').revert() function that drives the whole backward migration process!
Arguments:
repoPath(string) - path to the root of the reporepoOptions(object, optional) - object containingIPFSRepooptions, that should be used to construct the datastore instance.
The migration might need to perform specific tasks in browser or NodeJS environment. In such a case create
migration file /migrations/migration-<number>/index_browser.js which have to follow the same API is described before.
Then add entry in package.json to the browser field as follow:
'./migrations/migration-<number>/index.js': './migrations/migration-<number>/index_browser.js'
In browser environments then index.js will be replaced with index_browser.js.
Simple migrations should not need to distinguish between these environments as the datastore implementation will handle the main differences.
There are currently two main datastore implementations:
datastore-fsthat is backed by file system and is used mainly in the NodeJS environmentdatastore-idbthat is backed by LevelDB and is used mainly in the browser environment
Both implementations share the same API and hence are interchangeable.
When the migration is run in a browser environment, datastore-fs is automatically replaced with datastore-idb even
when it is directly imported (require('datastore-fs') will return datastore-idb in a browser).
So with simple migrations you shouldn't worry about the difference between datastore-fs and datastore-idb
and by default use the datastore-fs package (as the replace mechanism does not work vice versa).
The recommended way to write a new migration is to first bootstrap a dummy migration using the CLI:
> npm run new-migrationA new folder is created with the bootstrapped migration. You can then simply fill in the required fields and write the rest of the migration!
When a new migration is created, new version of this package have to be released. Afterwards version of this package in js-ipfs-repo have to be updated
together with the repo version that IPFSRepo expects. Then the updated version of js-ipfs-repo should be propagated to js-ipfs.
If a migration affects any of the following functionality, it must provide tests for the following functions to work under the version of the repo that it migrates to:
/src/repo/version.js:getVersion()- retrieving repository's version/src/repo/lock.js:lock()- locking repository that uses file system/src/repo/lock-memory.js:lock()- locking repository that uses memory
Every migration must have test coverage. Tests for migrations should be placed in the /test/migrations/ folder. Most probably
you will have to plug the tests into browser.js/node.js if they require specific bootstrapping on each platform.
For interop with go-ipfs it might be necessary just to bump a version of a repo without any actual modification as there might not be any changes needed in the JS implementation. For that purpose you can create an "empty migration".
The easiest way to do so is with the CLI:
> npm run new-migration -- --emptyThis will create an empty migration with the next version.
| IPFS repo version | JS IPFS version |
|---|---|
| 7 | v0.0.0 |
| 8 | v0.48.0 |
| 9 | v0.49.0 |
This is the initial version of the datastore, inherited from go-IPFS in an attempt to maintain cross-compatibility between the two implementations.
Blockstore keys are transformed into base32 representations of the multihash from the CID of the block.
Pins were migrated from a DAG to a Datastore - see ipfs/js-ipfs#2771
[email protected] upgrades the level-js dependency from 4.x.x to 5.x.x. This update requires a database migration to convert all string keys/values into buffers. Only runs in the browser, node is unaffected. See Level/level-js#179
In order to have good overview of what version of package contains what kind of migration, to every release there
should be appended version's metadata in format migr-<versionOfLatestMigration>. If for releasing is used aegir
you can use the release --metadata option.
Licensed under either of
- Apache 2.0, (LICENSE-APACHE / http://www.apache.org/licenses/LICENSE-2.0)
- MIT (LICENSE-MIT / http://opensource.org/licenses/MIT)
Contributions welcome! Please check out the issues.
Also see our contributing document for more information on how we work, and about contributing in general.
Please be aware that all interactions related to this repo are subject to the IPFS Code of Conduct.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.
