-
Notifications
You must be signed in to change notification settings - Fork 16
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
MongoDB Migration release notes for 2023-10 (#166)
* MongoDB Migration release notes for 2023-10 --------- Co-authored-by: Melissa Hilliard <[email protected]>
- Loading branch information
1 parent
b522b9a
commit d285028
Showing
6 changed files
with
293 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,6 @@ | ||
| yaml-files: | ||
| - '*.yaml' | ||
| - '*.yml' | ||
| - '*.yml' | ||
|
|
||
| ignore: | ||
| - /release-notes/2023-10/MongoDB_Migration/templates/*.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| apiVersion: v2 | ||
| name: ni_systemlink_enterprise_migrate_mongodb | ||
| description: Creates Jobs in the cluster to migrate data from individual service MongoDB instances to a central instance. See README.md for more details. | ||
| type: application | ||
| version: 0.1.0 | ||
| appVersion: '1.16.0' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # Existing MongoDB Data Migration to a Single Instance | ||
|
|
||
| ## Do you need to worry about what is in this folder? | ||
|
|
||
| SystemLink Enterprise installations prior to 2023-10 deployed a MongoDB instance per service. Version 2023-10 uses a global MongoDB connection string that allows customers to consolidate over 20 different MongoDB instances down to one. This content is for users who need to migrate existing SystemLink Enterprise MongoDB data to a single MongoDB instance. You do not need to perform the following if you are deploying a new installation of SystemLink Enterprise. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| Before you begin, you must provision your MongoDB instance with databases and users. Refer to [Configuring Systemlink Enterprise](https://www.ni.com/docs/en-US/bundle/systemlink-enterprise/page/config-systemlink-enterprise.html) for a list of databases and password fields to update if using separate credentials per database. NOTE: Ensure that you backup the current database passwords before the migration. | ||
|
|
||
| ## Enable a Single Instance | ||
|
|
||
| Refer to [Configuring Systemlink Enterprise](https://www.ni.com/docs/en-US/bundle/systemlink-enterprise/page/config-systemlink-enterprise.html) for instructions on editing the Helm values that determine this functionality. | ||
|
|
||
| ## Migration Plan | ||
|
|
||
| ### MongoDB Atlas | ||
|
|
||
| If you are using a [MongoDB Atlas](https://www.ni.com/r/mongodbatalas) cluster use the MongoDB Atlas migration tools to move data between the service databases and MongoDB Atlas. | ||
|
|
||
| ### Self-hosted MongoDB | ||
|
|
||
| Run the following command from inside the cluster to migrate the documents to the new MongoDB instance. | ||
|
|
||
| ```sh | ||
| mongodump --config=/etc/mongodump_config.yaml --archive | mongorestore --archive --config=/etc/mongorestore_config.yaml | ||
| ``` | ||
|
|
||
| Where | ||
|
|
||
| - `mongo*_config.yaml` is where the connection string and password are configured. | ||
| - `--archive` allows you to pipe (`|`) the output directly into the `mongorestore` command | ||
|
|
||
| #### Helm Chart | ||
|
|
||
| This folder includes a Helm Chart to install into your existing Kubernetes cluster to execute the plan described above for self-hosted MongoDB. Your Kubernetes cluster must be able to pull the `bitnami/mongodb:5.0.19-debian-11-r3` Docker image. | ||
|
|
||
| 1. Fill out the `values.yaml` file with MongoDB password secrets stored in `systemlink-secrets.yaml`. Refer to all `<ATTENTION>` comments. | ||
| 2. Change active directory on command line to the folder containing this file. | ||
| 3. Run `helm install ni-systemlink-enterprise-migrate-mongodb .` | ||
|
|
||
| You can observe the progress of each command Job through the Pod logs. | ||
|
|
||
| Before continuing, run `kubectl get jobs` and observe the `COMPLETIONS` column to ensure all Jobs have completed. `1/1` indicates the Job completed successfully. |
20 changes: 20 additions & 0 deletions
20
release-notes/2023-10/MongoDB_Migration/templates/configmap.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| {{- range $name, $data := .Values.migrating_from.services }} | ||
| --- | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: mongodump-{{ $name }}-config-yaml | ||
| data: | ||
| mongodump_config.yaml: |- | ||
| password: {{ $data.dump.password }} | ||
| uri: mongodb://{{ $data.username }}@{{ $data.dump.db0 }}:27017,{{ $data.dump.db1 }}:27017,{{ $data.dump.db2 }}:27017/{{ $data.database }} | ||
| --- | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: mongorestore-{{ $name }}-config-yaml | ||
| data: | ||
| mongorestore_config.yaml: |- | ||
| password: {{ $.Values.migrating_to.admin_password }} | ||
| uri: {{ trimSuffix "/" $.Values.migrating_to.connection_string_including_admin_username}}/{{ $data.database }}?authSource=admin | ||
| {{- end }} |
54 changes: 54 additions & 0 deletions
54
release-notes/2023-10/MongoDB_Migration/templates/job.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| {{- range $service := keys .Values.migrating_from.services }} | ||
| --- | ||
| apiVersion: batch/v1 | ||
| kind: Job | ||
| metadata: | ||
| name: mongodb-migration-{{ $service }} | ||
| spec: | ||
| backoffLimit: 1 | ||
| template: | ||
| spec: | ||
| automountServiceAccountToken: false | ||
| containers: | ||
| - name: mongodb-migration-{{ $service }} | ||
| image: bitnami/mongodb:5.0.19-debian-11-r3 | ||
| command: ['bin/bash'] | ||
| args: | ||
| [ | ||
| '-c', | ||
| 'mongodump --config=/etc/mongodump_config.yaml --archive | mongorestore --archive --config=/etc/mongorestore_config.yaml', | ||
| ] | ||
| volumeMounts: | ||
| - name: mongodump-config-yaml | ||
| mountPath: /etc/mongodump_config.yaml | ||
| subPath: mongodump_config.yaml | ||
| - name: mongorestore-config-yaml | ||
| mountPath: /etc/mongorestore_config.yaml | ||
| subPath: mongorestore_config.yaml | ||
| securityContext: | ||
| readOnlyRootFilesystem: true | ||
| allowPrivilegeEscalation: false | ||
| capabilities: | ||
| drop: | ||
| - ALL | ||
| resources: | ||
| requests: | ||
| cpu: 2 | ||
| memory: 8Gi | ||
| limits: | ||
| memory: 8Gi | ||
| volumes: | ||
| - name: mongodump-config-yaml | ||
| configMap: | ||
| name: mongodump-{{ $service }}-config-yaml | ||
| - name: mongorestore-config-yaml | ||
| configMap: | ||
| name: mongorestore-{{ $service }}-config-yaml | ||
| restartPolicy: Never | ||
| securityContext: | ||
| runAsNonRoot: true | ||
| runAsUser: 10001 | ||
| runAsGroup: 20001 | ||
| seccompProfile: | ||
| type: RuntimeDefault | ||
| {{- end }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,165 @@ | ||
| migrating_to: | ||
| admin_password: '' # <ATTENTION> | ||
| connection_string_including_admin_username: 'mongodb://...' # <ATTENTION> | ||
| migrating_from: | ||
| services: | ||
| assetservice: | ||
| database: niapm | ||
| username: niapm | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from assetservice.secrets.mongodb.servicePassword | ||
| db0: 'assetmanagementservice-mongodb-0.assetmanagementservice-mongodb-headless' | ||
| db1: 'assetmanagementservice-mongodb-1.assetmanagementservice-mongodb-headless' | ||
| db2: 'assetmanagementservice-mongodb-2.assetmanagementservice-mongodb-headless' | ||
| fileingestionservices: | ||
| database: files | ||
| username: files | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from fileingestion.secrets.mongodb.servicePassword | ||
| db0: 'fileingestion-fileingestion-mongodb-0.fileingestion-fileingestion-mongodb-headless' | ||
| db1: 'fileingestion-fileingestion-mongodb-1.fileingestion-fileingestion-mongodb-headless' | ||
| db2: 'fileingestion-fileingestion-mongodb-2.fileingestion-fileingestion-mongodb-headless' | ||
| license: | ||
| database: nilicense | ||
| username: nilicense | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from license.secrets.mongodb.servicePassword | ||
| db0: 'license-license-mongodb-0.license-license-mongodb-headless' | ||
| db1: 'license-license-mongodb-1.license-license-mongodb-headless' | ||
| db2: 'license-license-mongodb-2.license-license-mongodb-headless' | ||
| nbexecservice: | ||
| database: ninbexec | ||
| username: ninbexec | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from nbexecservice.secrets.mongodb.servicePassword | ||
| db0: 'nbexecservice-mongodb-0.nbexecservice-mongodb-headless' | ||
| db1: 'nbexecservice-mongodb-1.nbexecservice-mongodb-headless' | ||
| db2: 'nbexecservice-mongodb-2.nbexecservice-mongodb-headless' | ||
| nicomments: | ||
| database: comments | ||
| username: nicomments | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from comments.secrets.mongodb.servicePassword | ||
| db0: 'comments-comments-mongodb-0.comments-comments-mongodb-headless' | ||
| db1: 'comments-comments-mongodb-1.comments-comments-mongodb-headless' | ||
| db2: 'comments-comments-mongodb-2.comments-comments-mongodb-headless' | ||
| nidataframe: | ||
| database: nidataframe | ||
| username: nidataframe | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from dataframeservice.secrets.mongodb.servicePassword | ||
| db0: 'dataframeservice-mongodb-0.dataframeservice-mongodb-headless' | ||
| db1: 'dataframeservice-mongodb-1.dataframeservice-mongodb-headless' | ||
| db2: 'dataframeservice-mongodb-2.dataframeservice-mongodb-headless' | ||
| notification: | ||
| database: ninotification | ||
| username: ninotification | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from notification.secrets.mongodb.servicePassword | ||
| db0: 'notification-notification-mongodb-0.notification-notification-mongodb-headless' | ||
| db1: 'notification-notification-mongodb-1.notification-notification-mongodb-headless' | ||
| db2: 'notification-notification-mongodb-2.notification-notification-mongodb-headless' | ||
| repositoryservice: | ||
| database: nirepo | ||
| username: nirepo | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from repository.secrets.mongodb.servicePassword | ||
| db0: 'repository-repository-mongodb-0.repository-repository-mongodb-headless' | ||
| db1: 'repository-repository-mongodb-1.repository-repository-mongodb-headless' | ||
| db2: 'repository-repository-mongodb-2.repository-repository-mongodb-headless' | ||
| routines: | ||
| database: niroutines | ||
| username: niroutines | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from routineservice.secrets.mongodb.servicePassword | ||
| db0: 'routineservice-routines-mongodb-0.routineservice-routines-mongodb-headless' | ||
| db1: 'routineservice-routines-mongodb-1.routineservice-routines-mongodb-headless' | ||
| db2: 'routineservice-routines-mongodb-2.routineservice-routines-mongodb-headless' | ||
| routinescheduletrigger: | ||
| database: niroutinescheduletrigger | ||
| username: niroutinescheduletrigger | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from routinescheduletrigger.secrets.mongodb.servicePassword | ||
| db0: 'routinescheduletrigger-routineschedule-mongodb-0.routinescheduletrigger-routineschedule-mongodb-headless' | ||
| db1: 'routinescheduletrigger-routineschedule-mongodb-1.routinescheduletrigger-routineschedule-mongodb-headless' | ||
| db2: 'routinescheduletrigger-routineschedule-mongodb-2.routinescheduletrigger-routineschedule-mongodb-headless' | ||
| saltmaster-minions: | ||
| database: minions | ||
| username: minions | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from saltmaster.secrets.mongodb.minionsPassword | ||
| db0: 'saltmaster-saltmaster-mongodb-0.saltmaster-saltmaster-mongodb-headless' | ||
| db1: 'saltmaster-saltmaster-mongodb-1.saltmaster-saltmaster-mongodb-headless' | ||
| db2: 'saltmaster-saltmaster-mongodb-2.saltmaster-saltmaster-mongodb-headless' | ||
| saltmaster-pillars: | ||
| database: pillars | ||
| username: pillars | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from saltmaster.secrets.mongodb.pillarsPassword | ||
| db0: 'saltmaster-saltmaster-mongodb-0.saltmaster-saltmaster-mongodb-headless' | ||
| db1: 'saltmaster-saltmaster-mongodb-1.saltmaster-saltmaster-mongodb-headless' | ||
| db2: 'saltmaster-saltmaster-mongodb-2.saltmaster-saltmaster-mongodb-headless' | ||
| sysmgmtevent: | ||
| database: nisysmgmtevent | ||
| username: nisysmgmtevent | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from sysmgmtevent.secrets.mongodb.servicePassword | ||
| db0: 'sysmgmtevent-sysmgmtevent-mongodb-0.sysmgmtevent-sysmgmtevent-mongodb-headless' | ||
| db1: 'sysmgmtevent-sysmgmtevent-mongodb-1.sysmgmtevent-sysmgmtevent-mongodb-headless' | ||
| db2: 'sysmgmtevent-sysmgmtevent-mongodb-2.sysmgmtevent-sysmgmtevent-mongodb-headless' | ||
| systemsmanagementservice: | ||
| database: nisystemsmanagement | ||
| username: nisystems | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from systems.secrets.mongodb.servicePassword | ||
| db0: 'systemsmanagementservice-systems-mongodb-0.systemsmanagementservice-systems-mongodb-headless' | ||
| db1: 'systemsmanagementservice-systems-mongodb-1.systemsmanagementservice-systems-mongodb-headless' | ||
| db2: 'systemsmanagementservice-systems-mongodb-2.systemsmanagementservice-systems-mongodb-headless' | ||
| taghistorian: | ||
| database: taghistorian | ||
| username: taghistorian | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from taghistorian.secrets.mongodb.servicePassword | ||
| db0: 'taghistorian-taghistorian-mongodb-0.taghistorian-taghistorian-mongodb-headless' | ||
| db1: 'taghistorian-taghistorian-mongodb-1.taghistorian-taghistorian-mongodb-headless' | ||
| db2: 'taghistorian-taghistorian-mongodb-2.taghistorian-taghistorian-mongodb-headless' | ||
| tags: | ||
| database: tags | ||
| username: tags | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from tags.secrets.mongodb.servicePassword | ||
| db0: 'tags-tags-mongodb-0.tags-tags-mongodb-headless' | ||
| db1: 'tags-tags-mongodb-1.tags-tags-mongodb-headless' | ||
| db2: 'tags-tags-mongodb-2.tags-tags-mongodb-headless' | ||
| userdata: | ||
| database: niuserdata | ||
| username: niuserdata | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from userdata.secrets.mongodb.servicePassword | ||
| db0: 'userdata-userdata-mongodb-0.userdata-userdata-mongodb-headless' | ||
| db1: 'userdata-userdata-mongodb-1.userdata-userdata-mongodb-headless' | ||
| db2: 'userdata-userdata-mongodb-2.userdata-userdata-mongodb-headless' | ||
| userservices-users: | ||
| database: users | ||
| username: userservice | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from userservices.secrets.mongodb.userServicePassword | ||
| db0: 'userservices-userservices-mongodb-0.userservices-userservices-mongodb-headless' | ||
| db1: 'userservices-userservices-mongodb-1.userservices-userservices-mongodb-headless' | ||
| db2: 'userservices-userservices-mongodb-2.userservices-userservices-mongodb-headless' | ||
| userservices-keyservices: | ||
| database: keyservices | ||
| username: keyservice | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from userservices.secrets.mongodb.keyServicePassword | ||
| db0: 'userservices-userservices-mongodb-0.userservices-userservices-mongodb-headless' | ||
| db1: 'userservices-userservices-mongodb-1.userservices-userservices-mongodb-headless' | ||
| db2: 'userservices-userservices-mongodb-2.userservices-userservices-mongodb-headless' | ||
| webappservices: | ||
| database: webapps | ||
| username: webapps | ||
| dump: | ||
| password: '' # <ATTENTION> Replace with value from webappservices.secrets.mongodb.servicePassword | ||
| db0: 'webappservices-mongodb-0.webappservices-mongodb-headless' | ||
| db1: 'webappservices-mongodb-1.webappservices-mongodb-headless' | ||
| db2: 'webappservices-mongodb-2.webappservices-mongodb-headless' |