SG(A) install/upgrade testing matrix

[ajres]

Up to and including the SG(A) 1.5.0 the install and upgrade testing has been carried out manually using VM's on VirtualBox.

For the 1.5.1 release we want to automate some or all of the install/upgrade testing.

The initial proposal is to leverage the QE functional test framework that currently does some level on install testing as a side effect of standing up the functional test environments.

For SG 1.5 tests are run on centos (cc & di), RHEL(cc) and Win 10(cc)
For SGA 1.5 test are only run on centos

To achieve automated install/upgrade testing the following high level tasks must be completed:

• - Create functional tests that starts SG(A) and make a REST call to validate that the expected binary version is reported. This test will have the side effect of installing and uninstalling SG(A) from binary package.

• - Create functional tests that starts SG(A) and applies some load to SG(A), run sgcollect_info and validate the contents of the generated ZIP file. This test will have the side effect of installing and uninstalling SG(A) from binary package.

• - Extend the OS coverage of the above tests for all SG OS's, it is not required for all functional tests to pass on all OS's only the install/upgrade automation tests.

OS Download coverage

For the 1.5.1 release we want to achieve OS coverage parity with Couchbase Server on the couchbase.com downloads page. The current coverage for server and SG(A) are shown below.

Couchbase Server Enterprise, 5.0 Beta download OS coverage

  •  Ubuntu 16.04
  •  Ubuntu 14.04
  •  Red Hat 7
  •  Red Hat 6
  •  Windows (want more specific)
  •  mac OS (more specific)
  •  Debian 8
  •  Debian 7
  •  SuSE Linux 12
  •  SuSE Linux 11
  •  Oracle Linux 7
  •  Oracle Linux 6

Couchbase Server Community 4.6.0 (Developer Preview) Download OS coverage

  •  Ubuntu 14.04
  •  Ubuntu 12.04
  •  Red Hat 7
  •  Red Hat 6
  •  Windows
  •  mac OS
  •  Debian 8
  •  Debian 7
  •  SuSE Linux 11
  •  Oracle Linux 6

Sync Gateway 1.5.0 Download OS coverage

  •  Ubuntu
  •  Red Hat
  •  Windows
  •  mac OS

SG(A) downloads cover multiple OS versions, in these cases additional coverage on the downloads page can be achieved by duplicating the package binaries.
Other OS's that are currently not supported will minimally require updates to the installer scripts, but they may also require specific development changes.

[ajres]

The following implementation tickets have been created in the mobile-testkit repo for the tasks required to complete the automation of the installer testing.

Progress will be tracked on these tickets.

1. Create Jenkins jobs to run SG(A) install/upgrade/uninstall tests
2. For windows installation tests pre-load a Windows VM before running the install tests
3. Create a Minimal test that validates the SG is running by making a REST call on "/" endpoint on ADIM API
4. Create a Minimal test that validates the SGA is running by making a REST call on "/" endpoint on ADIM API
5. Validate that SGA test work on rhel and win 10
6. Work with QE to add support for running tests on Ubuntu
7. Work with QE to add support for running tests on macOS
8. Create a test to run cbcollect_info against SG and validate the contents of the generated .ZIP file
9. Create a test to run cbcollect_info against SGA and validate the contents of the generated .ZIP file
10. Write a test for SG to upgrade an existing installation, validate the upgrade by making a REST call on "/" endpoint on ADIM API before and after the upgrade to validate the running version
11. Write a test for SG(A) to upgrade an existing installation, validate the upgrade by making a REST call on "/" endpoint on ADIM API before and after the upgrade to validate the running version
12. Write a test to explicitly uninstall SG and validate that the service has been stopped and the installation directory removed.
13. Write a test to explicitly uninstall SGA and validate that the service has been stopped and the installation directory removed.