Metacat 3.1.0 Release Plan

[artntek]

list of production hosts

Matt said:

Also, from a scheduling perspective, I think it would be best to finish all of this by the end of November at the latest, sooner if possible.

Do in this order, with sufficient time in between to find any issues:

1. GOA
2. KNB
3. ADC

See latest per-host upgrade procedure

---

SUPERSEDED - SEE BELOW

Agreed with @taojing2002, @mbjones 10/15/24:

Skip 3.0.0; upgrade hosts from 2.19 to 3.1.0

Do in this order, with sufficient time in between to find any issues:

1. GOA (on current host)
   1. NOTE: move data to ceph /var/data/repos storage during the upgrade - see Matt's note below
2. KNB (on current host)
   1. NOTE: move data to ceph /var/data/repos storage during the upgrade - see Matt's note below
3. GOA (move to k8s)
4. KNB (move to k8s)
5. ADC (on current host)
   1. NOTE: move data to ceph /var/data/repos storage during the upgrade - see Matt's note below
6. ADC (move to k8s)

This way, all the database conversions and hashstore conversions happen in the existing legacy installations, before moving to k8s, and we minimize “changes all at once”.

Matt said:

_An important step for # 2 and # 5 is to be sure we are using the new ceph pdg-subvol storage for KNB and Arctica before we do the upgrade to k8s.
I would probaby move each host into our ceph /var/data/repos storage during the upgrade in 1, 2, and 5, which would then allow an easy switching from dedicatedVM -> k8s -> dedicatedVM (rollback only if needed)_

[artntek]

Release Plan

For the most up-to-date version, see whiteboard here

[artntek]

[artntek]

ACTION ITEMS

• (Jing) Add a flag to stop hashstore conversion starting automatically when DB upgrade done DONE - see Add a flag to stop hashstore conversion starting automatically when DB upgrade done #1987

• (Jing) Change code to fall back to legacy data store if object not found in hashstore

  • 10/21/24 = Jing & Matthew agreed no longer needed, since conversion for large corpuses happens before instance is online, and users see a metacat error for small instances, but these will be offline only for a very short time.

• (Jing) Document quickest way to dump the checksum table and load it again #1991 -- & see Matt's notes

• (anyone) Write a script to get delta of datasets and index them #1990

• (Matthew) Deploy ADC Wordpress as a bitnami chart

• (Matthew) Test hashstore-java conversion in k8s and address issues

[artntek]

ACTION ITEM: Document quickest way to dump the checksum table and load it again

@mbjones said:

I suspect that running sql COPY TO and COPY FROM would be fastest, especially if you disable indexes and other constraints first. Similarly, you can use pg_dump and pg_restore, but make sure you set flags to make it fast and parallel (-j). Postgres has a nice overview of bulk data loading here: https://www.postgresql.org/docs/current/populate.html

From that guide:
> If you are loading a freshly created table, the fastest method is to create the table,
> bulk load the table's data using COPY, then create any indexes needed for the table.
> Creating an index on pre-existing data is quicker than updating it incrementally as
> each row is loaded.

[artntek]

After discussion with Nick, we decided to apply the following IDs when copying data to ceph:

postrges uid: 59996 gid: 59996
tomcat   uid: 59997 gid: 59997

[artntek]

Test hashstore-java conversion in k8s and address issues

sysmeta contains checksum algorithm names that do not match the allowed list in hashstore. (e.g. SHA1 instead of SHA-1). These need to be fixed in the DB before hashstore conversion runs.

for k8s copy of ADC prod database

SELECT DISTINCT checksum_algorithm FROM systemmetadata WHERE checksum_algorithm NOT IN
   ('MD2','MD5','SHA-1','SHA-256','SHA-384','SHA-512','SHA-512/224','SHA-512/256');
 checksum_algorithm
--------------------
 SHA256
 SHA1
(2 rows)

UPDATE systemmetadata SET checksum_algorithm='SHA-1' WHERE checksum_algorithm='SHA1';
UPDATE 266243

UPDATE systemmetadata SET checksum_algorithm='SHA-256' WHERE checksum_algorithm='SHA256';
UPDATE 505317