Feature #2735 v6.0.0 beta6

.github/ISSUE_TEMPLATE/documentation.md

@@ -0,0 +1,68 @@
+---
+name: Documentation
+about: Update the documentation
+title: 'Documentation: '
+labels: 'alert: NEED ACCOUNT KEY, alert: NEED MORE DEFINITION, alert: NEED CYCLE ASSIGNMENT, type: documentation'
+assignees: ''
+
+---
+
+*Replace italics below with details for this issue.*
+
+## Describe the Task ##
+*Provide a description of the task here.*
+
+### Time Estimate ###
+*Estimate the amount of work required here.*
+*Issues should represent approximately 1 to 3 days of work.*
+
+### Sub-Issues ###
+Consider breaking the task down into sub-issues.
+- [ ] *Add a checkbox for each sub-issue here.*
+
+### Relevant Deadlines ###
+*List relevant project deadlines here or state NONE.*
+
+### Funding Source ###
+*Define the source of funding and account keys here or state NONE.*
+
+## Define the Metadata ##
+
+### Assignee ###
+- [ ] Select appropriate **assignee** for this issue
+
+### Labels ###
+- [ ] Review default **alert** labels
+- [ ] Select **component(s)**
+- [ ] Select **priority**
+- [ ] Select **requestor(s)**
+
+### Milestone and Projects ###
+- [ ] Select **Milestone** as a **METplus-Wrappers-X.Y.Z** version, **Consider for Next Release**, or **Backlog of Development Ideas**
+- [ ] For a **METplus-Wrappers-X.Y.Z** version, select the **METplus-Wrappers-X.Y.Z Development** project
+
+## Define Related Issue(s) ##
+Consider the impact to the other METplus components.
+- [ ] [METplus](https://github.com/dtcenter/METplus/issues/new/choose), [MET](https://github.com/dtcenter/MET/issues/new/choose), [METdataio](https://github.com/dtcenter/METdataio/issues/new/choose), [METviewer](https://github.com/dtcenter/METviewer/issues/new/choose), [METexpress](https://github.com/dtcenter/METexpress/issues/new/choose), [METcalcpy](https://github.com/dtcenter/METcalcpy/issues/new/choose), [METplotpy](https://github.com/dtcenter/METplotpy/issues/new/choose)
+
+## Task Checklist ##
+See the [METplus Workflow](https://metplus.readthedocs.io/en/latest/Contributors_Guide/github_workflow.html) for details.
+- [ ] Complete the issue definition above, including the **Time Estimate** and **Funding Source**.
+- [ ] Fork this repository or create a branch of **develop**.
+Branch name: `feature_<Issue Number>_<Description>`
+- [ ] Complete the development and test your changes.
+- [ ] Add/update log messages for easier debugging.
+- [ ] Add/update unit tests.
+- [ ] Add/update documentation.
+- [ ] Add any new Python packages to the [METplus Components Python Requirements](https://metplus.readthedocs.io/en/develop/Users_Guide/appendixA.html#metplus-components-python-packages) table.
+- [ ] For any new datasets, an entry to the [METplus Verification Datasets Guide](https://metplus.readthedocs.io/en/latest/Verification_Datasets/index.html).
+- [ ] Push local changes to GitHub.
+- [ ] Submit a pull request to merge into **develop**.
+Pull request: `feature <Issue Number> <Description>`
+- [ ] Define the pull request metadata, as permissions allow.
+Select: **Reviewer(s)** and **Development** issue
+Select: **Milestone** as the next official version
+Select: **METplus-Wrappers-X.Y.Z Development** project for development toward the next official release
+- [ ] Iterate until the reviewer(s) accept and merge your changes.
+- [ ] Delete your fork or branch.
+- [ ] Close this issue.

.github/ISSUE_TEMPLATE/enhancement_request.md

@@ -1,7 +1,7 @@
 ---
 name: Enhancement request
 about: Improve something that it's currently doing
-title: ''
+title: 'Enhancement: '
 labels: 'alert: NEED ACCOUNT KEY, alert: NEED MORE DEFINITION, alert: NEED CYCLE ASSIGNMENT, type: enhancement'
 assignees: ''
 

.github/jobs/docker_build_metplus_images.sh

@@ -17,7 +17,7 @@ fi
 metplus_version=${SOURCE_BRANCH:1}
 
 # Get MET tag and adjust MET Docker repo if develop
-met_tag=$("${GITHUB_WORKSPACE}"/metplus/component_versions.py -v "${metplus_version}" -o MET -f "{X}.{Y}-latest")
+met_tag=$("${GITHUB_WORKSPACE}"/metplus/component_versions.py -v "${metplus_version}" -o MET -f "{X}.{Y}-latest" --no-get_dev_version)
 echo "$met_tag"
 
 MET_DOCKER_REPO=met

.github/jobs/docker_setup.sh

@@ -32,7 +32,7 @@ echo "TIMING: docker pull ${DOCKERHUB_TAG} took `printf '%02d' $(($duration / 60
 export DOCKERFILE_PATH=${GITHUB_WORKSPACE}/internal/scripts/docker/Dockerfile
 
 metplus_version=$(head -n 1 "${GITHUB_WORKSPACE}/metplus/VERSION")
-MET_TAG=$("${GITHUB_WORKSPACE}"/metplus/component_versions.py -v "${metplus_version}" -o MET -f "{X}.{Y}-latest")
+MET_TAG=$("${GITHUB_WORKSPACE}"/metplus/component_versions.py -v "${metplus_version}" -o MET -f "{X}.{Y}-latest" --no-get_dev_version)
 
 MET_DOCKER_REPO=met-dev
 if [ "${MET_TAG}" != "develop" ]; then

docs/Users_Guide/release-notes.rst

@@ -43,6 +43,63 @@ METplus Wrappers Release Notes
 When applicable, release notes are followed by the
 `GitHub issue <https://github.com/dtcenter/METplus/issues>`__ number which
 describes the bugfix, enhancement, or new feature.
+Important issues are listed **in bold** for emphasis.
+
+METplus Version 6.0.0 beta6 Release Notes (2024-10-18)
+------------------------------------------------------
+
+  .. dropdown:: Enhancement
+
+     * Support for setting Point2Grid MET config variables
+       (`#2540 <https://github.com/dtcenter/METplus/issues/2540>`_)
+     * Support processing groups of forecast leads
+       (`#2612 <https://github.com/dtcenter/METplus/issues/2612>`_)
+     * Support separate climatology datasets for both the forecast and observation inputs
+       (`#2622 <https://github.com/dtcenter/METplus/issues/2622>`_)
+     * Support the new `-aggr` command line option in SeriesAnalysis wrapper
+       (`#2651 <https://github.com/dtcenter/METplus/issues/2651>`_)
+     * **Deprecate master_metplus.py**
+       (`#2714 <https://github.com/dtcenter/METplus/issues/2714>`_)
+     * Support for setting point_weight_flag and obtype_as_group_val_flag in PointStat and EnsembleStat
+       (`#2727 <https://github.com/dtcenter/METplus/issues/2727>`_)
+
+  .. dropdown:: Bugfix
+
+     * ASCII2NC file window bad default value and redundant initialization of wrappers
+       (`#2520 <https://github.com/dtcenter/METplus/issues/2520>`_)
+     * Inconsistent RUN_ID values when using instances
+       (`#2596 <https://github.com/dtcenter/METplus/issues/2596>`_)
+     * Fix GridStat_SeriesAnalysis _fcstNMME_obsCPC _seasonal_forecast use cases with poorly configured climatology settings
+       (`#2695 <https://github.com/dtcenter/METplus/issues/2695>`_)
+     * Improve SeriesAnalysis field info generation with regards to time
+       (`#2705 <https://github.com/dtcenter/METplus/issues/2705>`_)
+     * Clean up existing use cases wrt SonarQube
+       (`#2710 <https://github.com/dtcenter/METplus/issues/2710>`_)
+
+  .. dropdown:: New Wrapper
+
+     NONE
+
+  .. dropdown:: New Use Case
+
+     * Multivariate MODE for RRFS
+       (`#2647 <https://github.com/dtcenter/METplus/issues/2647>`_)
+
+  .. dropdown:: Documentation
+
+     * Add information to Contributor's Guide for adding new use cases that utilize METplotpy/METcalcpy/METdataio
+       (`#1882 <https://github.com/dtcenter/METplus/issues/1882>`_)
+     * Update Release Guide for MET releases to update version numbers in the installation.rst in the MET User's Guide
+       (`#2452 <https://github.com/dtcenter/METplus/issues/2452>`_)
+     * Update Documentation Overview and Conventions
+       (`#2489 <https://github.com/dtcenter/METplus/issues/2489>`_)
+     * Update the User Support section in the Contributor's Guide
+       (`#2679 <https://github.com/dtcenter/METplus/issues/2679>`_)
+
+  .. dropdown:: Internal
+
+     NONE
+
 
 METplus Version 6.0.0 Beta 5 Release Notes (2024-07-10)
 -------------------------------------------------------

internal/scripts/dev_tools/generate_release_notes.py

@@ -0,0 +1,129 @@
+#! /usr/bin/env python3
+
+import sys
+import os
+
+from github import Github
+from datetime import datetime, timezone
+
+GITHUB_ORG = 'dtcenter'
+
+CATEGORIES = (
+    'Enhancement',
+    'Bugfix',
+    'New Wrapper',
+    'New Use Case',
+    'Documentation',
+    'Internal',
+)
+
+
+def main(dev_name, dev_start_date, dev_end_date=datetime.today(), repo_name='METplus'):
+    token = os.getenv('GITHUB_TOKEN')
+    if not token:
+      print("ERROR: Must set GITHUB_TOKEN environment variable")
+      sys.exit(1)
+
+    all_issues = get_all_issues_since_dev_start(token, repo_name, dev_start_date)
+    issues_by_category = get_issues_by_category(all_issues)
+
+    print_banner('ADD THIS TO docs/Users_Guide/release-notes.rst')
+
+    print_header(repo_name, dev_name, dev_end_date)
+    print_issues_by_category(repo_name, issues_by_category)
+
+    print_banner('ADD THIS TO METplus Coordinated Release Acceptance Testing')
+
+    print_release_testing(repo_name, dev_name, all_issues)
+
+
+def get_all_issues_since_dev_start(token, repo_name, dev_start_date):
+    print(f"Finding issues in {GITHUB_ORG}/{repo_name} that were closed after {dev_start_date.strftime('%Y-%m-%d')}...")
+    github_obj = Github(token)
+    org = github_obj.get_organization(GITHUB_ORG)
+    repo = org.get_repo(repo_name)
+
+    all_issues = repo.get_issues(state='closed', since=dev_start_date)
+    all_issues = [issue for issue in all_issues if issue.pull_request is None]
+    all_issues = [issue for issue in all_issues if not issue.title.startswith('Update Truth')]
+    all_issues.sort(key=lambda x: x.number)
+    return all_issues
+
+
+def print_banner(msg):
+    print(f"\n{'*' * len(msg)}\n{msg}\n{'*' * len(msg)}\n")
+
+
+def print_header(repo_name, dev_name, dev_end_date):
+    dev_fmt = dev_name.replace('-', '').replace('beta', 'Beta ').replace('rc', 'RC ')
+    header = f"{repo_name} Version {dev_fmt} Release Notes ({dev_end_date.strftime('%Y-%m-%d')})"
+    print(header)
+    print('-' * len(header))
+
+
+def print_issues_by_category(repo_name, issues_by_category):
+    for category, issues in issues_by_category.items():
+        print()
+        if category != 'none':
+            print(f"  .. dropdown:: {category}\n")
+        else:
+            print('COULD NOT PARSE CATEGORY FROM THESE:\n')
+        if issues is None:
+            print('     NONE')
+            continue
+        for issue in issues:
+            title = issue.title.removeprefix(category).lstrip(' :')
+            num = issue.number
+            print(f"     * {title}")
+            print(f"       (`#{num} <https://github.com/{GITHUB_ORG}/{repo_name}/issues/{num}>`_)")
+
+
+def get_issues_by_category(all_issues):
+    issues_by_category = dict.fromkeys(CATEGORIES)
+    issues_by_category['none'] = []
+    for issue in all_issues:
+        found_cat = False
+        for category in CATEGORIES:
+            if issue.title.startswith(f"{category}:"):
+                if issues_by_category[category] is None:
+                    issues_by_category[category] = []
+                issues_by_category[category].append(issue)
+                found_cat = True
+                break
+        if not found_cat:
+            if issues_by_category['none'] is None:
+                issues_by_category['none'] = []
+            issues_by_category['none'].append(issue)
+    return issues_by_category
+
+
+def print_release_testing(repo_name, dev_name, all_issues):
+    dev_info = dev_name.split('-')[1]
+    for issue in all_issues:
+        num = issue.number
+        print(f"| **OPEN** || [#{num}](https://github.com/{GITHUB_ORG}/{repo_name}/issues/{num}) | {dev_info} |||")
+
+
+if __name__ == '__main__':
+    import argparse
+    parser = argparse.ArgumentParser()
+    parser.add_argument('dev_name',
+                        help="name of development cycle, e.g. 6.0.0-beta6")
+    parser.add_argument('start_date',
+                        help="YYYYMMDD date when development cycle started")
+    parser.add_argument('-r', '--repo',
+                        default='METplus',
+                        help="Repository to parse, default is METplus")
+    args = parser.parse_args()
+
+    ymd = args.start_date
+    try:
+        year = int(ymd[:4])
+        month = int(ymd[4:6])
+        day = int(ymd[6:8])
+    except ValueError:
+        print('ERROR: Argument must be YYYYMMDD')
+        sys.exit(1)
+
+    start_date = datetime(year, month, day, tzinfo=timezone.utc)
+    main(repo_name=args.repo, dev_name=args.dev_name, dev_start_date=start_date)

internal/tests/pytests/component_versions/test_component_versions.py

@@ -49,3 +49,18 @@ def test_get_coordinated_version(component, version, expected_result):
 @pytest.mark.util
 def test_get_component_version(input_component, input_version, output_component, output_format, expected_result):
     assert component_versions.get_component_version(input_component, input_version, output_component, output_format) == expected_result
+
+
+@pytest.mark.parametrize(
+    'input_version, get_dev, expected_result', [
+        ('5.1.0', True, 'v5.1.0'),
+        ('5.1.0', False, 'v5.1.0'),
+        ('5.1.0-beta3', True, 'v5.1.0-beta3'),
+        ('5.1.0-beta3', False, 'develop'),
+        ('5.1.0-rc1', True, 'v5.1.0-rc1'),
+        ('5.1.0-rc1', False, 'develop'),
+    ]
+)
+@pytest.mark.util
+def test_get_component_version_get_dev(input_version, get_dev, expected_result):
+    assert component_versions.get_component_version('METplus', input_version, 'METplus', get_dev=get_dev) == expected_result

metplus/VERSION

@@ -1 +1 @@
-6.0.0-beta6-dev
+6.0.0-beta6

metplus/component_versions.py

@@ -37,7 +37,7 @@
 DEFAULT_OUTPUT_FORMAT = "v{X}.{Y}.{Z}{N}"
 
 def get_component_version(input_component, input_version, output_component,
-                          output_format=DEFAULT_OUTPUT_FORMAT):
+                          output_format=DEFAULT_OUTPUT_FORMAT, get_dev=True):
     """!Get the version of a requested METplus component given another METplus
     component and its version. Parses out X.Y version numbers of input version
     to find desired version. Optionally specific format of output content.
@@ -51,10 +51,13 @@ def get_component_version(input_component, input_version, output_component,
      {X}, {Y}, and {Z} will be replaced with x, y, and z version numbers from
      X.Y.Z. {N} will be replaced with development version if found in the
      input version, e.g. "-beta3" or "-rc1"
+     @param get_dev (optional) if True, get corresponding -beta or -rc version.
+     If False, return "develop" if input is beta or rc.
      @returns string of requested version number, or "develop" if input version
      ends with "-dev", or None if version number could not be determined.
     """
-    if input_version.endswith('-dev'):
+    if ('-dev' in input_version or
+            (not get_dev and any(ext in input_version for ext in ['-beta', '-rc']))):
         return 'develop'
     coord_version = get_coordinated_version(input_component, input_version)
     versions = VERSION_LOOKUP.get(coord_version)
@@ -93,10 +96,12 @@ def main():
     parser = argparse.ArgumentParser()
     parser.add_argument('-i', '--input_component',
                         default='metplus',
-                        help='Name of METplus component to use to find version')
+                        help='Name of METplus component to use to find version,'
+                        ' default is METplus.')
     parser.add_argument('-v', '--input_version',
                         default=next(iter(VERSION_LOOKUP)),
-                        help='version of input_component to search')
+                        help='version of input_component to search,'
+                             ' default is upcoming version')
     parser.add_argument('-o', '--output_component', required=True,
                         help='name of METplus component to obtain version')
     parser.add_argument('-f', '--output_format',
@@ -106,9 +111,14 @@ def main():
                              ' z version numbers from X.Y.Z. {N} will be '
                              'replaced with development version if found in the'
                              'input version, e.g. "-beta3" or "-rc1"')
+    parser.add_argument('--get_dev_version', action=argparse.BooleanOptionalAction,
+                        default=True,
+                        help='If True, get corresponding -beta or -rc version. '
+                             'If False, return develop if development version.')
     args = parser.parse_args()
     return get_component_version(args.input_component, args.input_version,
-                                 args.output_component, args.output_format)
+                                 args.output_component, args.output_format,
+                                 args.get_dev_version)
 
 
 if __name__ == "__main__":