let `fre cmor run` process tripolar ocean data #303

ilaflott · 2024-12-24T16:06:33Z

this PR will ideally enable fre cmor run to process ocean data on a native tripolar grid. also will do some general clean up here and there. relevant background: #268 PCMDI/cmor#766 and PCMDI/cmor#767

Describe your changes

remove GitPython pip-only dependency that can only be fulfilled by terrible habits
add PCMDI/cmip6-cmor-tables as submodule dependency
set minimum python version to 3.9 minimum
remove redundant click context forwarding in fre cmor
remove redundant click context forwarding in most places, exception: fre catalog
re-set os.environ['HOME'] back to inital value in test_fre_make_cli.py
puts meta.yaml source: url: <this repo's URL> back to source: path: . in light of pipeline-error-induced insanity.
pylint edits to address scary-ish no-value-for-parameter errors revealed by adjusting click scaffolding
NotImplemented errors for fre functions that do not yet have any real guts to them in fre-cli
get rid of sys.exit where possible
removes "scratchwork" style fre cmor scripts used for by-hand local testing
add's gfdl/ppan-only tests in fre/cmor/tests/test_cmor_run_subtool_ppan_only.py that gracefully fail when not at pp/an with an appropriate warning
tweak pipeline comments
make pylint kinder to long lines (--max-line-length 100 --> 120)
~~remove unneeded __init__.py files in testing directories~~ ... added back in, affected coverage stats
last fre cmor run case of original 9 (Omon / sos / gn) case succeeds now

Issue ticket number and link (if applicable)

will close #298

Checklist before requesting a review

I ran my code
I tried to make my code readable
I tried to comment my code
I wrote a new test, if applicable
I wrote new instructions/documentation, if applicable
I ran pytest and inspected it's output
I ran pylint and attempted to implement some of it's feedback

…on pip-only dependency... nasty tech debt i should get rid of before its too late!

This reverts commit f59bd87.

ilaflott · 2024-12-24T17:42:15Z

so asking conda/mamba to use python3.10 or greater is surprisingly an issue. Dinking around with this locally, the first thing it complains about is cylc-uiserver. Given that this has yet-to-be-utilized in the actual repository, I comment it out.

The next issue is catalogbuilder... actually, really, a requirement of catalogbuilder's. intake-esm==2023.7.7, which seems to desire a python of 3.9 or less, but 3.9 is nearing EOL.

Given that this quick python version bump was supposed to be a quick-sidequest, i'll be restoring the dependencies and barking up that tree some other time.

…aving cmor>=3.9.0 because i know we should not bother with anything less than 3.9.0 for cmor

ilaflott · 2024-12-24T18:17:07Z

every now and then, a random conda build fails... here, it claims import git is causing the failure. I did remove that module, but i also removed that line

edit: same failure, same job. sanity demanding i ignore the conda_build workflow for a minute apparently

edit2: same failure, different job. xmas nightmare! moving on...

… with the zombie-problem in the github-repos conda build ci workflow

…singly pathological behavior. commit. wipe dir. re-clone and rebuild for my sanity's sake...

could it be?

…s pre- fre make cli tests.

….com/noaa-gfdl/fre-cli into 298-cmor-process-ocean-tripolar-grid

…tesy of Ray M., thank you ray.

…e list, fre pp, fre run, and fre test

This reverts commit 58c83c1.

… it was. catalogbuilder may need context forwarding.

…things

cmor_mixer now creates xh and yh bnds. this was not supposed to take so long... why? see note-to-self... ...note to self, if you need to write data to a netcdf file... don't use [x][y][z] style indexing. you HAVE to use [x,y,z] for the assignment or the assignment fails.

…pan-only* breadcrumb for now.

… push to make sure this xfails in the pipeline appropriately

…builtin. make max-line-length in pylint calls 120.

…e coveragerc to not ignore test directories- might be why the coverage results are wonky

…ching them... sheesh

…dule, for appropriate data collection. adjust paths to globs in coveragerc

ceblanton

It looks good but admittedly I did not try it and review hard. In the future we will do more rigorous reviewing, promise!

add in cmip6-cmor-tables as a submodule, remove un-maintained gitpyth…

6dce8ef

…on pip-only dependency... nasty tech debt i should get rid of before its too late!

ilaflott added bug Something isn't working new functioning New feature or request tricky Likely to encounter significant friction to solution priority: HIGH labels Dec 24, 2024

ilaflott self-assigned this Dec 24, 2024

ilaflott linked an issue Dec 24, 2024 that may be closed by this pull request

CMORmeal: successfully process ocean variables on tripolar grid with cmor #298

Closed

9 tasks

ilaflott added 3 commits December 24, 2024 11:32

i guess .gitmodules isnt meant to be edited by hand!

f59bd87

Revert "i guess .gitmodules isnt meant to be edited by hand!"

1168d5d

This reverts commit f59bd87.

actually add the submodule...

a197538

ilaflott added 2 commits December 24, 2024 12:47

restore deps with the exception of the pip-installed gitpython and le…

ff8d8ce

…aving cmor>=3.9.0 because i know we should not bother with anything less than 3.9.0 for cmor

syntax error

32abb9b

ilaflott and others added 5 commits December 24, 2024 13:38

whitespace-cleanup some files for lint complaints, poke the test file…

ac61693

… with the zombie-problem in the github-repos conda build ci workflow

add pyproject.toml to suppress pip warning.

91a0fbc

Merge branch 'main' into 298-cmor-process-ocean-tripolar-grid

176ea9d

trying to get a fre make test's output out of the way, getting surpri…

cec2a12

…singly pathological behavior. commit. wipe dir. re-clone and rebuild for my sanity's sake...

Update meta.yaml

e5daf46

could it be?

ilaflott mentioned this pull request Jan 3, 2025

change conda build source back to path: . from url: <repo url> #310

Merged

4 tasks

ilaflott and others added 10 commits January 6, 2025 10:48

Merge branch 'main' into 298-cmor-process-ocean-tripolar-grid

0c0b45d

remove a sys.exit or two. restore os.environ[HOME] back to what it wa…

fe0a656

…s pre- fre make cli tests.

Merge branch '298-cmor-process-ocean-tripolar-grid' of https://github…

9122ef3

….com/noaa-gfdl/fre-cli into 298-cmor-process-ocean-tripolar-grid

oops

3260539

update click scaffolding to be much more lean and mean. approach cour…

50498e2

…tesy of Ray M., thank you ray.

removing now-redundant click-scaffoldding from fre app, fre check, fr…

c92e4e8

…e list, fre pp, fre run, and fre test

fix an oops, apply same approach to fre/catalog

58c83c1

apply click-redundancy-removal logic to fre make

3e0096e

Revert "fix an oops, apply same approach to fre/catalog"

19bc7a1

This reverts commit 58c83c1.

remove needless shebang, put fre catalog click scaffolding back where…

aef76ee

… it was. catalogbuilder may need context forwarding.

ilaflott added 7 commits January 6, 2025 18:15

remove useless __init__ files in test directories. more pylint edits.

a074922

why last failure? unconvinced. fail again, conda_build, i dare you.

fb62498

adjust import statement in fre/cmor/tests

21e22b4

carefully tidy print statements, carefully changing order of certain …

54e02e3

…things

SO CLOSE

5911351

pretty sure i got it!

dfe9ccb

ilaflott marked this pull request as ready for review January 9, 2025 23:27

ilaflott mentioned this pull request Jan 9, 2025

CMORmeal: successfully process ocean variables on tripolar grid with cmor #298

Closed

9 tasks

ilaflott added 15 commits January 9, 2025 18:35

remove scratchwork script.

6d788b1

Merge branch 'main' into 298-cmor-process-ocean-tripolar-grid

8a3816e

lets just make run_test_file_cases a bunch of xfail pytests with a *p…

baf61dc

…pan-only* breadcrumb for now.

Merge branch 'main' into 298-cmor-process-ocean-tripolar-grid

8a71625

ok got a format and a conditional pytest.xfail approach i am OK with.…

da8b4d3

… push to make sure this xfails in the pipeline appropriately

cleaner tests, all should xfail appropriately in the CI/CD

5d7f77d

pylint things. rename cmor_lister to cmor_finder to avoid clash with …

1c987da

…builtin. make max-line-length in pylint calls 120.

fix an oops in the fre cmor cli tests. tweak pytest.ini trivially

aa6c612

update workflow file to upload the coverage xml for inspection. chang…

fd73c43

…e coveragerc to not ignore test directories- might be why the coverage results are wonky

why no coverage badge?? smh..

c428716

put coveragerc and pytest.ini back to what they were... sorry for tou…

dd21a0c

…ching them... sheesh

add */tests/__init__.py files back in... blegh

2d6730e

another testing init file

be4af5c

adjust pytest calls in workflows to use coverage to call pytest as mo…

1923e6d

…dule, for appropriate data collection. adjust paths to globs in coveragerc

whitespace newlines and fstrings thanks pylint you great friend

41a99f2

ilaflott requested review from singhd789, ceblanton and Ciheim January 13, 2025 22:01

ceblanton approved these changes Jan 16, 2025

View reviewed changes

ceblanton merged commit 3453c6b into main Jan 16, 2025
2 checks passed

ceblanton deleted the 298-cmor-process-ocean-tripolar-grid branch January 16, 2025 17:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

let `fre cmor run` process tripolar ocean data #303

let `fre cmor run` process tripolar ocean data #303

ilaflott commented Dec 24, 2024 •

edited

Loading

ilaflott commented Dec 24, 2024

ilaflott commented Dec 24, 2024 •

edited

Loading

ceblanton left a comment

let fre cmor run process tripolar ocean data #303

let fre cmor run process tripolar ocean data #303

Conversation

ilaflott commented Dec 24, 2024 • edited Loading

Describe your changes

Issue ticket number and link (if applicable)

Checklist before requesting a review

ilaflott commented Dec 24, 2024

ilaflott commented Dec 24, 2024 • edited Loading

ceblanton left a comment

Choose a reason for hiding this comment

let `fre cmor run` process tripolar ocean data #303

let `fre cmor run` process tripolar ocean data #303

ilaflott commented Dec 24, 2024 •

edited

Loading

ilaflott commented Dec 24, 2024 •

edited

Loading