2.SpeedingUpTheSlowOption #78

Ciheim · 2024-10-22T21:38:04Z

Improving the speed of the slow option by keep track of what we open

catalogbuilder/intakebuilder/gfdlcrawler.py

ceblanton · 2024-10-24T16:59:38Z

The dictionary's keys should be variable_id, not standard_name, I think. I think that could be the confusion here. e.g. this pseudocode

standard_names = {}

for file in $files
    variable_id = get_variable_id($file)
    check if standard_names[‘variable_id’] is set
     if yes, then use it
     if no,
            standard_name = get_standard_name($file)
            standard_names[variable_id] = standard_name

ceblanton · 2024-10-24T17:08:30Z

standard_names = {}

for file in $files
    variable_id = get_variable_id($file)
    realm = get_realm($file)
    check if standard_names[‘variable_id’][realm] is set
     if yes, then use it
     if no,
            standard_name = get_standard_name($file)
            standard_names[variable_id][realm] = standard_name

Ciheim · 2024-10-25T03:50:44Z

Currently testing this with a reduced dataset containing 3 realm types but many different var_ids. Script is working fine for this test but a more strenuous test case with proper standard names is needed.

Dictionary logic is working nicely and it doesn't seem that files are being unnecessarily opened anymore. I verified this in a few ways but mostly by forcibly stopping the script when a match was found in the unique_datasets dictionary.

Side note: It would be nice if we could find a creative way to identify whether or not a file will even have a real standard name before it is opened. This would stop us from opening a file just to get "NA" as standard name. May not be a possibility though...

aradhakrishnanGFDL · 2024-10-25T12:16:10Z

Currently testing this with a reduced dataset containing 3 realm types but many different var_ids. Script is working fine for this test but a more strenuous test case with proper standard names is needed.

Dictionary logic is working nicely and it doesn't seem that files are being unnecessarily opened anymore. I verified this in a few ways but mostly by forcibly stopping the script when a match was found in the unique_datasets dictionary.

Side note: It would be nice if we could find a creative way to identify whether or not a file will even have a real standard name before it is opened. This would stop us from opening a file just to get "NA" as standard name. May not be a possibility though...

Agreed, good point! Here is a possibility that MDTF will also appreciate so that xarray open call is still useful.

standard_name: (str) if a standard_name is not defined for a variable in the target file, use the equivalent CMIP6 standard_name or the long_name with underscores in place of spaces (e.g., air temperature -> air_temperature)

https://github.com/NOAA-GFDL/MDTF-diagnostics/blob/25671db7ddca8f51867ade76deb5cb206c09b4df/doc/sphinx/ref_catalogs.rst#L82

This means we will also grab the long_name (non-empty) when we open the file. We use long_name when standard name is empty, except replace the space to underscores.

E.g. in getinfo.py

https://github.com/NOAA-GFDL/CatalogBuilder/blob/main/catalogbuilder/intakebuilder/getinfo.py#L213

          cfname = filexr[variable_id].attrs["standard_name"]
      except KeyError:
          cfname = "NA"
          try:
              long_name = filexr[variable_id].attrs["long_name"]
          except KeyError:
              long_name = "NA"
          cfname = long_name.replace(" ", "_")

Note that we will be making up these standard names and they are not the official CF ones, but it serves the purpose here and for the broader analysis needs.

Ciheim · 2024-10-25T17:22:58Z

When timing the slow option on main and on this branch the difference is.... negligible. ~5 seconds on this branch and ~7 seconds on main. It would be interesting to see if the difference will be greater when testing with a larger dataset. Also, some code auditing is needed to ensure we're being efficient.

aradhakrishnanGFDL · 2024-10-25T18:43:08Z

When timing the slow option on main and on this branch the difference is.... negligible. ~5 seconds on this branch and ~7 seconds on main. It would be interesting to see if the difference will be greater when testing with a larger dataset. Also, some code auditing is needed to ensure we're being efficient.

I have a new test case for you. /archive/a1r/fre/FMS2024.02_OM5_20240724/testcase_1/gfdl.ncrc5-intel23-prod-openmp/pp/

pp/river mostly does not have standard name, so expect long name to be used.
pp/atmos_cmip mostly should have standard name
pp/ocean_monthly/av should not be processed at this time as its time-average

1- dmget all the data beforehand so we don't account for this in our timing tests.
2- You may be doing this already! This particular test case may take a while even though its a reduced test case, so using something like timeit python module to have a wrapper around your gen_intake_gfdl call.. or simply print time at the beginning and end within gen_intake_gfdl for now.

catalogbuilder/intakebuilder/gfdlcrawler.py

Ciheim · 2024-10-31T01:37:51Z

Test Instructions:

Create a new config file from template
Set everything in output path template to 'NA'
Set input path to /archive/a1r/fre/FMS2024.02_OM5_20240724/testcase_1/gfdl.ncrc5-intel23-prod-openmp/pp
Dmget all files
run gen_intake_gfdl.py --slow --config ../../configs/<name_of_config_file>

aradhakrishnanGFDL · 2024-10-31T13:26:30Z

Test Instructions:

Create a new config file from template

Set everything in output path template to 'NA'

Set input path to /archive/a1r/fre/FMS2024.02_OM5_20240724/testcase_1/gfdl.ncrc5-intel23-prod-openmp/pp

Dmget all files

run gen_intake_gfdl.py --slow --config ../../configs/<name_of_config_file>

@Ciheim should the PR be marked ready for review?

aradhakrishnanGFDL · 2024-10-31T13:27:11Z

Test Instructions:

Create a new config file from template

Set everything in output path template to 'NA'

Set input path to /archive/a1r/fre/FMS2024.02_OM5_20240724/testcase_1/gfdl.ncrc5-intel23-prod-openmp/pp

Dmget all files

run gen_intake_gfdl.py --slow --config ../../configs/<name_of_config_file>

can you paste the exact command you used to test including your template..

TODO added

ceblanton

The dictionary logic looks clear and correct to me, but I didn't actually run it. Nice work, Ciheim.

aradhakrishnanGFDL · 2024-10-31T18:04:17Z

There are a series of warnings raised. Do you see it in your test too @Ciheim. Can we find out why and document it in an issue. Also, add a line to not print those warnings unless --verbose is on.

example-

standard_name found northward_wind
INFO:local:Retrieving standard_name from atmos_cmip.000101-000512.ua_unmsk.nc
standard_name found eastward_wind
INFO:local:Retrieving standard_name from atmos_cmip.000101-000512.wap_unmsk.nc
standard_name found lagrangian_tendency_of_air_pressure
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.hfsnthermds.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found heat_flux_into_sea_water_due_to_snow_thermodynamics
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.col_height.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found The_height_of_the_water_column
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.mass_wt.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found The_column_mass_for_calculating_mass-weighted_average_properties
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.col_mass.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found The_column_integrated_in_situ_density
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.MLD_003_max.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found Mixed_layer_depth_(delta_rho_=_0.03)
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.ePBL_h_ML.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found Surface_boundary_layer_depth
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.MLD_003_min.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found Mixed_layer_depth_(delta_rho_=_0.03)
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.evs.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found water_evaporation_flux
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.MLD_003.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found Mixed_layer_depth_(delta_rho_=_0.03)
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.ficeberg.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found water_flux_into_sea_water_from_icebergs
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.MLD_EN1_max.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found Mixed_layer_depth_for_energy_value_set_to______25.00_J/m2_(Energy_set_by_1st_MLD_EN_VALS)
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.friver.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found water_flux_into_sea_water_from_rivers
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.MLD_EN1_min.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found Mixed_layer_depth_for_energy_value_set_to______25.00_J/m2_(Energy_set_by_1st_MLD_EN_VALS)
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.fsitherm.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)

aradhakrishnanGFDL · 2024-10-31T18:25:51Z

There are a series of warnings raised. Do you see it in your test too @Ciheim. Can we find out why and document it in an issue. Also, add a line to not print those warnings unless --verbose is on.

example-

standard_name found northward_wind
INFO:local:Retrieving standard_name from atmos_cmip.000101-000512.ua_unmsk.nc
standard_name found eastward_wind
INFO:local:Retrieving standard_name from atmos_cmip.000101-000512.wap_unmsk.nc
standard_name found lagrangian_tendency_of_air_pressure
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.hfsnthermds.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found heat_flux_into_sea_water_due_to_snow_thermodynamics
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.col_height.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found The_height_of_the_water_column
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.mass_wt.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found The_column_mass_for_calculating_mass-weighted_average_properties
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.col_mass.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found The_column_integrated_in_situ_density
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.MLD_003_max.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found Mixed_layer_depth_(delta_rho_=_0.03)
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.ePBL_h_ML.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found Surface_boundary_layer_depth
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.MLD_003_min.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found Mixed_layer_depth_(delta_rho_=_0.03)
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.evs.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found water_evaporation_flux
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.MLD_003.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found Mixed_layer_depth_(delta_rho_=_0.03)
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.ficeberg.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found water_flux_into_sea_water_from_icebergs
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.MLD_EN1_max.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found Mixed_layer_depth_for_energy_value_set_to______25.00_J/m2_(Energy_set_by_1st_MLD_EN_VALS)
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.friver.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found water_flux_into_sea_water_from_rivers
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.MLD_EN1_min.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
standard_name found Mixed_layer_depth_for_energy_value_set_to______25.00_J/m2_(Energy_set_by_1st_MLD_EN_VALS)
INFO:local:Retrieving standard_name from ocean_monthly.000101-000512.fsitherm.nc
/nbhome/Aparna.Radhakrishnan/conda/envs/catalogbuilder/lib/python3.12/site-packages/xarray/coding/times.py:992: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
  dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)

@Ciheim I've made some other minor changes to the PR branch. So pull new changes as needed. Once this reviewed and warnings are moved to logged.debug, this can be merged in. I did a simple test on a smaller test case since spear has some permission issues.
"/home/a1r/github/noaa-gfdl/catalogs/c96L65_am5f7b10r0_amip30_0806.json"

Ciheim · 2024-10-31T20:24:51Z

Config used for testing:

#what kind of directory structure to expect?
#For a directory structure like /archive/am5/am5/am5f3b1r0/c96L65_am5f3b1r0_pdclim1850F/gfdl.ncrc5-deploy-prod-openmp/pp

the output_path_template is set as follows.

#We have NA in those values that do not match up with any of the expected headerlist (CSV columns), otherwise we
#simply specify the associated header name in the appropriate place. E.g. The third directory in the PP path example
#above is the model (source_id), so the third list value in output_path_template is set to 'source_id'. We make sure
#this is a valid value in headerlist as well.
#The fourth directory is am5f3b1r0 which does not map to an existing header value. So we simply NA in output_path_template
#for the fourth value.

#catalog headers
#The headerlist is expected column names in your catalog/csv file. This is usually determined by the users in conjuction
#with the ESM collection specification standards and the appropriate workflows.

headerlist: ["activity_id", "institution_id", "source_id", "experiment_id",
"frequency", "realm", "table_id",
"member_id", "grid_label", "variable_id",
"time_range", "chunk_freq","platform","dimensions","cell_methods","standard_name","path"]

#what kind of directory structure to expect?
#For a directory structure like /archive/am5/am5/am5f3b1r0/c96L65_am5f3b1r0_pdclim1850F/gfdl.ncrc5-deploy-prod-openmp/pp

the output_path_template is set as follows.

#We have NA in those values that do not match up with any of the expected headerlist (CSV columns), otherwise we
#simply specify the associated header name in the appropriate place. E.g. The third directory in the PP path example
#above is the model (source_id), so the third list value in output_path_template is set to 'source_id'. We make sure
#this is a valid value in headerlist as well.
#The fourth directory is am5f3b1r0 which does not map to an existing header value. So we simply NA in output_path_template
#for the fourth value.

output_path_template: ['NA', 'NA', 'NA', 'NA', 'NA', 'NA', 'NA','NA','NA','NA','NA','NA','NA','NA','NA','NA','NA','NA']

output_file_template: ['realm','time_range','variable_id']

#OUTPUT FILE INFO is currently passed as command-line argument.
#We will revisit adding a csvfile, jsonfile and logfile configuration to the builder configuration file in the future.
#csvfile = #jsonfile = #logfile =

#######################################################

input_path: "/archive/a1r/fre/FMS2024.02_OM5_20240724/testcase_1/gfdl.ncrc5-intel23-prod-openmp/pp"
output_path: "/home/Ciheim.Brown/quick_test" # ENTER NAME OF THE CSV AND JSON, THE SUFFIX ALONE. e.g catalog (the builder then generates catalog.csv and catalog.json. This can also be an absolute path)

Ciheim · 2024-10-31T20:26:05Z

Not sure why it posted in a weird way..

2.SpeedingUpTheSlowOption

54794ad

aradhakrishnanGFDL reviewed Oct 23, 2024

View reviewed changes

catalogbuilder/intakebuilder/gfdlcrawler.py Outdated Show resolved Hide resolved

Fixing dictionary logic

4b5538b

Updating to grab long_name when standard_name is not found

8953b5e

aradhakrishnanGFDL reviewed Oct 28, 2024

View reviewed changes

catalogbuilder/intakebuilder/gfdlcrawler.py Show resolved Hide resolved

aradhakrishnanGFDL reviewed Oct 28, 2024

View reviewed changes

catalogbuilder/intakebuilder/gfdlcrawler.py Outdated Show resolved Hide resolved

aradhakrishnanGFDL reviewed Oct 28, 2024

View reviewed changes

catalogbuilder/intakebuilder/gfdlcrawler.py Show resolved Hide resolved

aradhakrishnanGFDL reviewed Oct 28, 2024

View reviewed changes

catalogbuilder/intakebuilder/gfdlcrawler.py Show resolved Hide resolved

Added TODO + changed print

a03d074

aradhakrishnanGFDL marked this pull request as ready for review October 31, 2024 16:47

aradhakrishnanGFDL added 2 commits October 31, 2024 12:49

Update catalogbuilder/intakebuilder/gfdlcrawler.py

dce18c1

TODO added

logger bugfix

622804e

ceblanton approved these changes Oct 31, 2024

View reviewed changes

aradhakrishnanGFDL added 2 commits October 31, 2024 14:15

help message added to --slow

6e2dd99

todo comments added per my review

9db4d7f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2.SpeedingUpTheSlowOption #78

2.SpeedingUpTheSlowOption #78

Ciheim commented Oct 22, 2024

ceblanton commented Oct 24, 2024

ceblanton commented Oct 24, 2024

Ciheim commented Oct 25, 2024

aradhakrishnanGFDL commented Oct 25, 2024

Ciheim commented Oct 25, 2024

aradhakrishnanGFDL commented Oct 25, 2024

Ciheim commented Oct 31, 2024

aradhakrishnanGFDL commented Oct 31, 2024

aradhakrishnanGFDL commented Oct 31, 2024

ceblanton left a comment

aradhakrishnanGFDL commented Oct 31, 2024

aradhakrishnanGFDL commented Oct 31, 2024

Ciheim commented Oct 31, 2024

Ciheim commented Oct 31, 2024

2.SpeedingUpTheSlowOption #78

Are you sure you want to change the base?

2.SpeedingUpTheSlowOption #78

Conversation

Ciheim commented Oct 22, 2024

ceblanton commented Oct 24, 2024

ceblanton commented Oct 24, 2024

Ciheim commented Oct 25, 2024

aradhakrishnanGFDL commented Oct 25, 2024

Ciheim commented Oct 25, 2024

aradhakrishnanGFDL commented Oct 25, 2024

Ciheim commented Oct 31, 2024

aradhakrishnanGFDL commented Oct 31, 2024

aradhakrishnanGFDL commented Oct 31, 2024

ceblanton left a comment

Choose a reason for hiding this comment

aradhakrishnanGFDL commented Oct 31, 2024

aradhakrishnanGFDL commented Oct 31, 2024

Ciheim commented Oct 31, 2024

the output_path_template is set as follows.

the output_path_template is set as follows.

Ciheim commented Oct 31, 2024