Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature #2701 ismn #2758

Merged
merged 14 commits into from
Jan 16, 2024
Merged

Feature #2701 ismn #2758

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
a1fe8c1
Add support for ISMN soil moisture data
JohnHalleyGotway Dec 1, 2023
0ac9f6f
Per #2701, update ascii2nc.cc with the ismn file option.
JohnHalleyGotway Dec 1, 2023
77355a2
Per #2701, update ascii2nc docs
JohnHalleyGotway Dec 1, 2023
c58e803
Per #2701, store the depth for precip as 0.
JohnHalleyGotway Dec 1, 2023
58cea27
Per #2701, parse the ISMN observation value from the correct column.
JohnHalleyGotway Dec 5, 2023
9107fc6
Per #2701, every time I run ascii2nc to I see a log message for 'DEBU…
JohnHalleyGotway Dec 5, 2023
6bf2fb6
Per #2701, as instructed by @anewman89, store the average of the dept…
JohnHalleyGotway Dec 6, 2023
f14d334
Per #2701, add a test of processing the ISMN data through ascii2nc.
JohnHalleyGotway Dec 6, 2023
9aa7723
Per #2701, fix parsing of year and month to subtract 1900 and 1, resp…
JohnHalleyGotway Dec 6, 2023
d4c5690
Per #2701, set use_var_id to true for ISMN inputs
JohnHalleyGotway Dec 6, 2023
9326888
Per #2701, even though we're mapping obs data to GRIB code names and …
JohnHalleyGotway Dec 6, 2023
fdb58db
Merge remote-tracking branch 'origin/develop' into feature_2701_ismn
Dec 13, 2023
6b1d5e7
Merge remote-tracking branch 'origin/develop' into feature_2701_ismn
Dec 13, 2023
ed07b27
Per #2701, fix typo in the name of the ascii2nc netcdf output file.
Dec 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/Users_Guide/reformat_point.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -456,6 +456,8 @@ While initial versions of the ASCII2NC tool only supported a simple 11 column AS

• `National Data Buoy (NDBC) Standard Meteorlogical Data format <https://www.ndbc.noaa.gov/measdes.shtml>`_. See the :ref:`MET_NDBC_STATIONS` environment variable.

• `International Soil Moisture Network (ISMN) Data format <https://ismn.bafg.de/en/>`_.

• `AErosol RObotic NEtwork (AERONET) versions 2 and 3 format <http://aeronet.gsfc.nasa.gov/>`_

• Python embedding of point observations, as described in :numref:`pyembed-point-obs-data`. See example below in :numref:`ascii2nc-pyembed`.
Expand Down Expand Up @@ -539,7 +541,7 @@ Required Arguments for ascii2nc
Optional Arguments for ascii2nc
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

3. The **-format ASCII_format** option may be set to "met_point", "little_r", "surfrad", "wwsis", "airnowhourlyaqobs", "airnowhourly", "airnowdaily_v2", "ndbc_standard", "aeronet", "aeronetv2", "aeronetv3", or "python". If passing in ISIS data, use the "surfrad" format flag.
3. The **-format ASCII_format** option may be set to "met_point", "little_r", "surfrad", "wwsis", "airnowhourlyaqobs", "airnowhourly", "airnowdaily_v2", "ndbc_standard", "ismn", "aeronet", "aeronetv2", "aeronetv3", or "python". If passing in ISIS data, use the "surfrad" format flag.

4. The **-config file** option is the configuration file for generating time summaries.

Expand Down
13 changes: 11 additions & 2 deletions internal/test_unit/xml/unit_ascii2nc.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,16 @@
</output>
</test>



<test name="ascii2nc_ismn_SNOTEL">
<exec>&MET_BIN;/ascii2nc</exec>
<param> \
-format ismn \
&DATA_DIR_OBS;/ismn/SNOTEL/*/*.stm \
&OUTPUT_DIR;/ascii2nc/ismn_SNOTEL_20220924_20220927.nc
</param>
<output>
<point_nc>&OUTPUT_DIR;/ascii2nc/ismn_SNOTEL_20220924_20220927.nc</point_nc>
</output>
</test>

</met_test>
4 changes: 3 additions & 1 deletion src/tools/other/ascii2nc/Makefile.am
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ ascii2nc_SOURCES = ascii2nc.cc \
ndbc_handler.cc ndbc_handler.h \
ndbc_locations.cc ndbc_locations.h \
airnow_locations.cc airnow_locations.h \
aeronet_handler.cc aeronet_handler.h $(OPT_PYTHON_SOURCES)
aeronet_handler.cc aeronet_handler.h \
ismn_handler.cc ismn_handler.h \
$(OPT_PYTHON_SOURCES)

ascii2nc_CPPFLAGS = ${MET_CPPFLAGS} -I../../../basic/vx_log
ascii2nc_LDFLAGS = ${MET_LDFLAGS}
Expand Down
28 changes: 25 additions & 3 deletions src/tools/other/ascii2nc/Makefile.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,8 @@ am__ascii2nc_SOURCES_DIST = ascii2nc.cc ascii2nc_conf_info.cc \
airnow_handler.h ndbc_handler.cc ndbc_handler.h \
ndbc_locations.cc ndbc_locations.h airnow_locations.cc \
airnow_locations.h aeronet_handler.cc aeronet_handler.h \
python_handler.h python_handler.cc
ismn_handler.cc ismn_handler.h python_handler.h \
python_handler.cc
@ENABLE_PYTHON_TRUE@am__objects_1 = ascii2nc-python_handler.$(OBJEXT)
am__objects_2 = $(am__objects_1)
am_ascii2nc_OBJECTS = ascii2nc-ascii2nc.$(OBJEXT) \
Expand All @@ -124,7 +125,8 @@ am_ascii2nc_OBJECTS = ascii2nc-ascii2nc.$(OBJEXT) \
ascii2nc-ndbc_handler.$(OBJEXT) \
ascii2nc-ndbc_locations.$(OBJEXT) \
ascii2nc-airnow_locations.$(OBJEXT) \
ascii2nc-aeronet_handler.$(OBJEXT) $(am__objects_2)
ascii2nc-aeronet_handler.$(OBJEXT) \
ascii2nc-ismn_handler.$(OBJEXT) $(am__objects_2)
ascii2nc_OBJECTS = $(am_ascii2nc_OBJECTS)
am__DEPENDENCIES_1 =
ascii2nc_DEPENDENCIES = $(am__DEPENDENCIES_1) $(am__DEPENDENCIES_1) \
Expand Down Expand Up @@ -153,6 +155,7 @@ am__depfiles_remade = ./$(DEPDIR)/ascii2nc-aeronet_handler.Po \
./$(DEPDIR)/ascii2nc-ascii2nc.Po \
./$(DEPDIR)/ascii2nc-ascii2nc_conf_info.Po \
./$(DEPDIR)/ascii2nc-file_handler.Po \
./$(DEPDIR)/ascii2nc-ismn_handler.Po \
./$(DEPDIR)/ascii2nc-little_r_handler.Po \
./$(DEPDIR)/ascii2nc-met_handler.Po \
./$(DEPDIR)/ascii2nc-ndbc_handler.Po \
Expand Down Expand Up @@ -387,7 +390,9 @@ ascii2nc_SOURCES = ascii2nc.cc \
ndbc_handler.cc ndbc_handler.h \
ndbc_locations.cc ndbc_locations.h \
airnow_locations.cc airnow_locations.h \
aeronet_handler.cc aeronet_handler.h $(OPT_PYTHON_SOURCES)
aeronet_handler.cc aeronet_handler.h \
ismn_handler.cc ismn_handler.h \
$(OPT_PYTHON_SOURCES)

ascii2nc_CPPFLAGS = ${MET_CPPFLAGS} -I../../../basic/vx_log
ascii2nc_LDFLAGS = ${MET_LDFLAGS}
Expand Down Expand Up @@ -514,6 +519,7 @@ distclean-compile:
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ascii2nc-ascii2nc.Po@am__quote@ # am--include-marker
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ascii2nc-ascii2nc_conf_info.Po@am__quote@ # am--include-marker
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ascii2nc-file_handler.Po@am__quote@ # am--include-marker
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ascii2nc-ismn_handler.Po@am__quote@ # am--include-marker
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ascii2nc-little_r_handler.Po@am__quote@ # am--include-marker
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ascii2nc-met_handler.Po@am__quote@ # am--include-marker
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ascii2nc-ndbc_handler.Po@am__quote@ # am--include-marker
Expand Down Expand Up @@ -710,6 +716,20 @@ ascii2nc-aeronet_handler.obj: aeronet_handler.cc
@AMDEP_TRUE@@am__fastdepCXX_FALSE@ DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@
@am__fastdepCXX_FALSE@ $(AM_V_CXX@am__nodep@)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(ascii2nc_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -c -o ascii2nc-aeronet_handler.obj `if test -f 'aeronet_handler.cc'; then $(CYGPATH_W) 'aeronet_handler.cc'; else $(CYGPATH_W) '$(srcdir)/aeronet_handler.cc'; fi`

ascii2nc-ismn_handler.o: ismn_handler.cc
@am__fastdepCXX_TRUE@ $(AM_V_CXX)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(ascii2nc_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -MT ascii2nc-ismn_handler.o -MD -MP -MF $(DEPDIR)/ascii2nc-ismn_handler.Tpo -c -o ascii2nc-ismn_handler.o `test -f 'ismn_handler.cc' || echo '$(srcdir)/'`ismn_handler.cc
@am__fastdepCXX_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/ascii2nc-ismn_handler.Tpo $(DEPDIR)/ascii2nc-ismn_handler.Po
@AMDEP_TRUE@@am__fastdepCXX_FALSE@ $(AM_V_CXX)source='ismn_handler.cc' object='ascii2nc-ismn_handler.o' libtool=no @AMDEPBACKSLASH@
@AMDEP_TRUE@@am__fastdepCXX_FALSE@ DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@
@am__fastdepCXX_FALSE@ $(AM_V_CXX@am__nodep@)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(ascii2nc_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -c -o ascii2nc-ismn_handler.o `test -f 'ismn_handler.cc' || echo '$(srcdir)/'`ismn_handler.cc

ascii2nc-ismn_handler.obj: ismn_handler.cc
@am__fastdepCXX_TRUE@ $(AM_V_CXX)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(ascii2nc_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -MT ascii2nc-ismn_handler.obj -MD -MP -MF $(DEPDIR)/ascii2nc-ismn_handler.Tpo -c -o ascii2nc-ismn_handler.obj `if test -f 'ismn_handler.cc'; then $(CYGPATH_W) 'ismn_handler.cc'; else $(CYGPATH_W) '$(srcdir)/ismn_handler.cc'; fi`
@am__fastdepCXX_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/ascii2nc-ismn_handler.Tpo $(DEPDIR)/ascii2nc-ismn_handler.Po
@AMDEP_TRUE@@am__fastdepCXX_FALSE@ $(AM_V_CXX)source='ismn_handler.cc' object='ascii2nc-ismn_handler.obj' libtool=no @AMDEPBACKSLASH@
@AMDEP_TRUE@@am__fastdepCXX_FALSE@ DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) @AMDEPBACKSLASH@
@am__fastdepCXX_FALSE@ $(AM_V_CXX@am__nodep@)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(ascii2nc_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -c -o ascii2nc-ismn_handler.obj `if test -f 'ismn_handler.cc'; then $(CYGPATH_W) 'ismn_handler.cc'; else $(CYGPATH_W) '$(srcdir)/ismn_handler.cc'; fi`

ascii2nc-python_handler.o: python_handler.cc
@am__fastdepCXX_TRUE@ $(AM_V_CXX)$(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(ascii2nc_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) -MT ascii2nc-python_handler.o -MD -MP -MF $(DEPDIR)/ascii2nc-python_handler.Tpo -c -o ascii2nc-python_handler.o `test -f 'python_handler.cc' || echo '$(srcdir)/'`python_handler.cc
@am__fastdepCXX_TRUE@ $(AM_V_at)$(am__mv) $(DEPDIR)/ascii2nc-python_handler.Tpo $(DEPDIR)/ascii2nc-python_handler.Po
Expand Down Expand Up @@ -857,6 +877,7 @@ distclean: distclean-am
-rm -f ./$(DEPDIR)/ascii2nc-ascii2nc.Po
-rm -f ./$(DEPDIR)/ascii2nc-ascii2nc_conf_info.Po
-rm -f ./$(DEPDIR)/ascii2nc-file_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-ismn_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-little_r_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-met_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-ndbc_handler.Po
Expand Down Expand Up @@ -915,6 +936,7 @@ maintainer-clean: maintainer-clean-am
-rm -f ./$(DEPDIR)/ascii2nc-ascii2nc.Po
-rm -f ./$(DEPDIR)/ascii2nc-ascii2nc_conf_info.Po
-rm -f ./$(DEPDIR)/ascii2nc-file_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-ismn_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-little_r_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-met_handler.Po
-rm -f ./$(DEPDIR)/ascii2nc-ndbc_handler.Po
Expand Down
6 changes: 3 additions & 3 deletions src/tools/other/ascii2nc/aeronet_handler.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
////////////////////////////////////////////////////////////////////////


#ifndef __AERONETHANDLER_H__
#define __AERONETHANDLER_H__
#ifndef __AERONET_HANDLER_H__
#define __AERONET_HANDLER_H__


////////////////////////////////////////////////////////////////////////
Expand Down Expand Up @@ -125,7 +125,7 @@ class AeronetHandler : public FileHandler
////////////////////////////////////////////////////////////////////////


#endif /* __AERONETHANDLER_H__ */
#endif /* __AERONET_HANDLER_H__ */


////////////////////////////////////////////////////////////////////////
Expand Down
2 changes: 1 addition & 1 deletion src/tools/other/ascii2nc/airnow_handler.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -648,7 +648,7 @@ time_t AirnowHandler::_getValidTime(const string &dateStr, const string &timeStr
struct tm time_struct;
memset(&time_struct, 0, sizeof(time_struct));

time_struct.tm_year = atoi(year.c_str()) -1900;
time_struct.tm_year = atoi(year.c_str()) - 1900;
time_struct.tm_mon = atoi(mon.c_str()) - 1;
time_struct.tm_mday = atoi(mday.c_str());
time_struct.tm_hour = atoi(hour.c_str());
Expand Down
6 changes: 3 additions & 3 deletions src/tools/other/ascii2nc/airnow_handler.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
////////////////////////////////////////////////////////////////////////


#ifndef __AIRNOWHANDLER_H__
#define __AIRNOWHANDLER_H__
#ifndef __AIRNOW_HANDLER_H__
#define __AIRNOW_HANDLER_H__


////////////////////////////////////////////////////////////////////////
Expand Down Expand Up @@ -198,7 +198,7 @@ class AirnowHandler : public FileHandler
////////////////////////////////////////////////////////////////////////


#endif /* __AERONETHANDLER_H__ */
#endif /* __AERONET_HANDLER_H__ */


////////////////////////////////////////////////////////////////////////
Expand Down
27 changes: 25 additions & 2 deletions src/tools/other/ascii2nc/ascii2nc.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,8 @@
// 019 07/06/22 Howard Soh METplus-Internal #19 Rename main to met_main
// 020 08/26/22 Dave Albo MET #2142 Add AirNow observations
// 021 10/03/22 Prestopnik MET #2227 Remove using namespace std from header files
// 022 10/07/22 Dave Albo MET #2276 Add NDBC Buoy data
// 022 10/07/22 Dave Albo MET #2276 Add NDBC buoy data
// 023 11/28/23 Halley Gotway MET #2701 Add ISMN soil moisture data
//
////////////////////////////////////////////////////////////////////////

Expand Down Expand Up @@ -86,6 +87,7 @@ using namespace std;
#include "aeronet_handler.h"
#include "airnow_handler.h"
#include "ndbc_handler.h"
#include "ismn_handler.h"

#ifdef ENABLE_PYTHON
#include "global_python.h"
Expand Down Expand Up @@ -113,6 +115,7 @@ enum ASCIIFormat {
ASCIIFormat_Airnow_hourlyaqobs,
ASCIIFormat_Airnow_hourly,
ASCIIFormat_NDBC_standard,
ASCIIFormat_ISMN,
ASCIIFormat_Aeronet_v2,
ASCIIFormat_Aeronet_v3,
ASCIIFormat_Python,
Expand Down Expand Up @@ -322,6 +325,10 @@ FileHandler *create_file_handler(const ASCIIFormat format, const ConcatString &a
return((FileHandler *) handler);
}

case ASCIIFormat_ISMN: {
return((FileHandler *) new IsmnHandler(program_name));
}

case ASCIIFormat_Aeronet_v2: {
AeronetHandler *handler = new AeronetHandler(program_name);
handler->setFormatVersion(2);
Expand Down Expand Up @@ -459,9 +466,21 @@ FileHandler *determine_ascii_format(const ConcatString &ascii_filename) {
delete ndbc_file;

//
// If we get here, we didn't recognize the file contents.
// See if this is an ISMN file.
//
f_in.rewind();
IsmnHandler *ismn_file = new IsmnHandler(program_name);

if(ismn_file->isFileType(f_in)) {
f_in.close();
return((FileHandler *) ismn_file);
}

delete ismn_file;

//
// If we get here, we didn't recognize the file contents.
//
mlog << Error << "\ndetermine_ascii_format() -> "
<< "could not determine file format based on file contents\n\n";

Expand Down Expand Up @@ -503,6 +522,7 @@ void usage() {
<< AirnowHandler::getFormatStringHourlyAqObs() << "\", \""
<< AirnowHandler::getFormatStringHourly() << "\", \""
<< NdbcHandler::getFormatStringStandard() << "\", \""
<< IsmnHandler::getFormatString() << "\", \""
<< AeronetHandler::getFormatString() << "\", \""
<< AeronetHandler::getFormatString_v2() << "\", \""
<< AeronetHandler::getFormatString_v3() << "\"";
Expand Down Expand Up @@ -585,6 +605,9 @@ void set_format(const StringArray & a) {
else if(NdbcHandler::getFormatStringStandard() == a[0]) {
ascii_format = ASCIIFormat_NDBC_standard;
}
else if(IsmnHandler::getFormatString() == a[0]) {
ascii_format = ASCIIFormat_ISMN;
}
else if(AeronetHandler::getFormatString() == a[0]
|| AeronetHandler::getFormatString_v2() == a[0]) {
ascii_format = ASCIIFormat_Aeronet_v2;
Expand Down
6 changes: 3 additions & 3 deletions src/tools/other/ascii2nc/file_handler.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
////////////////////////////////////////////////////////////////////////


#ifndef __FILEHANDLER_H__
#define __FILEHANDLER_H__
#ifndef __FILE_HANDLER_H__
#define __FILE_HANDLER_H__


////////////////////////////////////////////////////////////////////////
Expand Down Expand Up @@ -158,7 +158,7 @@ inline void FileHandler::setMessageTypeMap(map<ConcatString, ConcatString> m) {
////////////////////////////////////////////////////////////////////////


#endif /* __FILEHANDLER_H__ */
#endif /* __FILE_HANDLER_H__ */


////////////////////////////////////////////////////////////////////////
Expand Down
Loading
Loading