02 Mar 18:24

0ec340c

Version 2.5.3

The PIO team is pleased to release version 2.5.3 of the PIO library. This release includes enhanced netCDF integration, new API functions, documentation improvements, bug fixes, and more tests. Get the latest release from: https://github.com/NCAR/ParallelIO/releases/download/pio2_5_3/pio-2.5.3.tar.gz

See https://ncar.github.io/ParallelIO/ for documentation.

What is PIO?

The Parallel IO libraries (PIO) are high-level parallel I/O C and Fortran libraries for applications that need to do netCDF I/O from large numbers of processors on a HPC system.

PIO provides a netCDF-like API, and allows users to designate some subset of processors to perform IO. Computational code calls netCDF-like functions to read and write data, and PIO uses the IO processors to perform all necessary IO.

PIO also supports the creation of multiple computation components, each containing many processors, and one shared set of IO processors. The computational components can perform write operation asynchronously, and the IO processors will take care of all storage interaction.

See the PIO web site for more information: http://ncar.github.io/ParallelIO/.

Our PIO paper for the 2021 AMS meeting is here: https://www.researchgate.net/publication/348169990_THE_PARALLELIO_PIO_CFORTRAN_LIBRARIES_FOR_SCALABLE_HPC_PERFORMANCE.

New This Release

Fortran API for async interface (see #241).
Better async initialization (see #806).
Fixed memory errors in async mode (see #1846, #1847, #1267, #1830)
Removed unneeded dependency on netcdf-fortran (see #315)
Fixed build problems and code warnings with netCDF integration (see #1829, #1838, #1845)
More testing (see #1822, #1756, #1827, #1678, #1372).

Required Libraries

We recommend always using the latest releases of netCDF and pnetcdf.

This release requires:

netcdf-4.6.1 or newer. 4.7.3 or newer is required if --enable-netcdf-integration is used.
pnetcdf-1.9.0 or newer (unless PIO is built without pnetcdf support).
If netCDF is built with HDF5 support, then HDF5-1.10.4 or newer is required. Previous versions of HDF5 have a bug in parallel collective I/O which is triggered by a PIO unit test.

Assets 3

16 Sep 13:26

edwardhartnett

pio2_5_2

6f9fb5e

Version 2.5.2

The PIO team is pleased to release version 2.5.2 of the PIO library. This release includes enhanced netCDF integration, new API functions, documentation improvements, bug fixes, and more tests. Get the latest release from: https://github.com/NCAR/ParallelIO/releases/download/pio2_5_2/pio-2.5.2.tar.gz

See https://ncar.github.io/ParallelIO/ for documentation.

What is PIO?

The Parallel IO libraries (PIO) are high-level parallel I/O C and Fortran libraries for applications that need to do netCDF I/O from large numbers of processors on a HPC system.

See the PIO web site for more information: http://ncar.github.io/ParallelIO/

New This Release

Fix for bug appending to an existing file with PIO. (See #1670)
Testing on GitHub actions. (See #1675)
Inclusion of a build summary for autotools and cmake builds, and a pio_meta.h file which contains info about the PIO build. (See #1658)
Starting with netcdf-c-4.7.4 and HDF5-1.10.4, zlib compression may be used with the NETCDF4P IOTYPE (that is, with parallel I/O netCDF/HDF5 files). (See #1642).
Fixes for Fortran API functions PIO_def_var_chunking/PIO_inq_var_chunking. (See #1740 and #1739).
Clean up handling of pre-processor defines, especially in CMake build. (See #1721 and #1686).
Added --with-mpiexec option to configure (and WITH_MPIEXEC to CMake) to allow user to set mpiexec command used in testing. This is useful on openmpi, which requires the --oversubscribe argument. (See #1732)
We started using GitHub actions for testing, and testing with OpenMPI. (See #1675 and #906)
Improvements to the Fortran documentation. (See #1745).
Removed obsolete and no longer used bget memory optimization code. (See #1371).

Required Libraries

We recommend always using the latest releases of netCDF and pnetcdf.

This release requires:

netcdf-4.6.1 or newer. 4.7.3 or newer is required if --enable-netcdf-integration is used.
pnetcdf-1.9.0 or newer (unless PIO is built without pnetcdf support).
If netCDF is built with HDF5 support, then HDF5-1.10.4 or newer is required. Previous versions of HDF5 have a bug in parallel collective I/O which is triggered by a PIO unit test.

Assets 3

13 Jul 15:03

edwardhartnett

pio_2_5_1

aeefa82

Version 2.5.1

The PIO team is pleased to release version 2.5.1 of the PIO library. This release includes netCDF integration, new API functions, documentation improvements, bug fixes, and more tests. Get the latest release from: https://github.com/NCAR/ParallelIO/releases/download/pio_2_5_1/pio-2.5.1.tar.gz

See https://ncar.github.io/ParallelIO/ for documentation.

What is PIO?

The Parallel IO libraries (PIO) are high-level parallel I/O C and Fortran libraries for applications that need to do netCDF I/O from large numbers of processors on a HPC system.

See the PIO web site for more information: http://ncar.github.io/ParallelIO/

New This Release

Added PIO_SHORT data type to Fortran library.
Updated to match netCDF changes in netcdf-c-4.7.4.

Required Libraries

We recommend always using the latest releases of netCDF and pnetcdf.

This release requires:

netcdf-4.6.1 or newer. 4.7.3 is required if --enable-netcdf-integration is used.
pnetcdf-1.9.0 or newer (unless PIO is built without pnetcdf support).
If netCDF is built with HDF5 support, then HDF5-1.10.4 or newer is required. Previous versions of HDF5 have a bug in parallel collective I/O which is triggered by a PIO unit test.

Assets 3

27 Nov 22:33

edhartnett

pio2_5_0

751ff3d

Version 2.5.0

The PIO team is pleased to release version 2.5.0 of the PIO library. This release includes netCDF integration, new API functions, documentation improvements, bug fixes, and more tests. Get the latest release from: https://github.com/NCAR/ParallelIO/releases/download/pio2_5_0/pio-2.5.0.tar.gz

See https://ncar.github.io/ParallelIO/ for documentation.

What is PIO?

The Parallel IO libraries (PIO) are high-level parallel I/O C and Fortran libraries for applications that need to do netCDF I/O from large numbers of processors on a HPC system.

See the PIO web site for more information: http://ncar.github.io/ParallelIO/

New This Release

NetCDF Integration

When built with version 4.7.3 or later of netcdf-c, the PIO library now allows close integration with netCDF, allowing netCDF API code to be used instead of the PIOc_* functions. This allows legacy codes to be easily converted to use PIO. See #1555.

read_darray() Implemented for Async Mode

The PIOc_read_darray() function has been implemented in async mode. See #1601

Bug fixes

The header file pio.h used to use some ifdefs based on pre-processor macros defined in config.h, but it does not, and cannot include config.h. These definitions have been moved to pio_internal.h (see #1609

Required Libraries

We recommend always using the latest releases of netCDF and pnetcdf.

This release requires:

netcdf-4.6.1 or newer. 4.7.3 is required if --enable-netcdf-integration is used.
pnetcdf-1.9.0 or newer (unless PIO is built without pnetcdf support).
If netCDF is built with HDF5 support, then HDF5-1.10.4 or newer is required. Previous versions of HDF5 have a bug in parallel collective I/O which is triggered by a PIO unit test.

Assets 3

18 Aug 11:01

edhartnett

pio2_4_4

86ad446

Version 2.4.4

The PIO team is pleased to release version 2.4.4 of the PIO library. This release includes new API functions, documentation improvements, bug fixes, and more tests. Get the latest release from: https://github.com/NCAR/ParallelIO/releases/download/pio2_4_4/pio-2.4.4.tar.gz

See https://ncar.github.io/ParallelIO/ for documentation.

What is PIO?

The Parallel IO libraries (PIO) are high-level parallel I/O C and Fortran libraries for structured grid applications. PIO provides a netCDF-like API, and allows users to designate some subset of processors to perform IO. Computational code calls netCDF-like functions to read and write data, and PIO uses the IO processors to perform all necessary IO.

See the PIO web site for more information: http://ncar.github.io/ParallelIO/

New This Release

Name Change for Autotools-Built C Library

Previously, the C library was named libpioc.a in the Cmake build, and libpio.a in the autotools build. This has been corrected and now both builds use libpioc.a.

Support for MPE Library

PIO now supports the MPE library, when built with --enable-mpe.

Warning Fixes

The PIO C and Fortran builds now work without warnings for GNU compilers using -Wall.

Bug fixes

An integer overflow issue was found when writing large mpas datasets on summit.

A bug in the GPTL timer stop routine caused timers to fail and write a lot of messages to stderr.

A bug in pio_darray.c caused a segfault if using serial netcdf output when the array size on iotask 0 was smaller than that on other io tasks.

Required Libraries

We recommend always using the latest releases of netCDF and pnetcdf.

This release requires:

netcdf-4.6.1 or newer.
pnetcdf-1.9.0 or newer (unless PIO is built without pnetcdf support).
If netCDF is built with HDF5 support, then HDF5-1.10.4 or newer is required. Previous versions of HDF5 have a bug in parallel collective I/O which is triggered by a PIO unit test.

Assets 3

14 Jun 10:36

edhartnett

pio2_4_3

539b7ea

Version 2.4.3

The PIO team is pleased to release version 2.4.3 of the PIO library. This release includes new API functions, documentation improvements, bug fixes, and more tests. Get the latest release from: https://github.com/NCAR/ParallelIO/releases/download/pio2_4_3/pio-2.4.3.tar.gz

See https://ncar.github.io/ParallelIO/ for documentation.

What is PIO?

See the PIO web site for more information: http://ncar.github.io/ParallelIO/

New This Release

Added vard Functions for Distributed Array Access

To better match the look and feel of the netCDF API, we have added the following functions to do distributed array reads and writes. These functions are similar to a PIOc_setfreame() call, followed by a PIOc_read_darray() or PIOc_write_darray().

    /* Data reads - vard. */
    int PIOc_get_vard(int ncid, int varid, int decompid, const PIO_Offset recnum, void *buf);
    int PIOc_get_vard_text(int ncid, int varid, int decompid, const PIO_Offset recnum,
                           char *buf);
    int PIOc_get_vard_schar(int ncid, int varid, int decompid, const PIO_Offset recnum,
                            signed char *buf);
    int PIOc_get_vard_short(int ncid, int varid, int decompid, const PIO_Offset recnum,
                            short *buf);
    int PIOc_get_vard_int(int ncid, int varid, int decompid, const PIO_Offset recnum,
                          int *buf);
    int PIOc_get_vard_float(int ncid, int varid, int decompid, const PIO_Offset recnum,
                            float *buf);
    int PIOc_get_vard_double(int ncid, int varid, int decompid, const PIO_Offset recnum,
                             double *buf);
    int PIOc_get_vard_uchar(int ncid, int varid, int decompid, const PIO_Offset recnum,
                            unsigned char *buf);
    int PIOc_get_vard_ushort(int ncid, int varid, int decompid, const PIO_Offset recnum,
                             unsigned short *buf);
    int PIOc_get_vard_uint(int ncid, int varid, int decompid, const PIO_Offset recnum,
                           unsigned int *buf);
    int PIOc_get_vard_longlong(int ncid, int varid, int decompid, const PIO_Offset recnum,
                               long long *buf);
    int PIOc_get_vard_ulonglong(int ncid, int varid, int decompid, const PIO_Offset recnum,
                                unsigned long long *buf);

    /* Data writes - vard. */
    int PIOc_put_vard(int ncid, int varid, int decompid, const PIO_Offset recnum,
                      const void *buf);
    int PIOc_put_vard_text(int ncid, int varid, int decompid, const PIO_Offset recnum,
                           const char *op);
    int PIOc_put_vard_schar(int ncid, int varid, int decompid, const PIO_Offset recnum,
                            const signed char *op);
    int PIOc_put_vard_short(int ncid, int varid, int decompid, const PIO_Offset recnum,
                            const short *op);
    int PIOc_put_vard_int(int ncid, int varid, int decompid, const PIO_Offset recnum,
                          const int *op);
    int PIOc_put_vard_float(int ncid, int varid, int decompid, const PIO_Offset recnum,
                            const float *op);
    int PIOc_put_vard_double(int ncid, int varid, int decompid, const PIO_Offset recnum,
                             const double *op);
    int PIOc_put_vard_uchar(int ncid, int varid, int decompid, const PIO_Offset recnum,
                            const unsigned char *op);
    int PIOc_put_vard_ushort(int ncid, int varid, int decompid, const PIO_Offset recnum,
                             const unsigned short *op);
    int PIOc_put_vard_uint(int ncid, int varid, int decompid, const PIO_Offset recnum,
                           const unsigned int *op);
    int PIOc_put_vard_longlong(int ncid, int varid, int decompid, const PIO_Offset recnum,
                               const long long *op);
    int PIOc_put_vard_ulonglong(int ncid, int varid, int decompid, const PIO_Offset recnum,
                                const unsigned long long *op);

Improved Function Nomenclature

The PIOc_free_iosystem() function has been added as a duplicate of PIOc_finalize(), which has been deprecated and de-emphasized in the docs. (See #1463).

Changes in Memory Use

This release contains changes to cause many internal PIO maps to be stored on the heap, not the stack. This will prevent stack overflow. This work is not yet complete, a workaround is to increase the stack allocation when running PIO programs. (See #1341)

Documentation

Documentation has been updated. (See #1219).

Bug fixes

Passing in a negative component count to PIOc_init_async() causes a memory error, instead of returning an error code. This has been fixed. (See #1414).

More Tests

Travis build has been changed to require warning-free doxygen build for all PRs.
Changes have been made to performance tests tests/cunit/test_perf2.c and test_async_perf.c as part of an effort to benchmark PIO on several HPC systems.

Required Libraries

We recommend always using the latest releases of netCDF and pnetcdf.

This release requires:

netcdf-4.6.1 or newer.
pnetcdf-1.9.0 or newer (unless PIO is built without pnetcdf support).
If netCDF is built with HDF5 support, then HDF5-1.10.4 is required. Previous versions of HDF5 have a bug in parallel collective I/O which is triggered by a PIO unit test.

Assets 3

01 Jun 16:12

edhartnett

pio2_4_2

1ff5c4e

Version 2.4.2

The PIO team is pleased to release version 2.4.2 of the PIO library. This release includes performance improvements, bug fixes, and more tests. Get the latest release from the PIO GitHub site: https://github.com/NCAR/ParallelIO.

What is PIO?

The Parallel IO library (PIO) is a high-level parallel I/O C/Fortran library for structured grid applications. PIO provides a netCDF-like API, and allows users to designate some subset of processors to perform IO. Computational code calls netCDF-like functions to read and write data, and PIO uses the IO processors to perform all necessary IO.

See the PIO web site for more information: http://ncar.github.io/ParallelIO/

New This Release

Changes in Memory Use

This release contains changes to cause many internal PIO maps to be stored on the heap, not the stack. This will prevent stack overflow. This work is not yet complete, a workaround is to increase the stack allocation when running PIO programs.
This release changes the cmake build to use malloc instead of the bget code for memory allocation. The bget code was intended to improve performance but no longer seems to do so, and will be removed. (The autotools build always used malloc, never bget.)

Autotools Build System

The autotools build system can now build and test the Fortran library, with configure option --enable-fortran.
The autotools build system can now produce the doxygen-based documentation.

Bug fixes

In async mode, different compute groups were overwriting each others msg integer when sending messages. This has been corrected.
An error in computing the largest block size for data transfers caused communications to occur in smaller than optimal chunks. This has been corrected.
init_decomp for the box rearranger has been optimized by aggregating some small messages

More Tests

With this release our travis/GitHub continuous integration testing has been upgraded to include both C and Fortran libraries, with both autotools and cmake build systems. Travis builds are also done with the GNU memory sanitizer turned on, to find memory leaks.

Required Libraries

We recommend always using the latest releases of netCDF and pnetcdf.

This release requires:

netcdf-4.6.1 or newer.
pnetcdf-1.9.0 or newer (unless PIO is built without pnetcdf support).
If netCDF is built with HDF5 support, then HDF5-1.10.4 is required. Previous versions of HDF5 have a bug in parallel collective I/O which is triggered by a PIO unit test.

Assets 3

20 Mar 11:31

edwardhartnett

pio2_4_1

544f6c7

Version 2.4.1

The PIO team is pleased to release version 2.4.1 of the PIO library. This release includes performance improvements, bug fixes, and more tests. Get the latest release from the PIO GitHub site: https://github.com/NCAR/ParallelIO.

What is PIO?

See the PIO web site for more information: http://ncar.github.io/ParallelIO/

New This Release

Performance Improvements

This release contains the following performance improvements:

Linked list were replaced with hashtables for improved performance scaling.
The sorting algorithm for out of order dofs was replaced with qsort.

Bug fixes

Fixed over-sensitive testing of error messages in test_pioc. This tests have been made less sensitive so that they work with all recent versions of netCDF and parallel-netcdf.

More Tests

With this release we being using travis/GitHub for additional continuous integration testing, adding to our existing CI systems.

Required Libraries

We recommend always using the latest releases of netCDF and pnetcdf.

This release requires:

netcdf-4.6.0 or newer.
pnetcdf-1.9.0 or newer (unless PIO is built without pnetcdf support).
If netCDF is built with HDF5 support, then HDF5-1.10.4 is required. Previous versions of HDF5 have a bug in parallel collective I/O which is triggered by a PIO unit test.

Assets 3

11 Feb 12:54

edhartnett

pio2_4_0

6e3be58

Version 2.4.0

The PIO team is pleased to release version 2.4.0 of the PIO library. This release includes a build system improvements, performance improvements, bug fixes, and more tests. Get the latest release from the PIO GitHub site: https://github.com/NCAR/ParallelIO.

What is PIO?

See the PIO web site for more information: http://ncar.github.io/ParallelIO/

New This Release

Performance Improvements

This release contains the following performance improvements:

skip creating an MPI derived datatype if a request is contiguous

Autotools Build System Improvements

The autotools build system is independent from the CMake build system. It builds and tests the C library only. Fortran users should continue to use the CMake build system.

In this release the autotools build system was improved:

Now builds examples.

Bug fixes

Fixed fill value bug.
Fixed use of NULL in nc_put_vars_*() which causes segfaults with netCDF 4.6.2.

More Tests

This release includes more testing of:

Fill values.
Async darray functions.

Required Libraries

This release requires:

netcdf-4.5.0 or newer.
pnetcdf-1.9.0 or newer (unless PIO is built without pnetcdf support).
If netCDF is built with HDF5 support, then HDF5-1.10.4 is required. Previous versions of HDF5 have a bug in parallel collective I/O which is triggered by a PIO unit test.

Assets 3

03 Jan 16:27

edhartnett

pio2_3_1

77c4219

Version 2.3.1

The PIO team is pleased to release version 2.3.1 of the PIO library. This release includes a build system improvements, performance improvements, bug fixes, and more tests.

What is PIO?

See the PIO web site for more information: http://ncar.github.io/ParallelIO/

New This Release

Performance Improvements

PIO uses an internal buffer for data written/read through the darray functions. In this release, the size of this buffer was increased from 10 MB to 100 MB. The buffer size may now also be set by the user in either the CMake or autotools build with the PIO_BUFFER_SIZE parameter.

Autotools Build System Improvements

The autotools build system is independant from the CMake build system. It builds and tests the C library only. Fortran users should continue to use the CMake build system.

In this release the autotools build system was improved:

Supports setting the PIO_BUFFER_SIZE parameter.
Supports turning logging on/off (off by default).

Bug fixes

More reasonable maxblock value.
Limit to 2GiB due to romio bug.

More Tests

This release includes more testing of async functionality, including tests with more than one computation component.

Required Libraries

This release requires:

netcdf-4.4.1 or newer.
pnetcdf-1.8.1 or newer (unless PIO is built without pnetcdf support).

Assets 2

Releases: NCAR/ParallelIO

Version 2.5.3

What is PIO?

New This Release

Required Libraries

Version 2.5.2

What is PIO?

New This Release

Required Libraries

Version 2.5.1

What is PIO?

New This Release

Required Libraries

Version 2.5.0

What is PIO?

New This Release

NetCDF Integration

read_darray() Implemented for Async Mode

Bug fixes

Required Libraries

Version 2.4.4

What is PIO?

New This Release

Name Change for Autotools-Built C Library

Support for MPE Library

Warning Fixes

Bug fixes

Required Libraries

Version 2.4.3

What is PIO?

New This Release

Added vard Functions for Distributed Array Access

Improved Function Nomenclature

Changes in Memory Use

Documentation

Bug fixes

More Tests

Required Libraries

Version 2.4.2

What is PIO?

New This Release

Changes in Memory Use

Autotools Build System

Bug fixes

More Tests

Required Libraries

Version 2.4.1

What is PIO?

New This Release

Performance Improvements

Bug fixes

More Tests

Required Libraries

Version 2.4.0

What is PIO?

New This Release

Performance Improvements

Autotools Build System Improvements

Bug fixes

More Tests

Required Libraries

Version 2.3.1

What is PIO?

New This Release

Performance Improvements

Autotools Build System Improvements

Bug fixes

More Tests

Required Libraries