Refactor H5FD and package initialization (#4934)

- Reverts PR#1024, which (unnecessarily) switched from deferred
  package initialization to centralized initialization of all packages
  and introduced H5FDperform_init() to wrap an internal routine
  to initialize VFD plugins.
- Went back to deferred package initialization (primarily), to
  eliminate unnecessary resource use. (Performance has been
  verified to be the same either way)
- Switched VFD plugins to use “#define (H5OPEN, )” pattern,
  with registration of internal VFD plugins at library initialization
  time. Eliminates calling API routine (H5FDperform_init) from
  within the library, which was deadlocking threadsafe concurrency.
  And also eliminates exposing internal library routines in a public
  header file.
- Removed copy-and-paste replicas of the H5OPEN macro and put
  a (better) single definition in H5public.h
- Separated API and internal routine calls in stdio and multi VFD
  plugins into separate source files, so that the library doesn’t
  invoke API routines internally (also a deadlock problem for
  threadsafe concurrency). Also needed a “private” header for these
  plugins.
- Separated registering/unregistering a VFD plugin from initializing
  /finalizing the plugin, instead of blurring those ideas together.
  Defers the VFD plugin init to when it’s actually used, which reduces
  resource usage, especially for the MPI-based plugins like the subfiling,
  etc.
- Refactored the copy-and-pasted check for locking into a central
  location in the H5FD.c code.
- Fixed a bunch of compiler warnings, especially ones that trigger CI
  failures with -Werror

.github/workflows/netcdf.yml

@@ -77,7 +77,8 @@ jobs:
       run: |
         cd netcdf-c
         autoreconf -if
-        CFLAGS=${CFLAGS} LDFLAGS=${LDFLAGS} LD_LIBRARY_PATH=${LD_LIBRARY_PATH} ./configure --enable-hdf5 --enable-dap --disable-dap-remote-tests --enable-external-server-tests
+        # NOTE: --disable-byterange should be removed when the HTTP VFD has been updated
+        CFLAGS=${CFLAGS} LDFLAGS=${LDFLAGS} LD_LIBRARY_PATH=${LD_LIBRARY_PATH} ./configure --enable-hdf5 --enable-dap --disable-dap-remote-tests --disable-byterange --enable-external-server-tests
         cat config.log
         cat libnetcdf.settings
         CFLAGS=${CFLAGS} LDFLAGS=${LDFLAGS} LD_LIBRARY_PATH=${LD_LIBRARY_PATH} make -j

release_docs/RELEASE.txt

@@ -47,21 +47,34 @@ New Features
 
     Configuration:
     -------------
+    - Added a configuration option for internal threading/concurrency support:
+
+          CMake:      HDF5_ENABLE_THREADS (ON/OFF) (Default: ON)
+          Autotools:  --enable-threads    (yes/no) (Default: yes)
+
+      This option enables support for threading and concurrency algorithms
+      within the HDF5 library.  It is required for, but separate from, the
+      'threadsafe' configure option, which makes the HDF5 API safe to call from
+      multiple threads.  It is possible to enable the 'threads' option and
+      disable the 'threadsafe' option, but not vice versa.  The 'threads' option
+      must be on to enable the subfiling VFD.
+
     - Added support for native zlib-ng compression.
 
       Changed the zlib-ng CMake logic to prefer the native zlib-ng library. Added
       #ifdef around the compression function calls. Added including the correct
       header file with the same #ifdef.
 
     - Renamed remaining HDF5 library CMake options except for CMake BUILD* variables
-    
+
       DEFAULT_API_VERSION to HDF5_DEFAULT_API_VERSION
       DISABLE_PDB_FILES to HDF5_DISABLE_PDB_FILES
       ONLY_SHARED_LIBS to HDF5_ONLY_SHARED_LIBS
       ALLOW_UNSUPPORTED to HDF5_ALLOW_UNSUPPORTED
       TEST_SHELL_SCRIPTS to HDF5_TEST_SHELL_SCRIPTS
 
       All other HDF5 library CMake options are prefixed with HDF5_
+
     - bin/cmakehdf5 has been removed
 
       This was an unsupported build script that made building HDF5 via CMake
@@ -104,21 +117,14 @@ New Features
       We have updated the build files to set the C standard to C11, though
       some platforms use gnu11 to get some GNU things to work.
 
-    - Added configuration option for internal threading/concurrency support:
-
-          CMake:      HDF5_ENABLE_THREADS (ON/OFF) (Default: ON)
-          Autotools:  --enable-threads    (yes/no) (Default: yes)
-
-      This option enables support for threading and concurrency algorithms
-      within the HDF5 library.  It is required for, but separate from, the
-      'threadsafe' configure option, which makes the HDF5 API safe to call from
-      multiple threads.  It is possible to enable the 'threads' option and
-      disable the 'threadsafe' option, but not vice versa.  The 'threads' option
-      must be on to enable the subfiling VFD.
 
 
     Library:
     --------
+    - Removed H5FDperform_init API routine.  Virtual File Driver (VFD)
+      developers who wish to provide an ID for their driver should create
+      a routine specific to their individual implementation.
+
     - H5Pset_external() now uses HDoff_t, which is always a 64-bit type
 
       The H5Pset_external() call took an off_t parameter in HDF5 1.14.x and

src/CMakeLists.txt

@@ -236,17 +236,18 @@ set (H5FD_SOURCES
     ${HDF5_SRC_DIR}/H5FDmpi.c
     ${HDF5_SRC_DIR}/H5FDmpio.c
     ${HDF5_SRC_DIR}/H5FDmulti.c
+    ${HDF5_SRC_DIR}/H5FDmulti_int.c
     ${HDF5_SRC_DIR}/H5FDonion.c
     ${HDF5_SRC_DIR}/H5FDonion_header.c
     ${HDF5_SRC_DIR}/H5FDonion_history.c
     ${HDF5_SRC_DIR}/H5FDonion_index.c
-    ${HDF5_SRC_DIR}/H5FDperform.c
     ${HDF5_SRC_DIR}/H5FDros3.c
     ${HDF5_SRC_DIR}/H5FDs3comms.c
     ${HDF5_SRC_DIR}/H5FDsec2.c
     ${HDF5_SRC_DIR}/H5FDspace.c
     ${HDF5_SRC_DIR}/H5FDsplitter.c
     ${HDF5_SRC_DIR}/H5FDstdio.c
+    ${HDF5_SRC_DIR}/H5FDstdio_int.c
     ${HDF5_SRC_DIR}/H5FDtest.c
     ${HDF5_SRC_DIR}/H5FDwindows.c
 )
@@ -754,7 +755,6 @@ set (H5_MODULE_HEADERS
     ${HDF5_SRC_DIR}/H5ESmodule.h
     ${HDF5_SRC_DIR}/H5Fmodule.h
     ${HDF5_SRC_DIR}/H5FAmodule.h
-    ${HDF5_SRC_DIR}/H5FDdrvr_module.h
     ${HDF5_SRC_DIR}/H5FDmodule.h
     ${HDF5_SRC_DIR}/H5FLmodule.h
     ${HDF5_SRC_DIR}/H5FSmodule.h