Merge remote-tracking branch 'origin/master' into test_webui

bin/run-demos.sh

@@ -7,7 +7,6 @@ fi
 
 oq info venv
 oq info cfg
-oq webui start & # test extract_dependencies
 
 # create .tmp.ini files with oqparam.to_ini()
 python -m openquake.calculators.checkers "$1"

debian/changelog

@@ -1,3 +1,7 @@
+  [Michele Simionato]
+  * Extended `gmfs_file` to multiple .hdf5 files, assuming they correspond to
+    disjoint sites and events
+
   [Paolo Tormene]
   * Added a `duplicates_strategy` argument to `read_df` and changed the
     approach used while reading station data, to calculate the average values

doc/getting-started/installation-instructions/server.md

@@ -155,6 +155,53 @@ gunicorn -w N wsgi:application
 
 where `N` is the number of workers. We suggest `N = 4`.
 
+### Limit systemd services with control group (slice)
+
+If you need to set a limit on the resources available for the OpenQuake service, Systemd offers a simple solution to create resource limits for a service,
+a unit type called "slice".
+
+This is a control group, which may apply limits that affect all processes in this slice/control group.
+
+To create a slice called webui.slice:
+
+```console
+systemctl edit openquake-webui.slice
+
+[Unit]
+Description=OpenQuake Webui Slice
+Before=slices.target
+
+[Slice]
+MemoryAccounting=true
+MemoryLimit=10G
+CPUAccounting=true
+CPUQuota=50%
+TasksMax=4096
+```
+
+Edit the webui.service file:
+```console
+systemctl edit --full openquake-webui.service
+```
+
+and add in the [Service] section:
+```console
+Slice=openquake-webui.slice
+```
+
+After changing systemd config files a reload of the daemon is required and a restart of the service:
+
+```console
+systemctl daemon-reload
+systemctl stop openquake-webui.service
+systemctl start openquake-webui.service
+```
+
+To check if the service is in the correct control group, run:
+
+```console
+systemctl status  openquake-webui.service
+```
 
 ### nginx
 

doc/underlying-science/secondary-perils.rst

@@ -78,7 +78,7 @@ The equation that describes this probability is:
 
 .. math::
 
-	P(L) = \frac{P(L | PGA=a) \cdot P_{ml}}{K_m K_w} \quad (1)
+	P(L) = \frac{P(L | PGA=a) \cdot P_{ml}}{K_m K_w} \\ (1)
 
 :math:`P(L|PGA=a)` is the conditional probability that a site will fail based on the :math:`PGA` and the :math:`LSC`. 
 :math:`P_{ml}` is the fraction of the total mapped area that will experience liquefaction if :math:`P(L|PGA=a)` reaches
@@ -120,28 +120,28 @@ Zhu et al. (2015)
 
 The model by `Zhu et al. (2015) <https://journals.sagepub.com/doi/abs/10.1193/121912EQS353M>`_, is a logistic 
 regression model requiring specification of the :math:`V_{s30} \, [\text{m/s}]`, the Compound Topographic 
-Index, :math:`CTI`, a proxy for soil wetness or groundwater depth :math:`gwd \, [\text{m}]`, 
+Index, :math:`CTI`, a proxy for soil wetness or groundwater depth, :math:`gwd \, [\text{m}]`, 
 the :math:`PGA_{M,SM} \, [\text{g}]` experienced at a site, and the magnitude of the causative earthquake.
 
 The model is quite simple. An explanatory variable :math:`X` is calculated as:
 
 .. math::
 
-	X = 24.1 + 2.067\, ln\, PGA_{M,SM} + 0.355\,CTI - 4.784\, ln\, V_{s30} \quad (2)
+	X = 24.1 + 2.067\, ln\, PGA_{M,SM} + 0.355\,CTI - 4.784\, ln\, V_{s30} \\ (2)
 
 and the final probability is the logistic function:
 
 .. math::
 
-	P(L) = \frac{1}{1+e^X} \quad (3)
+	P(L) = \frac{1}{1+e^X} \\ (3)
 
 The term :math:`PGA_{M,SM}` is the :math:`PGA` corrected by magnitude scaling factor, :math:`MSF`, that serves as proxy
 for earthquake duration. The :math:`MSF` is calculated as per `Youd et al. (2001) 
 <https://ascelibrary.org/doi/10.1061/%28ASCE%291090-0241%282001%29127%3A4%28297%29>`_:
 
 .. math::
 
-	MSF = \frac{10^{2.24}}{M^{2.56}} \quad (4)
+	MSF = \frac{10^{2.24}}{M^{2.56}} \\ (4)
 
 Both the :math:`CTI` and the :math:`V_{s30}` may be derived from digital elevation data. The :math:`Vs30` may be
 estimated from the topographic slope through the equations of `Wald and Allen (2007) 
@@ -157,7 +157,7 @@ relation:
 
 .. math::
 
-	CTI = \ln (d_a / \tan \delta) \quad (5)
+	CTI = \ln (d_a / \tan \delta) \\ (5)
 
 where :math:`d_{a}` is the upstream drainage area per unit width through the flow direction (i.e. relating to the DEM 
 resolution). It ranges from :math:`0` to :math:`20`. It was developed for hillslopes, and is not meaningful in certain
@@ -179,7 +179,7 @@ with the ADASYN sampling algorithm. The explanatory variable :math:`X` is comput
 
 .. math::
 
-	X = -11.489 + 3.864\, ln\, PGA_{M} + 2.328\,CTI - 0.091\, ln\, V_{s30} \quad (6)
+	X = -11.489 + 3.864\, ln\, PGA_{M} + 2.328\,CTI - 0.091\, ln\, V_{s30} \\ (6)
 
 and the probability of liquefaction in calculated using equation (3).
 
@@ -216,7 +216,7 @@ Model 1:
 
 .. math:: 
 
-   -0.0287 \sqrt{d_{c}} + 0.0666 d_{r} - 0.0369 \sqrt{d_{c}} \cdot d_{r} \quad (7)
+   -0.0287 \sqrt{d_{c}} + 0.0666 d_{r} - 0.0369 \sqrt{d_{c}} \cdot d_{r} \\ (7)
 
 
 Model 2: 
@@ -227,7 +227,7 @@ Model 2:
 
 .. math:: 
 
-   -0.2054 d_{w} - 0.0333 wtd \quad (8)
+   -0.2054 d_{w} - 0.0333 wtd \\ (8)
 
 and the probability of liquefaction is calculated using equation (3). Zero probability is heuristically assigned if 
 :math:`PGV < 3 \, \text{cm/s}` or :math:`V_{s30} > 620 \, \text{m/s}`.
@@ -241,10 +241,11 @@ improve the fit. The regression coefficients are given in Table 2.:
 
 .. math::
 
-	L(P) = \frac{a}{1+b\,e^{-c\,P}}^2 \quad (9)
+	L(P) = \frac{a}{\left( 1 + b\,e^{-c\,P} \right)^2} \\ (9)
+
 .. raw:: latex
 
-   \vspace{10pt}
+   \vspace{15pt}
 
 +----------------+-------------+-------------+
 | **Parameters** | **Model 1** | **Model 2** |
@@ -297,7 +298,7 @@ Model 1:
 
 .. math:: 
 
-   -0.309 \ln(d_{r} + 1) - 0.164 \sqrt{Z_{wb}} \quad (10)
+   -0.309 \ln(d_{r} + 1) - 0.164 \sqrt{Z_{wb}} \\ (10)
 
 
 Model 2: 
@@ -308,7 +309,7 @@ Model 2:
 
 .. math:: 
 
-   -0.291 \ln(d_{r} + 1) - 0.205 \sqrt{Z_{wb}} \quad (11)
+   -0.291 \ln(d_{r} + 1) - 0.205 \sqrt{Z_{wb}} \\ (11)
 
 
 and the probability of liquefaction is calculated using equation (3). Zero probability is heuristically assigned if 
@@ -355,29 +356,29 @@ The expected permanent displacement due to lateral spreading given the susceptib
 
 .. math::
 
-	E[PGD_{sc}] = K_{\Delta} \, E[PGD|(PGA/PL_{sc}) = a] \quad (12)
+	E[PGD_{sc}] = K_{\Delta} \, E[PGD|(PGA/PL_{sc}) = a] \\ (12)
 
 Where: :math:`E[PGD|(PGA/PL_{SC})=a]` is the expected ground displacement given the susceptibility category under a 
 specified level of normalised shaking, and is calculated as: 
 
 .. math:: 
 
-   12\, x - 12\ for\ 1 < PGA/PGA(t) < 2 \quad (13) 
+   12\, x - 12\ for\ 1 < PGA/PGA(t) < 2 \\ (13) 
 
 .. math:: 
 
-   18\, x - 24\ for\ 2 < PGA/PGA(t) < 3 \quad (14)
+   18\, x - 24\ for\ 2 < PGA/PGA(t) < 3 \\ (14)
 
 .. math:: 
 
-   70\, x - 180\ for\ 3 < PGA/PGA(t) < 4 \quad (15)
+   70\, x - 180\ for\ 3 < PGA/PGA(t) < 4 \\ (15)
 
 :math:`PGA(t)` is theminimum shaking level to induce liquefaction (see Table 1) :math:`K_{\Delta}` is the 
 displacement correction factor given that modify the displacement term for magnitudes other than :math:`M7.5`: 
 
 .. math:: 
 
-   K_{Delta} = 0.0086M^3\ - 0.0914M^2\ + 0.4698M\ - 0.9835 \quad (16)
+   K_{Delta} = 0.0086M^3\ - 0.0914M^2\ + 0.4698M\ - 0.9835 \\ (16)
 
 ############################
 Vertical settlements (Hazus)
@@ -423,14 +424,14 @@ sliding surface, and is calculated as:
 
 .. math:: 
 
-   a_{c} =(F_{s}-1)\ - \sin(\alpha)\cdot g \quad (17)
+   a_{c} =(F_{s}-1)\ - \sin(\alpha)\cdot g \\ (17)
 
 The lower bound of :math:`a_{c}` is set to 0.05 to avoid unrealistically large displacements. The static factor of 
 safety is calculated as:
 
 .. math::
 
-    F_s = \frac{c'}{\gamma t \sin(\alpha)} + \frac{\tan(\phi')}{\tan(\alpha)} - \frac{m \gamma_{w} \tan(\phi')}{\gamma \tan(\alpha)} \quad(18)
+    F_s = \frac{c'}{\gamma t \sin(\alpha)} + \frac{\tan(\phi')}{\tan(\alpha)} - \frac{m \gamma_{w} \tan(\phi')}{\gamma \tan(\alpha)} \\(18)
 
 where: :math:`c \, [\text{Pa}]` is the effective cohession with typical values ranging from :math:`20 \text{kPa}` for
 soils up to :math:`20 \, {MPa}` for unfaulted rocks. :math:`\alpha^\circ` is the slope angle. :math:`\phi'^\circ` is 
@@ -457,7 +458,7 @@ equation can be used to estimate the probability of slope failure as a function
 
 .. math:: 
 
-    P(f) = 0.335\ [1 - e^{-0.048 \cdot D^{1.565}}] \quad (19)
+    P(f) = 0.335\ [1 - e^{-0.048 \cdot D^{1.565}}] \\ (19)
 
 The rock-slope failures are the other common effect observed in earthquakes. The methodology proposed by `Grant et al., 
 (2016) <https://www.sciencedirect.com/science/article/abs/pii/S0013795216302460?via%3Dihub>`_ captures the brittle 
@@ -466,22 +467,22 @@ is computed as:
 
 .. math:: 
 
-    F_s = \frac{2 c \sin(\beta)}{\gamma h (\beta-\alpha) \sin(\alpha)} + \frac{\tan(\phi}{\tan(\alpha} \quad (20)
+    F_s = \frac{2 c \sin(\beta)}{\gamma h (\beta-\alpha) \sin(\alpha)} + \frac{\tan(\phi)}{\tan(\alpha)} \\ (20)
 
 where: :math:`c \, [\text{Pa}]` is the cohession with typical values ranging from :math:`20 \, {kPa}` for soils up to 
 :math:`20 \, {MPa}` for unfaulted rocks. The cohesion provided by the root systems of vegetated hillslopes, 
 :math:`c_{r}`, is adopted as 0. :math:`\alpha^\circ` is the slope angle. :math:`\gamma \, [\text{kg/m^3}]` is the dry 
 density of the soil or rock. It ranges from :math:`1500 \, \text{kg/m^3}` for soils to 
-:math:`2500 - 3200 \, \text{kg/m^3}`. :math:`[m]` is the vertical height of the failure mass and it corresponds to 1/4 
+:math:`2500 \text{ to } 3200 \, \text{kg/m^3}`. :math:`[m]` is the vertical height of the failure mass and it corresponds to 1/4 
 of the local relief :math:`H` calculated based on the moving window analysis. :math:`\phi^\circ` is the effective 
 friction angle with typical values ranging from :math:`30^\circ` to :math:`40^\circ`. :math:`\beta` is the slope's 
 critical angle calculated as: 
 
 .. math:: 
 
-   \beta = \frac{\alpha + \phi}{0.5}\ (21)
+   \beta = \frac{\alpha + \phi}{0.5} \\ (21)
 
-The critical acceleration is computed similarly to equation (1). For rock- slope failures, the :math:`\alpha` term is 
+The critical acceleration is computed similarly to equation (17). For rock- slope failures, the :math:`\alpha` term is 
 replaced with :math:`\beta`.
 
 Finally, the coseismic displacements are estimated using the sliding block displacement regression equation proposed by
@@ -511,21 +512,23 @@ Explanatory variable :math:`X` is calculated as:
 
 .. math:: 
 
-   + \beta \cdot landcover + 0.03 CTI - 0.01 \ln(PGV) \cdot Slope \quad (22)
+   + \beta \cdot landcover + 0.03 CTI - 0.01 \ln(PGV) \cdot Slope \\ (22)
 
 Coefficients \alpha and \beta values are estimated for several rock and landcover classes. The 
 reader is reffered to the original study by `Nowicki Jessee et al. (2018) <https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2017JF004494>`_, 
 where the coefficient values are reported in Table 3. 
 
 Probability of landsliding is then evaluated using logistic regression.
+
 .. math::
 
-	P(L) = \frac{1}{1+e^X} \quad (23)
+	P(L) = \frac{1}{1+e^X} \\ (23)
+
+These probabilities are converted to areal percentages to unbias the predictions.
 
-These probabilities are converted to areal percentages to unbias the precitions.
 .. math::
 
-	L_{P}(P) = e^(-7.592 + 5.237 \cdot P - 3.042 \cdot P^2 + 4.035 \cdot P^3) \quad (24)
+	L_{P}(P) = e^{-7.592 + 5.237 \cdot P - 3.042 \cdot P^2 + 4.035 \cdot P^3} \\ (24)
 
 Furthermore, we introduced modifications by the USGS, capping the peak ground velocity at :math:`PGV = 211 \, \text{cm/s}`, 
 and compound topographic index at :math:`CTI = 19`. To exclude high probabilities of landsliding in nearly flat areas due to 

openquake/baselib/general.py

@@ -40,6 +40,7 @@
 import subprocess
 import collections
 import multiprocessing
+from importlib.metadata import version, PackageNotFoundError
 from contextlib import contextmanager
 from collections.abc import Mapping, Container, Sequence, MutableSequence
 import numpy
@@ -462,6 +463,19 @@ def removetmp():
                 pass
 
 
+def check_extension(fnames):
+    """
+    Make sure all file names have the same extension
+    """
+    if not fnames:
+        return
+    _, extension = os.path.splitext(fnames[0])
+    for fname in fnames[1:]:
+        _, ext = os.path.splitext(fname)
+        if ext != extension:
+            raise NameError(f'{fname} does not end with {ext}')
+
+
 def engine_version():
     """
     :returns: __version__ + `<short git hash>` if Git repository found
@@ -545,10 +559,15 @@ def check_dependencies():
     with open(os.path.join(repodir, reqfile)) as f:
         lines = f.readlines()
     for pkg, expected in extract_dependencies(lines):
-        version = __import__(pkg).__version__
-        if version != expected:
+        try:
+            installed_version = version(pkg)
+        except PackageNotFoundError:
+            # handling cases such as "No package metadata was found for zmq"
+            # (in other cases, e.g. timezonefinder, __version__ is not defined)
+            installed_version = __import__(pkg).__version__
+        if installed_version != expected:
             logging.warning('%s is at version %s but the requirements say %s' %
-                            (pkg, version, expected))
+                            (pkg, installed_version, expected))
 
 
 def run_in_process(code, *args):

openquake/baselib/tests/general_test.py

@@ -27,7 +27,7 @@
 from openquake.baselib.general import (
     block_splitter, split_in_blocks, assert_close, rmsdiff,
     deprecated, DeprecationWarning, cached_property,
-    compress, decompress, random_choice, get_duplicates)
+    compress, decompress, random_choice, get_duplicates, check_dependencies)
 
 
 class BlockSplitterTestCase(unittest.TestCase):
@@ -234,3 +234,7 @@ def test_get_duplicates():
     arr = numpy.array(lst, dtlist)
     dic = get_duplicates(arr, 'lon', 'lat')
     assert dic == {(2.1, 1.0): 2}, dic
+
+
+def test_check_dependencies():
+    check_dependencies()