Optimize model and serialization + corrections + tests

feat: manage geom through protocol

- Fix bug changeActiveTab
- Add displaying conditional site group geometry through protocol

Reviewed-by: andriacap

fix: create site group without geom

Reviewed-by: andriacap

hotfix: remove import AppConfig

Error when using docker build

Reviewed-by: andriacap

Resolve conflicts

Use load_default for marshmallow schema - cf #268

 add id_digititer to generic observation.json  - cf #274

fix migrations downgrade - cf #269

create site with type without custom properties - cf #266

Correction frontend

[frontend] Style margin and map height - cf #270

fix site not visible in visit (#272)

fix error on site create from module (#275)

fix site property not displayed (#277)

lint prettier 3.1.0 (#278)

Remove laizy joined

Correction yml

Correction fixture sites

Add asset type de site (#282)

* Add asset type de site

Lint prettier (#281)

* Lint prettier 3.1.0

add dev-suivi-eolien in pytest.yml

[SqlAlchemy 1.4] compat sqlalchemy 1.4

Corrections + Ajout de  tests

Remove export as csv test

Fix/migrations (#285)

* Move create cor_type_site bib_type_site cor_module_type to geonature

* Remove id_nomenclature_type_site and migrate model to GeoNature

* Move t_observations to GeoNature

* Rename cor_type_site to cor_site_type

* Bump GeoNature

db downgrade does not work with data (#280)

Fix test export route (#293)

fix: organism_actors attributes

fix: clean python files imports + BLACK 24 (#290)

* fix: remove relative imports and unused libs

* fix: remove unused modules

* Add import fixture monitoring_users

* Reorder imports

* Black

---------

Co-authored-by: Mathieu ROUDAUT <[email protected]>
Co-authored-by: amandine-sahl <[email protected]>

fix: use table models migrated in geonature core BibTypesSites (#292)

* fix: use table models migrated in geonature core

* fix: imports in tests files

* Import TNomenclature from pypnnomenclature

---------

Co-authored-by: Mathieu ROUDAUT <[email protected]>
Co-authored-by: amandine-sahl <[email protected]>

Fix/clean transfert permissions ALL TO MONTORING_XXX (#291)

* Script sql de migration des permissions

* Changelog

Fix : sqlalchemy query

Fix/detail site url - cf #298 (#299)

* Correction url lien vers détail site

* Harmonisation popup carto

Fix/synchro on synthese (#288)

* remove silented errors when insert in synthese

* add migration for trigger delete in synthese
---------

Co-authored-by: amandine-sahl <[email protected]>

Feat/multiple geom type (#296)

* allow multiple geom type

* Sites avec plusieurs type de géométrie par défaut

---------

Co-authored-by: amandine-sahl <[email protected]>

Correction affichage propriétés spécifiques - cf #303 (#304)

Fix : migration modification for permissions (#305)

* Fix : migration modification for permissions

Ajout interaction carte liste (#300)

Fix downgrade declare available types sites

feat: add unit tests for generic monitoring routes and utils (#295)

* feat: add test for config route

* feat: add data_utils tests

* fix: response code for data utils route

* fix: rename test to match source file

* fix: remove useless file

* fix: remove legacy code

* feat: add tests for utils and error

Harmonisation des processus d'exécution de sql (#308)

* Harmonisation des processus d'exécution de sql

* Add test forbidden sql instructions

fix: filter ,sort and page on datatable (#310)

* fix: filter ,sort and page on datatable

Missing changes to check according to event on table(sort, filter, page)
Refact code for ngOnChanges

Reviewed-by: andriac

* style(front): apply prettier

Reviewed-by: andriacap

.github/workflows/lint.yml

@@ -13,6 +13,7 @@ jobs:
         uses: psf/black@stable
         with:
           src: "setup.py ./backend/gn_module_monitoring"
+
   frontend:
     runs-on: ubuntu-latest
     steps:
@@ -23,4 +24,5 @@ jobs:
         uses: creyD/[email protected]
         with:
           dry: True
-          prettier_options: --config frontend/.prettierrc --ignore-path frontend/.prettierignore --check frontend/**/*.ts
+          prettier_version: 3.1.0
+          prettier_options: --config frontend/.prettierrc --check frontend/**/*.ts

.github/workflows/pytest.yml

@@ -7,12 +7,15 @@ on:
       - hotfixes
       - develop
       - workshop
+      - dev-suivi-eolien
   pull_request:
     branches:
       - master
       - hotfixes
       - develop
       - workshop
+      - workshop
+      - dev-suivi-eolien
 
 jobs:
   build:
@@ -23,10 +26,6 @@ jobs:
       fail-fast: false
       matrix:
         include:
-          - name: "Debian 10"
-            python-version: "3.7"
-            postgres-version: 11
-            postgis-version: 2.5
           - name: "Debian 11"
             python-version: "3.9"
             postgres-version: 13

backend/gn_module_monitoring/blueprint.py

@@ -4,11 +4,12 @@
 """
 
 from flask import Blueprint, current_app
-from geonature.core.admin.admin import admin as flask_admin
+
 from geonature.utils.env import DB
+from geonature.core.admin.admin import admin as flask_admin
 
 from gn_module_monitoring.monitoring.admin import BibTypeSiteView
-from .command.cmd import commands
+from gn_module_monitoring.command.cmd import commands
 
 blueprint = Blueprint(
     "monitorings", __name__, template_folder=current_app.config["MEDIA_FOLDER"] + "/monitorings"

backend/gn_module_monitoring/command/cmd.py

@@ -1,29 +1,26 @@
-import os
-from pydoc import cli
 import click
 
 from pathlib import Path
 from flask.cli import with_appcontext
-from sqlalchemy.sql import text
+from sqlalchemy.sql import text, select
 
-from geonature.utils.env import DB, BACKEND_DIR
+from geonature.utils.env import DB
 from geonature.core.gn_synthese.models import TSources
 from geonature.core.gn_synthese.utils.process import import_from_table
 from geonature.core.gn_commons.models import TModules
 
+from gn_module_monitoring.config.repositories import get_config
+from gn_module_monitoring.config.utils import monitoring_module_config_path
+from gn_module_monitoring.monitoring.models import TMonitoringModules
+from gn_module_monitoring.modules.repositories import get_simple_module
 
-from ..monitoring.models import TMonitoringModules
-from ..config.repositories import get_config
-from ..config.utils import json_from_file, monitoring_module_config_path
-from ..modules.repositories import get_module, get_simple_module
-
-from .utils import (
-    process_export_csv,
+from gn_module_monitoring.command.utils import (
     process_available_permissions,
     remove_monitoring_module,
     add_nomenclature,
     available_modules,
     installed_modules,
+    process_sql_files,
 )
 
 
@@ -33,9 +30,18 @@
 def cmd_process_all(module_code):
     """
     Met à jour les paramètres de configuration pour un module
+        Fichiers sql synthese et export
     """
-    # process export csv
-    process_export_csv(module_code)
+    if module_code:
+        modules = [module_code]
+    else:
+        modules = [module["module_code"] for module in installed_modules()]
+
+    for module in modules:
+        # process Synthese
+        process_sql_files(dir=None, module_code=module, depth=1)
+        # process Exports
+        process_sql_files(dir="exports/csv", module_code=module, depth=None, allowed_files=None)
 
 
 @click.command("process_export_csv")
@@ -45,7 +51,13 @@ def cmd_process_export_csv(module_code):
     """
     Met à jour les fichiers pour les exports pdf
     """
-    process_export_csv(module_code)
+    if module_code:
+        modules = [module_code]
+    else:
+        modules = [module["module_code"] for module in installed_modules()]
+
+    for module in modules:
+        process_sql_files(dir="exports/csv", module_code=module, depth=None, allowed_files=None)
 
 
 @click.command("install")
@@ -98,8 +110,10 @@ def cmd_install_monitoring_module(module_code):
     except Exception:
         pass
 
-    # process export csv
-    process_export_csv(module_code)
+    # process Synthese
+    process_sql_files(dir=None, module_code=module_code, depth=1)
+    # process Exports
+    process_sql_files(dir=None, module_code=module_code, depth=None, allowed_files=None)
 
     config = get_config(module_code, force=True)
 
@@ -141,23 +155,6 @@ def cmd_install_monitoring_module(module_code):
     process_available_permissions(module_code, session=DB.session)
     DB.session.commit()
 
-    #  run specific sql
-    if (module_config_dir_path / "synthese.sql").exists:
-        click.secho("Execution du script synthese.sql")
-        sql_script = module_config_dir_path / "synthese.sql"
-        try:
-            DB.engine.execute(
-                text(
-                    open(sql_script, "r")
-                    .read()
-                    .replace(":'module_code'", "'{}'".format(module_code))
-                    .replace(":module_code", "{}".format(module_code))
-                ).execution_options(autocommit=True)
-            )
-        except Exception as e:
-            print(e)
-            click.secho("Erreur dans le script synthese.sql", fg="red")
-
     # insert nomenclature
     add_nomenclature(module_code)
 
@@ -195,14 +192,13 @@ def cmd_process_available_permission_module(module_code):
         module_code ([string]): code du sous module
 
     """
-
     if module_code:
-        process_available_permissions(module_code, session=DB.session)
-        DB.session.commit()
-        return
+        modules = [module_code]
+    else:
+        modules = [module["module_code"] for module in installed_modules()]
 
-    for module in installed_modules():
-        process_available_permissions(module["module_code"], session=DB.session)
+    for module in modules:
+        process_available_permissions(module, session=DB.session)
     DB.session.commit()
 
 
@@ -238,7 +234,9 @@ def synchronize_synthese(module_code, offset):
     Synchronise les données d'un module dans la synthese
     """
     click.secho(f"Start synchronize data for module {module_code} ...", fg="green")
-    module = TModules.query.filter_by(module_code=module_code).one()
+    module = DB.session.execute(
+        select(TModules).where(TModules.module_code == module_code)
+    ).scalar_one()
     table_name = "v_synthese_{}".format(module_code)
     import_from_table(
         "gn_monitoring",