ftest coverage functionality is broken

[fgalan]

Code coverage reporting feature implemented in makefile could be broken (and I don't know since when).

It seems that the coverage target for unit tests works (see this) but problems happens with the functional test pass:

$ make coverage_functional_test INSTALL_DIR=~
...
 11/1130 Test   #11: geoquery_circle_deprecated ............................................................***Failed  Error regular expression found in output. Regex=[[^a-z]Fail] 11.95 sec
          Start   12: get_entity_dates_with_options
  12/1130 Test   #12: get_entity_dates_with_options .........................................................   Passed    4.12 sec
          Start   13: legacy_geoquery_bad_coords
  13/1130 Test   #13: legacy_geoquery_bad_coords ............................................................   Passed    1.59 sec
          Start   14: legacy_geoquery_circle_json
  14/1130 Test   #14: legacy_geoquery_circle_json ...........................................................***Failed  Error regular expression found in output. Regex=[[^a-z]Fail] 11.87 sec
          Start   15: legacy_geoquery_polygon_json
  15/1130 Test   #15: legacy_geoquery_polygon_json ..........................................................***Failed  Error regular expression found in output. Regex=[[^a-z]Fail] 15.46 sec
          Start   16: legacy_location_no_actual_location_change
  16/1130 Test   #16: legacy_location_no_actual_location_change .............................................   Passed    1.52 sec
          Start   17: legacy_wgs84
  17/1130 Test   #17: legacy_wgs84 ..........................................................................***Failed  Error regular expression found in output. Regex=[[^a-z]Fail] 11.83 sec
          Start   18: log_deprecate_warning
  18/1130 Test   #18: log_deprecate_warning .................................................................   Passed    1.69 sec
          Start   19: log_deprecate_warning_dynamic
  19/1130 Test   #19: log_deprecate_warning_dynamic .........................................................   Passed    2.12 sec
...

If we run testHarness just after that (i.e. without rebuilding the contextBroker binary) we can see the same tests are failing:

$ ./testHarness.sh cases/0000_deprecated_checkings/legacy_geoquery_circle_json.test 
mar 05 sep 2023 11:24:55 CEST
Run tests 0 to 0
0001/1: legacy_geoquery_circle_json.test .............................................................................  (ERROR 9 - .out and .regexpect differ) 
0001/1: legacy_geoquery_circle_json.test ............................................................................. (intent 2) (ERROR 9 - .out and .regexpect differ) 
0001/1: legacy_geoquery_circle_json.test ............................................................................. (intent 3) (FAIL 9 - .out and .regexpect differ) 
12 seconds
Total test time: 12.08 seconds

Orion Functional Test Log File:
================================================================================
Orion Functional tests starting
Tue Sep  5 11:24:55 CEST 2023

-----  Geo query (circle) JSON test (based on deprecated location metadata)  -----
(legacy_geoquery_circle_json.test) output not as expected
VALIDATION ERROR: input file and reference file have different line count


================================================================================

----------- Failing tests ------------------
  o  legacy_geoquery_circle_json.test

Looking into CB traces in /tmp/contextBroker.log we see:

time=2023-09-05T09:25:07.438Z | lvl=WARN | corr=1a765fba-4bce-11ee-b794-000c29e06b83 | trans=1693905904-163-00000000005 | from=127.0.0.1 | srv= | subsrv= | comp=Orion | op=AlarmManager.cpp[505]:badInput | msg=Raising alarm BadInput 127.0.0.1: location scope was found but your MongoDB version doesn't support it. Please upgrade MongoDB server to 2.4 or newer)

This is very weird... if we rebuild contextBroker binary again doing make di INSTALL_DIR=~ and run ./testHarness.sh cases/0000_deprecated_checkings/legacy_geoquery_circle_json.test again it works.

[fgalan]

The cause could be in the following part:

prepare_coverage: compile_info
        mkdir -p  BUILD_COVERAGE || true
        cd BUILD_COVERAGE && cmake .. -DCMAKE_BUILD_TYPE=DEBUG -DBUILD_ARCH=$(BUILD_ARCH) -DUNIT_TEST=True -DCOVERAGE=True -DCMAKE_INSTALL_PREFIX=$(INSTALL_DIR)

The UNIT_TEST declaration could be problematic for the contextBroker binary (it is good for the unitTest binary). Look for the #ifdef UNIT_TEST and #ifndef UNIT_TEST in the code.

Maybe activating UNIT_TEST the logic that check geo-query capabilities in the code gets disabled and that's is causing the above problem (just a theory, needs to be checked).

[fgalan]

I have tried to disable the UNIT_TEST declaration this way:

iff --git a/makefile b/makefile
index a279b0adf..c13bfe87b 100644
--- a/makefile
+++ b/makefile
@@ -74,7 +74,7 @@ prepare_debug: compile_info
 
 prepare_coverage: compile_info
        mkdir -p  BUILD_COVERAGE || true
-       cd BUILD_COVERAGE && cmake .. -DCMAKE_BUILD_TYPE=DEBUG -DBUILD_ARCH=$(BUILD_ARCH) -DUNIT_TEST=True -DCOVERAGE=True -DCMAKE_INSTALL_PREFIX=$(INSTALL_DIR)
+       cd BUILD_COVERAGE && cmake .. -DCMAKE_BUILD_TYPE=DEBUG -DBUILD_ARCH=$(BUILD_ARCH) -DCOVERAGE=True -DCMAKE_INSTALL_PREFIX=$(INSTALL_DIR)
 
 prepare_unit_test: compile_info
        @echo '------------------------------- prepare_unit_test starts ---------------------------------'

But I get this error:

$ make clean   # to remove old binaries in BUILD_COVERAGE and ensure re-build is done
$ make coverage_functional_test INSTALL_DIR=~
...
Finished .info-file creation
# Execute test for coverage
echo "Executing coverage test"
Executing coverage test
if [ -z "YES" ]; then \
    echo "Execute '. scripts/testEnv.sh' before executing the tests"; \
    exit 1; \
fi
make test -C BUILD_COVERAGE ARGS="-D ExperimentalTest" TEST_VERBOSE=1 || true
make[1]: se entra en el directorio '/home/fermin/src/fiware-orion/BUILD_COVERAGE'
make[1]: *** No hay ninguna regla para construir el objetivo 'test'.  Alto.
make[1]: se sale del directorio '/home/fermin/src/fiware-orion/BUILD_COVERAGE'
cat: BUILD_COVERAGE/Testing/TAG: No existe el fichero o el directorio
warning: failed to load external entity "BUILD_COVERAGE/Testing//Test.xml"
unable to parse BUILD_COVERAGE/Testing//Test.xml
make: *** [makefile:304: coverage_functional_test] Error 6

[fgalan]

This needs to be debuged and tested. Maybe a first step in the solution is not to use the same prepare_coverage target for both contextBroker and unitTest binaries (maybe split it into prepare_coverage_unittest and prepare_coverage_ftests).

Re-writting the coverage related targets from scratch in the makefile file could be also a good idea.

[fgalan]

Fixed by PR #4607