single node timeout in make test

[danshearer]

make test fails with error 124 after four hours , in the testcase systablelockingextralockgenerated .

I don't think #2311 helps.

To Reproduce

Steps to reproduce the behavior:

1. Fresh git clone of comdb2 as of today
2. make ; make test
3. See error

Expected behavior

I expected make test to pass all tests, and hopefully well within 4 hours.

Screenshots

Here is the end of build/Testing/Temporary/LastTest.log:

ESTID=66869 running in /tmp/comdb2/tests/test_66869 systable_locking_extralock_generated 13/362
!systable_locking_extralock_generated: creating db systablelockingextralockgenerated66869
!systable_locking_extralock_generated: starting single node
!systable_locking_extralock_generated: waiting until ready
!systable_locking_extralock_generated: started running with timeout 30m at 2020-08-07 13:44:51
!systable_locking_extralock_generated: finished running with timeout 30m at 2020-08-07 14:14:51
!systable_locking_extralock_generated: timeout (logs in /tmp/comdb2/tests/test_66869/logs/systablelockingextralockgenerated66869.testcase)
!systable_locking_extralock_generated: stopping
make[2]: *** [/tmp/comdb2/tests/testcase.mk:59: test] Error 124
make[1]: *** [Makefile:89: systable_locking_extralock_generated] Error 2
<end of output>
Test time = 14460.52 sec
----------------------------------------------------------
Test Failed.
"all-tests" end time: Aug 07 14:18 BST
"all-tests" time elapsed: 04:01:00
----------------------------------------------------------

End testing: Aug 07 14:18 BST

However the testcase log file does not seem to contain any error information:

 :
 :
13:45:17> Completed 0 create and drop tables and views
13:45:53> Completed 10 stat_all_tables
13:46:34> Completed 10 write_tables
13:47:13> Completed 20 stat_all_tables
13:48:21> Completed 20 write_tables
13:48:29> Completed 30 stat_all_tables
13:49:53> Completed 40 stat_all_tables
13:49:53> Completed 10 create and drop tables and views
13:50:10> Completed 30 write_tables
13:51:04> Completed 50 stat_all_tables
13:51:57> Completed 40 write_tables
13:52:28> Completed 60 stat_all_tables
Duration 2049 seconds

Environment (please complete the following information):

• Operating System and Version: [e.g. Ubuntu Linux 18.04]

Ubuntu Linux 20.04.1 AMD64

[markhannum]

Thank you for submitting this issue! I am working on a fix for it now.

[adizaimi]

The test in question timed out:
!systable_locking_extralock_generated: timeout (logs in /tmp/comdb2/tests/test_66869/logs/systablelockingextralockgenerated66869.testcase)
the make all-tests method runs all tests sequentially and on an timeout (or error) will stop running any further tests. Please see comments to #2391 about ways to run in parallel and not stop at the first error encountered (basically make -k).

[adizaimi]

We have improved the detection when a test is stuck and should be better handling this scenario past its timeout (which in this case was 30min), fail this particular test and move on to the next.