Fix saving artifacts

[ochaplashkin]

What's the problem?

When creating the logic for saving artifacts ¹ a global variable has been created as current_test in the runner.lua module. Luatest has collected all the tests and runs a simple loop. At the beginning of each iteration the current test is written to the current_test.

Server saved the value of the current test. However at some point the current test in the global variable and in the server object were different (the server stored information about the previous test). Also server does not store information about which test it was used in.

What is the solution?

Redesign the logic of working with artifacts: now each test knows which server is linked to it, and each server knows about the test.

For example:

foo = Server:new()
bar = Server:new()
foo:start()
bar:start()

-- only `foo` artifacts
g.test_with_foo = function()
    foo:exec(...)
end

-- no server artifacts
g.test_without_servers = function()
    ...
end

How it works?

Common behavior with hooks

g.before_all(function()
    g.all = Server:new({alias = 'all'})
    g.all:start()
end)

g.before_each(function()
    g.each = Server:new({alias = 'each'})
    g.each:start()
end)

g.before_test('test_foo', function()
    g.test = Server:new({alias = 'test'})
    g.test:start()
end)


g.test_foo = function()
    g.internal = Server:new({alias = 'internal'})
    g.internal:start()
    ...
    t.assert(false)
end

Output

Failed tests:
-------------
...
stack traceback:
	...
	[C]: in function 'xpcall'
artifacts:
	all -> /tmp/t/artifacts/all-lPoX7PIOFj8e
	internal -> /tmp/t/artifacts/internal-LztDfWPAf7Rw
	each -> /tmp/t/artifacts/each--xhguIzWMweU
	test -> /tmp/t/artifacts/test-Q-UQS2ExN_Fz

Advanced behavior with hooks

Now there is no confusion between tests and servers. Only the server that is "linked" to the test will always be specified.

g.before_all(function()
    g.all = Server:new({alias = 'all'})
    g.all:start()
end)

g.before_each(function()
    g.each = Server:new({alias = 'each'})
    g.each:start()
end)

g.test_foo = function()
    t.assert(false)
end

g.test_bar = function()
    t.assert(false)
end

Output

Failed tests:
-------------

1) artifacts.test_foo
...
artifacts:
	each -> /tmp/t/artifacts/each-UlTDVHm1ptO4
2) artifacts.test_bar
...
artifacts:
	each -> /tmp/t/artifacts/each-RJTeIJKszZJW
Ran 2 tests in 0.314 seconds, 0 succeeded, 2 failed

❤️ Dynamic linking upon use

New feature: when using any server method, we will link it dynamically to the test. (thanks for the help @igormunkin ❤️ )

g.s = Server:new()

g.test_foo = function()
    t.assert_equals(1, 2)
end

g.test_bar = function()
    g.s:start()
    t.assert_equals(1, 2)
end

Output

Failed tests:
-------------

1) artifacts.test_foo
...
2) artifacts.test_bar
...
artifacts:
	server -> /tmp/t/artifacts/server-ZmK5Z2O4fPo2
Ran 2 tests in 0.106 seconds, 0 succeeded, 2 failed

📤 TAP 13 output

[077] replication-luatest/gh_6036_qsync_order_test.l>                 [ fail ]
[077] Test failed! Output from reject file /tmp/t/rejects/replication-luatest/gh_6036_qsync_order.reject:
[077] Tarantool version is 3.0.0-alpha1-148-g9af36ae10
[077] TAP version 13
[077] 1..2
[077] # Started on Mon Aug 28 09:25:37 2023
[077] # Starting group: gh-6036
[077] r1 | Deprecated option replication_connect_quorum, please use bootstrap_strategy instead
[077] r2 | Deprecated option replication_connect_quorum, please use bootstrap_strategy instead
[077] r3 | Deprecated option replication_connect_quorum, please use bootstrap_strategy instead
[077] ok     1	gh-6036.test_qsync_order
[077] ok     2	gh-6036.test_promote_order
[077] not ok 2	gh-6036.test_promote_order
[077] #   Failure in after_all hook: ...ace/vk/tarantool/test-run/lib/luatest/luatest/server.lua:356: attempt to index field 'process' (a nil value)
[077] #   stack traceback:
[077] #   	...ol/test/replication-luatest/gh_6036_qsync_order_test.lua:51: in function <...ol/test/replication-luatest/gh_6036_qsync_order_test.lua:50>
[077] #   	...
[077] #   	[C]: in function 'xpcall'
[077] #   artifacts:
[077] #   r2 -> /tmp/t/077_replication-luatest/artifacts/rs-SnDxoviAgtRd/r2-1RGjJrLxsMtA
[077] #   r3 -> /tmp/t/077_replication-luatest/artifacts/rs-SnDxoviAgtRd/r3-i0EoH_86Yiys
[077] #   r1 -> /tmp/t/077_replication-luatest/artifacts/rs-SnDxoviAgtRd/r1-2z9fjndrBZa8
[077] # Ran 2 tests in 1.997 seconds, 1 succeeded, 1 errored

Artifacts:

/tmp/t/077_replication-luatest/artifacts/rs-SnDxoviAgtRd/
├── r1-2z9fjndrBZa8
│   ├── 00000000000000000000.snap
│   ├── 00000000000000000000.xlog
│   ├── 00000000000000000022.xlog
│   └── r1.log
├── r2-1RGjJrLxsMtA
│   ├── 00000000000000000003.snap
│   ├── 00000000000000000003.xlog
│   ├── 00000000000000000023.xlog
│   └── r2.log
└── r3-i0EoH_86Yiys
    ├── 00000000000000000006.snap
    ├── 00000000000000000006.xlog
    ├── 00000000000000000014.xlog
    └── r3.log

3 directories, 12 files

• Tests
• Changelog
• Documentation

Close #299

Footnotes

1. Keep artifacts of server(logs, snap, etc) if test has been failed #296 ↩

[ochaplashkin]

I found a corner case:

g.s = Server:new()
g.s:start()


g.test_foo = function()
    t.assert(true)
end

g.test_bar = function()
    t.assert(false)
end

g.after_all(function()
    g.s:get_vclock()
end)

Expected:

test_foo - success
test_bar - failed without server artifacts (we didn't use it in this test).

Actual:

Failed tests:
-------------

1) logs.test_bar
/Users/mac/workspace/vk/luatest///test/logs_test.lua:43: expected: a value evaluating to true, actual: false
stack traceback:
	...
	[C]: in function 'xpcall'
artifacts:
	server -> /tmp/t/artifacts/server-zzHLh70opUsx
Ran 2 tests in 0.001 seconds, 1 succeeded, 1 failed

=========================================================
Failed tests:

logs.test_bar