Make ping endpoints deterministic #335

Kuba314 · 2023-10-11T15:11:44Z

Description

Continued work on endpoints started in #318. Ping endpoint IPs are now taken directly from configured IPs and not from ips_filter(), + some other changes.

Previously sequentially run pings are now run in parallel.

Tests

Early testing job of all distinct production-ready recipe,perf_stream types with thresholds: J:8445495

Reviews

@LNST-project/lnst-developers

olichtne · 2023-10-18T08:52:27Z

Ipsec*Recipe: Remove perf_reverse logic

just a note on this commit... you're correct in the logic of how this CURRENTLY works, but it USED to be that this parameter was in BaseEnrt and so ipsec supported perf_reverse as well... i'm not sure why the refactor changed this but it could be seen as a bug....

HOWEVER.... we don't actually run any reversed ipsec scenarios so i guess this is fine for now and we can add the perfreverse mixin to this once we actually want this functionality back.

Kuba314 · 2023-10-18T12:06:51Z

@olichtne Passing ping_bidirect to ipsec would result in an error either way. The earlier refactor could've possibly introduced the bug, but we didn't use the param before. I just removed the usage of this param, which wasn't currently able to be executed anyhow.

olichtne

looks ok overall

olichtne · 2023-10-18T11:36:04Z

lnst/Recipes/ENRT/helpers.py

+    def ip_version_string(ip_address: BaseIpAddress) -> str:
+        return "ipv4" if isinstance(ip_address, Ip4Address) else "ipv6"
+
+    for ip_version in ip_versions:


these loops should imo be inverted - this organization will re-sort the endpoint_pairs based on the collection of ip versions.

imo a function filter(condition, list) should not change the sort order of the list, only skip the items which don't fit the condition.

This was intentional. I felt that first filtering by IP type, then doing product, was much cleaner than doing a product over all IPs of both endpoints end returning only endpoint pairs where both IPs are of the same type.

We shouldn't rely on the returned endpoints to be sorted in any way. Should I switch to first doing product, then filtering? Is there a good reason for it?

this function doesn't do the product of endpoints though... it already accepts the list of endpoints and then returns a filtered list of endpoints. so it serves as a generic filter function for a list of objects. and imo that sort of filtering should be STABLE wrt. how the input list was ordered.

Ah, I thought we're talking about the ping_endpoint_pairs function and not filter_ip_endpoint_pairs. Sure, I agree the filter should be stable.

The reason for this is to display ipv4 and ipv6 endpoints one after the other in results/logs since all are run in parallel, but this ordering is already done in ping_endpoint_pairs. I'll change the filter to be stable.

Kuba314 · 2023-10-23T14:37:25Z

Up-to-date quick-check job: J:8466627

Kuba314 · 2023-10-24T07:41:24Z

New quick-check job: J:8469963 (had an error in my UUID-picking script, which picked tests with no thresholds instead of tests with thresholds, this time it should fail on threshold indexes).

olichtne · 2023-10-24T07:46:39Z

Up-to-date quick-check job: J:8466627

this interestingly failed for 2 tests:

for the GreTunnelRecipe:

FAIL 35_TestResult:	Packet assert unsuccessful, packets received 88, expected min(100) max(0)

for the GeneveTunnelRecipe:

FAIL 29_TestResult:	Packet assert unsuccessful, packets received 88, expected min(100) max(0)

i find it interesting that it's min(100) and max(0)?

Kuba314 · 2023-10-24T09:40:38Z

@olichtne only min check if effective if max is falsy. See https://github.com/LNST-project/lnst/blob/master/lnst/RecipeCommon/PacketAssert.py#L79. max is not provided and thus is 0 (falsy).

Kuba314 · 2023-10-31T11:38:44Z

Running tests again, this time properly in separate jobs for rhel 8 and 9:

RHEL8: J:8502193
RHEL9: J:8502194

Edit: I cancelled the jobs, because we decided to first refactor our internal threshold-index handling. This PR is blocked until that's done. This might take a while.

olichtne · 2023-10-31T14:17:43Z

postponed until we completely get rid of result index usage for baseline evaluation

Ipsec*Recipes don't inherit from PerfReversibleMixin and thus don't even accept the perf_reverse argument.

It makes sense for this method to be a generator since it's in the name.

This just makes the class implementation shorter.

This function yields only IP endpoints of IP type requested by the ip_versions argument.

ENRT/helpers included EnrtConfiguration from BaseEnrtRecipe which itself imports ENRT/helpers creating a cyclic import.

(instead of reversing PingConfs)

AFAICT, SimpleMacsecRecipe's generate_ping_configurations differed from BaseEnrtRecipe's implementation only in the fact that it did not actually implement ping_bidirect. Defining ping endpoints is enough.

This parameter was used to ensure that recipes unable to run multiple pings at the same time didn't do so. This will be handled per-recipe's generate_ping_endpoints() configuration where each recipe now decides which endpoint pairs it wants to ping-test in parallel.

This ensures that Ipsec*Recipes don't ever run with `ping_bidirect=True` instead of logging a debug message and throwing out parallel endpoint tests.

Defining generate_ping_endpoints() is enough. But we can't use the ip_endpoint_pairs helper function, because we have to respect the selected IPs from the subconfiguration and not just take all available ipv4 and ipv6 addresses as the helper function does.

We are returning lists, we should typehint the methods as such. Previously we generated collections, which are easier to put together, because they're covariant (`x: Collection[Base] = [Derived()]` is possible). Now this is not needed and can be achieved elsehow.

We have no use-case for testing ping endpoints sequentially.

`functools.permutations` generate even reversed endpoints (both AB and BA). This should be controlled by `BaseEnrtRecipe`'s `ping_bidirect`.

Kuba314 force-pushed the ping-endpoints branch 3 times, most recently from ee7c884 to 5589bab Compare October 17, 2023 16:27

olichtne previously approved these changes Oct 18, 2023

View reviewed changes

Kuba314 dismissed olichtne’s stale review via c9380f2 October 18, 2023 14:34

Kuba314 force-pushed the ping-endpoints branch from 3c64edb to 9ba38eb Compare October 23, 2023 14:36

Kuba314 force-pushed the ping-endpoints branch from 9ba38eb to 66a2296 Compare October 31, 2023 11:41

Kuba314 added 15 commits January 17, 2024 14:53

Ipsec*Recipe: Remove perf_reverse logic

601c4ad

Ipsec*Recipes don't inherit from PerfReversibleMixin and thus don't even accept the perf_reverse argument.

BaseFlowMeasurementGenerator: Make generate_perf_endpoints() a generator

cbb9a64

It makes sense for this method to be a generator since it's in the name.

Convert PingConf to a dataclass

7a53c98

This just makes the class implementation shorter.

ENRT/helpers: Add filter_ip_endpoint_pairs function

0b88e84

This function yields only IP endpoints of IP type requested by the ip_versions argument.

Move EnrtConfiguration to its own file

1fc36b3

ENRT/helpers included EnrtConfiguration from BaseEnrtRecipe which itself imports ENRT/helpers creating a cyclic import.

Make ping endpoints deterministic

9df4a1d

BaseEnrtRecipe: Implement ping_bidirect by reversing endpoint pairs

8e93508

(instead of reversing PingConfs)

SimpleMacsecRecipe: Don't override generate_ping_configurations

bce3f81

AFAICT, SimpleMacsecRecipe's generate_ping_configurations differed from BaseEnrtRecipe's implementation only in the fact that it did not actually implement ping_bidirect. Defining ping endpoints is enough.

Ipsec*Recipe: Make ping_bidirect a ConstParam

c0a508f

This ensures that Ipsec*Recipes don't ever run with `ping_bidirect=True` instead of logging a debug message and throwing out parallel endpoint tests.

Make all ping endpoints testing parallel

303b95f

We have no use-case for testing ping endpoints sequentially.

Fix various small mistakes

c8d8650

VxlanMulticastRecipe: Use functools.combinations for ping endpoints

e1fa702

`functools.permutations` generate even reversed endpoints (both AB and BA). This should be controlled by `BaseEnrtRecipe`'s `ping_bidirect`.

L2TPTunnelRecipe.py: Fix storing tunneled IP addresses

6297046

Kuba314 force-pushed the ping-endpoints branch from 66a2296 to 6297046 Compare January 17, 2024 13:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make ping endpoints deterministic #335

Make ping endpoints deterministic #335

Kuba314 commented Oct 11, 2023 •

edited

Loading

olichtne commented Oct 18, 2023

Kuba314 commented Oct 18, 2023

olichtne left a comment

olichtne Oct 18, 2023

Kuba314 Oct 18, 2023

olichtne Oct 19, 2023

Kuba314 Oct 19, 2023

Kuba314 Oct 19, 2023

Kuba314 commented Oct 23, 2023

Kuba314 commented Oct 24, 2023

olichtne commented Oct 24, 2023

Kuba314 commented Oct 24, 2023

Kuba314 commented Oct 31, 2023 •

edited

Loading

olichtne commented Oct 31, 2023

Make ping endpoints deterministic #335

Are you sure you want to change the base?

Make ping endpoints deterministic #335

Conversation

Kuba314 commented Oct 11, 2023 • edited Loading

Description

Tests

Reviews

olichtne commented Oct 18, 2023

Kuba314 commented Oct 18, 2023

olichtne left a comment

Choose a reason for hiding this comment

olichtne Oct 18, 2023

Choose a reason for hiding this comment

Kuba314 Oct 18, 2023

Choose a reason for hiding this comment

olichtne Oct 19, 2023

Choose a reason for hiding this comment

Kuba314 Oct 19, 2023

Choose a reason for hiding this comment

Kuba314 Oct 19, 2023

Choose a reason for hiding this comment

Kuba314 commented Oct 23, 2023

Kuba314 commented Oct 24, 2023

olichtne commented Oct 24, 2023

Kuba314 commented Oct 24, 2023

Kuba314 commented Oct 31, 2023 • edited Loading

olichtne commented Oct 31, 2023

Kuba314 commented Oct 11, 2023 •

edited

Loading

Kuba314 commented Oct 31, 2023 •

edited

Loading