Fix or remove flaky device tests

[jamescrosswell]

We have disabled a number of tests on iOS and Android.

In cases where these are flaky both on CI and on a local dev machine, these are disabled with an #if directive:

#if __IOS__
        Skip.If(true, "Flaky on iOS");
#endif

When they're only flaky when run in CI, we check for an environment variable instead:

Skip.If(TestEnvironment.IsGitHubActions, "This test is flaky in CI");

Ultimately we need to either work out why the tests fail and fix the code/environment/tests or simply remove the tests. There's no point in having tests that we don't trust in the repository. Preferably we'd fix the tests though.

Flaky Tests on iOS

Logging Tests

sentry-dotnet/test/Sentry.Extensions.Logging.Tests/LoggingTests.cs

Lines 15 to 19 in 0acaff1

	public void Log_CapturesEvent(LogLevel logLevel)
	{
	#if __IOS__
	Skip.If(true, "Flaky on iOS");
	#endif

sentry-dotnet/test/Sentry.Extensions.Logging.Tests/LoggingTests.cs

Lines 61 to 65 in 0acaff1

	public void Log_AddsBreadcrumb(LogLevel logLevel)
	{
	#if __IOS__
	Skip.If(true, "Flaky on iOS");
	#endif

The error message for both of the above tests is similar:

NSubstitute.Exceptions.ReceivedCallsException : Expected to receive exactly 1 call matching:
	EnqueueEnvelope(e => (e.Items.Select(i => i.Payload).OfType().Select(i => i.Source).OfType().Single().Breadcrumbs.SingleOrDefault(b => ((Convert(b.Level, Int32) == Convert(value(Sentry.Extensions.Logging.Tests.LoggingTests+<>c__DisplayClass1_0).logLevel.ToBreadcrumbLevel(), Int32)) AndAlso (b.Message == "test message"))) != null))
Actually received no matching calls.
Received 1 non-matching call (non-matching arguments indicated with '*' characters):
	EnqueueEnvelope(*Envelope*)
at NSubstitute.Core.ReceivedCallsExceptionThrower.Throw(ICallSpecification callSpecification, IEnumerable`1 matchingCalls, IEnumerable`1 nonMatchingCalls, Quantity requiredQuantity)
   at NSubstitute.Routing.Handlers.CheckReceivedCallsHandler.Handle(ICall call)
   at NSubstitute.Routing.Route.Handle(ICall call)
   at NSubstitute.Core.CallRouter.Route(ICall call)
   at NSubstitute.Proxies.CastleDynamicProxy.CastleForwardingInterceptor.Intercept(IInvocation invocation)
   at Castle.DynamicProxy.AbstractInvocation.Proceed()
   at NSubstitute.Proxies.CastleDynamicProxy.ProxyIdInterceptor.Intercept(IInvocation invocation)
   at Castle.DynamicProxy.AbstractInvocation.Proceed()
   at Castle.Proxies.ObjectProxy_3.EnqueueEnvelope(Envelope envelope)
   at Sentry.Extensions.Logging.Tests.LoggingTests.Log_AddsBreadcrumb(LogLevel logLevel)
   at System.Reflection.MethodInvoker.InterpretedInvoke(Object obj, Span`1 args, BindingFlags invokeAttr)

Maui Screenshot Tests

sentry-dotnet/test/Sentry.Maui.Tests/SentryMauiScreenshotTests.cs

Lines 36 to 42 in 0acaff1

	#if __MOBILE__
	[SkippableFact]
	public async Task CaptureException_WhenAttachScreenshots_ContainsScreenshotAttachmentAsync()
	{
	#if __IOS__
	Skip.If(true, "Flaky on iOS");
	#endif

sentry-dotnet/test/Sentry.Maui.Tests/SentryMauiScreenshotTests.cs

Lines 73 to 78 in 0acaff1

	[SkippableFact]
	public async Task CaptureException_RemoveScreenshot_NotContainsScreenshotAttachmentAsync()
	{
	#if __IOS__
	Skip.If(true, "Flaky on iOS");
	#endif

The error messages is similar for both of the above tests:

      <test name="Sentry.Maui.Tests.SentryMauiScreenshotTests.CaptureException_RemoveScreenshot_NotContainsScreenshotAttachmentAsync" type="Sentry.Maui.Tests.SentryMauiScreenshotTests" method="CaptureException_RemoveScreenshot_NotContainsScreenshotAttachmentAsync" time="0.0890636" result="Fail">
        <failure exception-type="Xunit.Sdk.XunitException">
          <message><![CDATA[Expected envelope not to be <null>.]]></message>
          <stack-trace><![CDATA[   at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message)
   at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
   at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
   at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
   at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
   at FluentAssertions.Execution.AssertionScope.FailWith(String message)
   at FluentAssertions.Primitives.ReferenceTypeAssertions`2[[Sentry.Protocol.Envelopes.Envelope, Sentry, Version=4.11.0.0, Culture=neutral, PublicKeyToken=fba2ec45388e2af0],[Sentry.Testing.EnvelopeAssertions, Sentry.Testing, Version=4.11.0.0, Culture=neutral, PublicKeyToken=fba2ec45388e2af0]].NotBeNull(String because, Object[] becauseArgs)
   at Sentry.Maui.Tests.SentryMauiScreenshotTests.CaptureException_RemoveScreenshot_NotContainsScreenshotAttachmentAsync()
--- End of stack trace from previous location ---]]></stack-trace>
        </failure>
      </test>

Flaky Tests on Android

We're not getting much useful information back about why the tests are failing on Android. Xharness simply says that the tests timed out after 900 seconds.

XHarness exit code: 70 (TIMED_OUT)

There are some adb logs but it's very hard to get any useful information out of these.

Logging Tests

sentry-dotnet/test/Sentry.Extensions.Logging.Tests/LoggingTests.cs

Lines 15 to 20 in 0acaff1

	public void Log_CapturesEvent(LogLevel logLevel)
	{
	#if __IOS__
	Skip.If(true, "Flaky on iOS");
	#endif
	Skip.If(TestEnvironment.IsGitHubActions, "Flaky on CI");

Hub Tests

sentry-dotnet/test/Sentry.Tests/HubTests.cs

Lines 1137 to 1139 in 0acaff1

	#if __ANDROID__
	Skip.If(true, "Flaky on Android");
	#endif

SDK Tests

https://github.com/getsentry/sentry-dotnet/blob/0acaff17253c1d1e3c572f0e28c779ced5987f5f/test/Sentry.Tests/SentrySdkTests.cs#L430-L432

Android API Levels

On Android, it appears the reliability of tests is impacted by the API Level that we're targeting.

We used to test against API levels 27 -31, for example, but the tests would almost invariably fail in one or more of those API levels, so we dropped back to testing against just 27 and 31 in this PR.

In PR #3628 the tests were often passing against API level 27 but failing against API level 33.

This may be related to Restrictions on non-SDK interfaces but, if so, I don't understand how.