Flaky RunCoverageTest tests

[adhiamboperes]

CI fails sometimes. I am flagging this because @Rd4dev reported this issue in #5453, so that can serve as a starting point for the investigation.

Based on part of the stacktrace:

Error in download_and_extract: java.io.IOException: Error extracting /home/runner/.cache/bazel/_bazel_runner/ef69b31fa14ea8341d6589b0a8832bdd/sandbox/linux-sandbox/513/execroot/__main__/_tmp/488e5f417e3c334e5203cccc5cfbe2bc/_bazel_runner/7184452ee16a7091323f5aab7f432b5f/external/remotejdk11_linux/temp13727209821490770147/zulu11.56.19-ca-jdk11.0.15-linux_x64.tar.gz to /home/runner/.cache/bazel/_bazel_runner/ef69b31fa14ea8341d6589b0a8832bdd/sandbox/linux-sandbox/513/execroot/__main__/_tmp/488e5f417e3c334e5203cccc5cfbe2bc/_bazel_runner/7184452ee16a7091323f5aab7f432b5f/external/remotejdk11_linux/temp13727209821490770147: write (No space left on device)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)ERROR: An error occurred during the fetch of repository 'remotejdk11_linux':
...
	FAILED: Build did NOT complete successfully (50 packages loaded, 644 targets configured)
ERROR: Couldn't start the build. Unable to run tests
	at org.oppia.android.scripts.common.BazelClient.executeBazelCommand(BazelClient.kt:221)
	at org.oppia.android.scripts.common.BazelClient.executeBazelCommand$default(BazelClient.kt:208)
	at org.oppia.android.scripts.common.BazelClient.runCoverageForTestTarget(BazelClient.kt:146)
	at org.oppia.android.scripts.coverage.CoverageRunner.retrieveCoverageResult(CoverageRunner.kt:51)
	at org.oppia.android.scripts.coverage.CoverageRunner.access$retrieveCoverageResult(CoverageRunner.kt:24)
	at org.oppia.android.scripts.coverage.CoverageRunner$runWithCoverageAsync$1.invokeSuspend(CoverageRunner.kt:41)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
...
Caused by: java.lang.IllegalStateException: Expected non-zero exit code (not 1) for command: [bazel, coverage, //coverage/test/java/com/example:TwoSumTest,

there seems to be a memory issue, causing errors extracting remotejdk11, which the Bazel setup requires to run.

This is all just speculation at this point, I don't have enough information to file an issue, so I am going to dump it here.

These are the failing tests:

1. testRunCoverage_ignoreCaseMarkdownArgument_returnsCoverageData(org.oppia.android.scripts.coverage.RunCoverageTest)
   java.lang.IllegalStateException: Expected non-zero exit code (not 1) for command: [bazel, coverage, //coverage/test/java/com/example:TwoSumTest, --instrumentation_filter=//coverage/...].

2. testRunCoverage_sharedAndLocalTestsHtmlFormat_returnsCoverageData(org.oppia.android.scripts.coverage.RunCoverageTest)
   java.lang.IllegalStateException: Expected non-zero exit code (not 36) for command: [bazel, query, --noshow_progress, --keep_going, set(app/sharedTest/java/com/example/TwoSumTest.kt app/test/java/com/example/TwoSumLocalTest.kt)].

3. testRunCoverage_sampleTestsMarkdownFormat_returnsCoverageData(org.oppia.android.scripts.coverage.RunCoverageTest)
   java.lang.IllegalStateException: Expected non-zero exit code (not 1) for command: [bazel, coverage, //coverage/test/java/com/example:TwoSumTest, --instrumentation_filter=//coverage/...].

4. testRunCoverage_sampleTestsDefaultFormat_returnsCoverageData(org.oppia.android.scripts.coverage.RunCoverageTest)
   java.lang.IllegalStateException: Expected non-zero exit code (not 1) for command: [bazel, coverage, //coverage/test/java/com/example:TwoSumTest, --instrumentation_filter=//coverage/...].

There was a corresponding Exit code 134(timeout) on gradle, so all this could just be a CI flake overall. I should be able to update if this happens again.

[Rd4dev]

To add to this (if useful), even recently encountered this scenario, but it doesn't occur with the proper shard implementation for the respective tests.

Without or with low shard_count
[stack trace]

With shard_count set to 24 (RunCoverageTest)
[stack trace]

[adhiamboperes]

@Rd4dev has the shard count been increased in your open PR?

[Rd4dev]

Yeah, I initially introduced a shard_count of 4 with #5453 then had it bumped to 24 (as 4 was not sufficient enough while running coverages on these tests) with PRs #5461 #5465.

[adhiamboperes]

Okay, sounds like we will need to watch the CI after #5461 and #5465 are merged then close the discussion if fully resolved.

[Rd4dev]

Sure, Ben also suggested having a [RunAllTests] run right after they are merged to get a better idea of how all the coverages perform. That should provide a clearer estimation of how expensive the coverage workflows might become and allow us to adjust timeouts, test sizes, and shard_counts accordingly.