This library comes with an Allure plugin that can be used to generate test reports based on +the metadata collected through each test execution.
+Add the plugin to your project:
+ +A test report is automatically generated when connectedDebugAndroidTest is executed if you're
+using InstrumentedReportRule from here
After each test execution, the plugin will generate the allure results +at:
+ +Then just run allure serve or allure generate to generate the reports and you should see something like this:
The following attachments are included out of the box for every failure:
+To customize the plugin use the following:
+allureReport {
+ /**
+ * The name of the test task that will be invoked
+ * to generate the report on `connectedAllureReport`
+ */
+ testTask = "connectedDebugAndroidTest"
+ /**
+ * By default, logcat files are not included if the test passes
+ */
+ keepLogcatOnSuccess = false
+ /**
+ * The report output path.
+ * Can be used to merge reports of multiple modules by using the same directory
+ * Null means `built/outputs/allure-results` is used
+ */
+ outputDir = null
+}
+You can aggregate all test reports by defining the outputDir option for the plugin.
+Place this in your root project build.gradle file:
subprojects {
+ /**
+ * Move all allure-results to the same directory,
+ * so that all tests from the project are seen in a single report
+ */
+ plugins.withId("com.rubensousa.carioca.report.allure") {
+ extensions.getByType(AllureReportExtension::class).apply {
+ outputDir = rootProject.file("build/outputs/allure-results")
+ }
+ }
+}
+This will move all test reports of all modules with the plugin to the same build/outputs/allure-results in your root
+project
The sample project in the official repository contains a few +examples for some reports
+ + + + + + + + + + + + + +Creates an AndroidComposeTestRule for EmptyHiltActivity, an empty activity that doesn't set any content.
This allows testing Composables that require hilt injections.
AndroidComposeTestRule.setContent must be called inside the test body
Creates an AndroidComposeTestRule for EmptyHiltActivity, an empty activity that doesn't set any content.
Launches a Fragment in the Activity's root view container android.R.id.content, with given arguments hosted by an empty FragmentActivity themed by themeResId, and waits for it to reach initialState.
Launches a Fragment in the Activity's root view container android.R.id.content, with given arguments hosted by an empty EmptyHiltActivity themed by themeResId, and waits for it to reach initialState.
Launches a Fragment in the Activity's root view container android.R.id.content, with given arguments hosted by an empty FragmentActivity themed by themeResId, and waits for it to reach initialState.
This method cannot be called from the main thread.
a fragment class to instantiate
a bundle to passed into fragment
a style resource id to be set to the host activity's theme
The initial Lifecycle.State. Passing in DESTROYED will result in an IllegalArgumentException.
Launches a Fragment in the Activity's root view container android.R.id.content, with given arguments hosted by an empty EmptyHiltActivity themed by themeResId, and waits for it to reach initialState.
This method cannot be called from the main thread.
a fragment class to instantiate
the activity that will host this fragment
a bundle to passed into fragment
a style resource id to be set to the host activity's theme
The initial Lifecycle.State. Passing in DESTROYED will result in an IllegalArgumentException.
FragmentAction interface should be implemented by any class whose instances are intended to be executed by the main thread. A Fragment that is instrumented by the FragmentScenario is passed to FragmentAction.perform method.
You should never keep the Fragment reference as it will lead to unpredictable behaviour. It should only be accessed in FragmentAction.perform scope.
Finishes the managed fragments and cleans up device's state. This method blocks execution until the host activity becomes Lifecycle.State.DESTROYED.
FragmentScenario provides API to start and drive a Fragment's lifecycle state for testing. It works with arbitrary fragments and works consistently across different versions of the Android framework.
FragmentScenario only supports androidx.fragment.app.Fragment. If you are using a deprecated fragment class such as android.support.v4.app.Fragment or android.app.Fragment, please update your code to androidx.fragment.app.Fragment.
If your testing Fragment has a dependency to specific theme such as Theme.AppCompat, use the theme ID parameter in launch method.
The Fragment class being tested
a scenario API for Activity
FragmentAction interface should be implemented by any class whose instances are intended to be executed by the main thread. A Fragment that is instrumented by the FragmentScenario is passed to FragmentAction.perform method.
Finishes the managed fragments and cleans up device's state. This method blocks execution until the host activity becomes Lifecycle.State.DESTROYED.
Moves Fragment state to a new state.
Runs a given action on the current Activity's main thread.
Recreates the host Activity.
Run block using HiltFragmentScenario.onFragment, returning the result of the block.
Moves Fragment state to a new state.
If a new state and current state are the same, this method does nothing. It accepts all Lifecycle.States. DESTROYED is a terminal state. You cannot move to any other state after the Fragment reaches that state.
This method cannot be called from the main thread.
Runs a given action on the current Activity's main thread.
Note that you should never keep Fragment reference passed into your action because it can be recreated at anytime during state transitions.
Throwing an exception from action makes the host Activity crash. You can inspect the exception in logcat outputs.
This method cannot be called from the main thread.
Recreates the host Activity.
After this method call, it is ensured that the Fragment state goes back to the same state as its previous state.
This method cannot be called from the main thread.
Launches a Fragment in the Activity's root view container android.R.id.content, with given arguments hosted by an empty FragmentActivity and waits for it to reach initialState.
Run block using HiltFragmentScenario.onFragment, returning the result of the block.
Launches a Fragment in the Activity's root view container android.R.id.content, with given arguments hosted by an empty FragmentActivity and waits for it to reach initialState.
This method cannot be called from the main thread.
a bundle to passed into fragment
a style resource id to be set to the host activity's theme
the initial Lifecycle.State. Passing in DESTROYED will result in an IllegalArgumentException.
Run block using HiltFragmentScenario.onFragment, returning the result of the block.
If any exceptions are raised while running block, they are rethrown.
A container activity for instrumented tests that need Hilt dependencies
Default Hilt test runner that Hilt does not provide out of the box Useful for running Hilt tests isolated from the rest of the application, like feature module testing
dispatcher that will be used for Dispatchers.Main
A test rule that replaces the Dispatchers.Main with another CoroutineDispatcher. See here for more information
Usage:
class TestClass {
@get:Rule
val dispatcherRule = MainDispatcherRule()
}Do not use this for instrumented tests
dispatcher that will be used for Dispatchers.Main
dispatcher that will be used for Dispatchers.Main
Annotates either a test class or a test method for total executions of 1 + times. This is extremely useful for checking if a test is flaky, but otherwise dangerous.
If this annotation is used in the same class and test methods, the annotation of the method takes precedence.
Example:
@RepeatTest(times = 1000)
class SampleTest() {
@RepeatTest(times = 10)
@Test
fun testSomething() {
// This test will be executed 10x or until it fails
}
@Test
fun testAnotherThing() {
// This test will be executed 1000x or until it fails,
// because of the class configuration
}
}Annotates either a test class or a test method with a retry request of times.
If this annotation is used in the same class and test methods, the annotation of the method takes precedence
Example:
@RetryTest(times = 5)
class SampleTest() {
@get:Rule
val retryRule = RetryTestRule()
@RetryTest(times = 2)
@Test
fun testSomething() {
// This test will be retried 2x if it fails
}
@Test
fun testAnotherThing() {
// This test will be retried 5x if it fails,
// because of the class configuration
}
}A test rule that replaces the Dispatchers.Main with another CoroutineDispatcher. See here for more information
Annotates either a test class or a test method for total executions of 1 + times. This is extremely useful for checking if a test is flaky, but otherwise dangerous.
A Rule that catches the Description of the test running
Dumps the semantic nodes as a string representation. Currently only works for the first/root Composable rendered in the screen
Find within merged composables like Buttons
A CariocaInstrumentedInterceptor that dumps the compose hierarchy on test failures
Find within merged composables like Buttons
Dumps the semantic nodes as a string representation. Currently only works for the first/root Composable rendered in the screen
A CariocaInstrumentedInterceptor that dumps the compose hierarchy on test failures
Creates a report for a set of steps. This is almost equivalent to calling step multiple times, but in a more re-usable way
Takes a screenshot with the configuration set through the report rule.
Creates a report for a set of steps. This is almost equivalent to calling step multiple times, but in a more re-usable way
Takes a screenshot with the configuration set through the report rule.
The generated file will be pulled from the device once the test runner finishes running all tests
the description of the screenshot for the report
the optional options that will override the setup from the test. If null, the default options set in the rule will apply
Creates an individual section of a test
the name of the step
an optional persistent step id
the step block that will be executed
Public constructor that uses TestStorageProvider to save the reports
A report rule for coroutine tests. Please check InstrumentedReportRule for more info on how to use the basic report functionality
Public constructor that uses TestStorageProvider to save the reports
Same as runTest
Runs the report inside a coroutine scope
Same as runTest
Runs the report inside a coroutine scope
A re-usable set of stages that can be used across multiple tests. Use this carefully and only when you have a stable set of steps that need to execute in an consistent order
the description of this scenario
a persistent id for tracking multiple executions of this scenario. Default: same as title
The main entry point for stage reports for tests that need coroutines
Creates a report for a set of steps. This is almost equivalent to calling step multiple times, but in a more re-usable way
Takes a screenshot with the configuration set through the report rule.
Creates a report for a set of steps. This is almost equivalent to calling step multiple times, but in a more re-usable way
Takes a screenshot with the configuration set through the report rule.
The generated file will be pulled from the device once the test runner finishes running all tests
the description of the screenshot for the report
the optional options that will override the setup from the test. If null, the default options set in the rule will apply
Creates an individual section of a test
the name of the step
an optional persistent step id
the step block that will be executed
Creates a report for a set of steps. This is almost equivalent to calling step multiple times, but in a more re-usable way
Takes a screenshot with the configuration set through the report rule.
A report rule for coroutine tests. Please check InstrumentedReportRule for more info on how to use the basic report functionality
A re-usable set of stages that can be used across multiple tests. Use this carefully and only when you have a stable set of steps that need to execute in an consistent order
The main entry point for stage reports for tests that need coroutines
Lifecycle of a report:
Each stage can start a nested stage before it's over. Example: scenario is a stage that starts nested step stages
A CariocaInstrumentedInterceptor that dumps the view hierarchy to a file whenever the test fails
A CariocaInstrumentedInterceptor that logs the different stages
the options for the screenshot file
the description of the attachment file
A CariocaInstrumentedInterceptor that triggers a screenshot when the test fails using the configuration provided via screenshotOptions
the options for the screenshot file
the description of the attachment file
Lifecycle of a report:
A CariocaInstrumentedInterceptor that dumps the view hierarchy to a file whenever the test fails
A CariocaInstrumentedInterceptor that logs the different stages
A CariocaInstrumentedInterceptor that triggers a screenshot when the test fails using the configuration provided via screenshotOptions
true if screen recording should start for the test. Default: true
the bitrate of the video file. Default: 16 mbps
the video scale in relation to the original display size. Default: 75% of the screen resolution
true if the recording should be kept if the test passes, false if it should be deleted
the minimum amount of time to wait after the recording starts. Default: 500ms
the minimum amount of time to wait before the recording should be stopped. Default: 1 second
the minimum amount of time to wait before continuing to the recording of a next test Default: 250ms
the orientation of the screen recording. Default: RecordingOrientation.NATURAL
the minimum amount of time to wait before continuing to the recording of a next test Default: 250ms
See more information here
true if screen recording should start for the test. Default: true
the bitrate of the video file. Default: 16 mbps
the video scale in relation to the original display size. Default: 75% of the screen resolution
true if the recording should be kept if the test passes, false if it should be deleted
the minimum amount of time to wait after the recording starts. Default: 500ms
the minimum amount of time to wait before the recording should be stopped. Default: 1 second
the minimum amount of time to wait before continuing to the recording of a next test Default: 250ms
the orientation of the screen recording. Default: RecordingOrientation.NATURAL
true if the recording should be kept if the test passes, false if it should be deleted
the orientation of the screen recording. Default: RecordingOrientation.NATURAL
the minimum amount of time to wait after the recording starts. Default: 500ms
Returns a representation of an immutable list of all enum entries, in the order they're declared.
This method may be used to iterate over the enum entries.
The orientation of the screen recording
Returns the enum constant of this type with the specified name. The string must match exactly an identifier used to declare an enum constant in this type. (Extraneous whitespace characters are not permitted.)
Returns an array containing the constants of this enum type, in the order they're declared.
Returns the enum constant of this type with the specified name. The string must match exactly an identifier used to declare an enum constant in this type. (Extraneous whitespace characters are not permitted.)
if this enum type has no constant with the specified name
Returns an array containing the constants of this enum type, in the order they're declared.
This method may be used to iterate over the constants.
the absolute file path to the video recording
the relative file path inside the test storage directory
the filename of the video recording file
the temporary file of the recording, while it is being recorded
A recording taken during a test report
the absolute file path to the video recording
the relative file path inside the test storage directory
the filename of the video recording file
the temporary file of the recording, while it is being recorded
the relative file path inside the test storage directory
Signals a test to be screen recorded
See more information here
The orientation of the screen recording
A recording taken during a test report
Signals a test to be screen recorded
a Uri that points to the new screenshot
the target directory from the test storage where the screenshot will be saved
the configurable options for this screenshot
the desired screenshot filename. Defaults to a random uuid
the scale of the screenshot in relation to the original display size. Default: 0.5
the quality of the screenshot. From 0 to 100. Default: 80
the image format of the screenshot. Default: JPG
true if images should be kept if the test passes. Default: true
the file extension of the format of these screenshot options
Configurable screenshot options that apply to all tests.
To override a configuration for a single test, annotate it with TestScreenshot
the scale of the screenshot in relation to the original display size. Default: 0.5
the quality of the screenshot. From 0 to 100. Default: 80
the image format of the screenshot. Default: JPG
true if images should be kept if the test passes. Default: true
true if images should be kept if the test passes. Default: true
Overrides the screenshot options for a single test
Configurable screenshot options that apply to all tests.
Overrides the screenshot options for a single test
Creates a report for a set of steps. This is almost equivalent to calling step multiple times, but in a more re-usable way
Takes a screenshot with the configuration set through the report rule.
Creates a report for a set of steps. This is almost equivalent to calling step multiple times, but in a more re-usable way
the re-usable set of stages
Takes a screenshot with the configuration set through the report rule.
The generated file will be pulled from the device once the test runner finishes running all tests
the description of the screenshot for the report
the optional options that will override the setup from the test. If null, the default options set in the rule will apply
Creates an individual section of a test
the name of the step
an optional persistent step id
the step block that will be executed
The main entry point for stage reports
Default scopes allowed: InstrumentedStageScope
A re-usable set of stages that can be used across multiple tests. Use this carefully and only when you have a stable set of steps that need to execute in an consistent order
the description of this scenario
a persistent id for tracking multiple executions of this scenario. Default: same as title
A StageReport for instrumented tests. Attachments should be stored in storageProvider
The main entry point for stage reports
Creates a report for a set of steps. This is almost equivalent to calling step multiple times, but in a more re-usable way
Takes a screenshot with the configuration set through the report rule.
Creates a report for a set of steps. This is almost equivalent to calling step multiple times, but in a more re-usable way
the re-usable set of stages
Takes a screenshot with the configuration set through the report rule.
The generated file will be pulled from the device once the test runner finishes running all tests
the description of the screenshot for the report
the optional options that will override the setup from the test. If null, the default options set in the rule will apply
Creates an individual section of a test
the name of the step
an optional persistent step id
the step block that will be executed
Returns a representation of an immutable list of all enum entries, in the order they're declared.
This method may be used to iterate over the enum entries.
The default instrumented stage types for each report
Returns the enum constant of this type with the specified name. The string must match exactly an identifier used to declare an enum constant in this type. (Extraneous whitespace characters are not permitted.)
Returns an array containing the constants of this enum type, in the order they're declared.
Returns the enum constant of this type with the specified name. The string must match exactly an identifier used to declare an enum constant in this type. (Extraneous whitespace characters are not permitted.)
if this enum type has no constant with the specified name
Returns an array containing the constants of this enum type, in the order they're declared.
This method may be used to iterate over the constants.
The main entry point for all reports.
Get the metadata of this test through metadata and/or getProperty.
To get the stages for reporting, use getTestStages
The public API for each report. This is the main entry for each test report.
Creates a report for a set of steps. This is almost equivalent to calling step multiple times, but in a more re-usable way
Takes a screenshot with the configuration set through the report rule.
The main entry point for stage reports
A re-usable set of stages that can be used across multiple tests. Use this carefully and only when you have a stable set of steps that need to execute in an consistent order
A StageReport for instrumented tests. Attachments should be stored in storageProvider
The main entry point for stage reports
The default instrumented stage types for each report
The main entry point for all reports.
The public API for each report. This is the main entry for each test report.
Adapted from: androidx.test.platform.io.TestDirCalculator to ensure we can delete files
An implementation of ReportStorageProvider that uses the test platform's directories
Adapted from: androidx.test.platform.io.TestDirCalculator to ensure we can delete files
An implementation of ReportStorageProvider that uses the test platform's directories
The basic report structure.
Default implementation is InstrumentedReportRule.
You can extend this class directly to create your own scopes, by returning your own test report class in createTest
Use this if you're interested in recording ignored tests, as there is no other way to do it at the moment
The relative output directory for this report.
the test report to be saved
A template reporter that uses JsonReportWriter to save the reports to the test storage
the metadata of the test
report to be written
the storage provider to query for a OutputStream
Public constructor that uses TestStorageProvider to save the reports
Use this to track @After methods separately from the other stages
Use this to track @Before methods separately from the other stages
A test rule that builds a detailed report for a test, including its steps.
You can also extend this class to provide a default report configuration across all tests:
class SampleReportRule : InstrumentedReportRule(
reporter = CariocaAllureReporter(),
recordingOptions = RecordingOptions(
bitrate = 20_000_000,
resolutionScale = 1.0f,
),
screenshotOptions = ScreenshotOptions(
scale = 1f
),
interceptors = listOf(LoggerInterceptor(), DumpHierarchyInterceptor())
)@get:Rule
val report = SampleReportRule()
@Test
fun sampleTest() = report {
step("Open notification and quick settings") {
step("Open notification") {
device.openNotification()
screenshot("Notification bar visible")
}
step("Open quick settings") {
device.openQuickSettings()
screenshot("Quick settings displayed")
}
}
step("Press home") {
device.pressHome()
}
}the instrumented reporter for this report
the default recording options for this report
the default screenshot options for this report
the interceptors that will receive multiple stage events during the lifecycle of this report
Public constructor that uses TestStorageProvider to save the reports
Use this to track @After methods separately from the other stages
Use this to track @Before methods separately from the other stages
Same as test, but without the extra method call
Use this to start reporting the main test body
Same as test, but without the extra method call
Use this to start reporting the main test body
The relative output directory for this report.
the test report to be saved
Implement this to generate your own test reports, in any format you want. DefaultInstrumentedReporter is the default implementation that is flexible enough to convert to any other format
the metadata of the test
report to be written
the storage provider to query for a OutputStream
The basic report structure.
Use this if you're interested in recording ignored tests, as there is no other way to do it at the moment
A template reporter that uses JsonReportWriter to save the reports to the test storage
Implement this to generate your own test reports, in any format you want. DefaultInstrumentedReporter is the default implementation that is flexible enough to convert to any other format
A test rule that builds a detailed report for a test, including its steps.
Returns a representation of an immutable list of all enum entries, in the order they're declared.
This method may be used to iterate over the enum entries.
Returns the enum constant of this type with the specified name. The string must match exactly an identifier used to declare an enum constant in this type. (Extraneous whitespace characters are not permitted.)
Returns an array containing the constants of this enum type, in the order they're declared.
Returns the enum constant of this type with the specified name. The string must match exactly an identifier used to declare an enum constant in this type. (Extraneous whitespace characters are not permitted.)
if this enum type has no constant with the specified name
Returns an array containing the constants of this enum type, in the order they're declared.
This method may be used to iterate over the constants.
Should be future proof, as we find the directory with our own name, without assuming the parent directory structure
Parses the reports generated by a previous test run. Use parseTestReports to retrieve them from a directory or parseTestReport to parse an individual report
A reporter that saves the test results in a json format.
Use writeReport to save the report to a file or output stream
Parses the reports generated by a previous test run. Use parseTestReports to retrieve them from a directory or parseTestReport to parse an individual report
A reporter that saves the test results in a json format.
Parses the TestMetadata from a test's Description
Retrieves the metadata of TestReport from a test's Description
Parses the TestMetadata from a test's Description
Retrieves the metadata of TestReport from a test's Description
the unique identifier for the stage. It changes across different executions
if the stage failed, this contains the error thrown
the execution status of this stage
indicates when this stage started its execution
indicates when this stage stopped its execution
if the stage failed, this contains the error thrown
Metadata for the execution for a test, step or scenario
the unique identifier for the stage. It changes across different executions
if the stage failed, this contains the error thrown
the execution status of this stage
indicates when this stage started its execution
indicates when this stage stopped its execution
Returns a representation of an immutable list of all enum entries, in the order they're declared.
This method may be used to iterate over the enum entries.
The different status of every report
Returns the enum constant of this type with the specified name. The string must match exactly an identifier used to declare an enum constant in this type. (Extraneous whitespace characters are not permitted.)
Returns an array containing the constants of this enum type, in the order they're declared.
Returns the enum constant of this type with the specified name. The string must match exactly an identifier used to declare an enum constant in this type. (Extraneous whitespace characters are not permitted.)
if this enum type has no constant with the specified name
Returns an array containing the constants of this enum type, in the order they're declared.
This method may be used to iterate over the constants.
a human readable representation of this attachment
the relative path of this attachment
the mime type of the file
true if the attachment should be kept if the test passes, false otherwise
An attachment linked to a given stage
a human readable representation of this attachment
the relative path of this attachment
the mime type of the file
true if the attachment should be kept if the test passes, false otherwise
true if the attachment should be kept if the test passes, false otherwise
Adds an extra property to this report
identifier for this property
value of this property
an attachment that will be linked to this report
attachment to be deleted from the filesystem
the attachment to remove from this report
all attachments registered through attach
the execution metadata associated to this stage
the map of parameters created with setParameter
all properties registered through addProperty
the property registered previously through addProperty or null if not found or does not match the expected type
the child stages that started in a @After method
the child stages that started in a @Before method
the child stages that started within this stage inside the test method
Marks this stage as skipped
The building block of all test reports
To get the main result of this report, use getExecutionMetadata
To get the reports within this own report, use: getStagesBefore, getTestStages, getStagesAfter
Adds an extra property to this report
Marks this stage as passed
Clears all the metadata associated to this report
the unique key of the parameter
the value of the parameter
the package name of the test class
the class under test
the name of the test method
The configuration built from the TestReport annotation
This will be used to fill the report with extra properties
A human-readable description of the test
Metadata for the execution for a test, step or scenario
The different status of every report
An attachment linked to a given stage
The building block of all test reports
Holds the current state of the stage execution.
The configuration built from the TestReport annotation
Plugin id: com.rubensousa.carioca.report.allure
2024-09-28
+2024-09-27
+outputDir option #442024-09-23
+2024-09-22
+