Smarter testbeds

[mockersf]

Objective

• Improve CI when testing rendering by having smarter testbeds

Solution

• CI testing no longer need a config file and will run with a default config if not found
• It is now possible to give a name to a screenshot instead of just a frame number
• 2d and 3d testbeds are now driven from code
  • a new system in testbed will watch for state changed
  • on state changed, trigger a screenshot 100 frames after (so that the scene has time to render) with the name of the scene
  • when the screenshot is taken (Captured component has been removed), switch scene
• this means less setup to run a testbed (no need for a config file), screenshots have better names, and it's faster as we don't wait 100 frames for the screenshot to be taken

Testing

• cargo run --example testbed_2d --features bevy_ci_testing

[rparrett]

This seems to be screenshotting the first scene twice.

cargo run --example testbed_2d --features=bevy_ci_testing

2025-01-28T13:01:59.281223Z  INFO bevy_dev_tools::ci_testing::systems: Took a screenshot at frame 100 for Shapes.
2025-01-28T13:02:00.822169Z  INFO bevy_render::view::window::screenshot: Screenshot saved to ./screenshot-Shapes.png
2025-01-28T13:02:00.826628Z  INFO testbed_2d: Switching scene
2025-01-28T13:02:01.766528Z  INFO bevy_dev_tools::ci_testing::systems: Took a screenshot at frame 206 for Bloom.
2025-01-28T13:02:03.339948Z  INFO bevy_render::view::window::screenshot: Screenshot saved to ./screenshot-Bloom.png
2025-01-28T13:02:03.344531Z  INFO testbed_2d: Switching scene
2025-01-28T13:02:04.189420Z  INFO bevy_dev_tools::ci_testing::systems: Took a screenshot at frame 311 for Text.
2025-01-28T13:02:05.626205Z  INFO bevy_render::view::window::screenshot: Screenshot saved to ./screenshot-Text.png
2025-01-28T13:02:05.631124Z  INFO testbed_2d: Switching scene
2025-01-28T13:02:06.477569Z  INFO bevy_dev_tools::ci_testing::systems: Took a screenshot at frame 416 for Sprite.
2025-01-28T13:02:07.919943Z  INFO bevy_render::view::window::screenshot: Screenshot saved to ./screenshot-Sprite.png
2025-01-28T13:02:07.923709Z  INFO testbed_2d: Switching scene
2025-01-28T13:02:08.760123Z  INFO bevy_dev_tools::ci_testing::systems: Took a screenshot at frame 521 for Shapes.
2025-01-28T13:02:10.214282Z  INFO bevy_render::view::window::screenshot: Screenshot saved to ./screenshot-Shapes.png
2025-01-28T13:02:10.224075Z  INFO bevy_dev_tools::ci_testing::systems: Exiting after 527 frames. Test successful!

To be honest, I'm slightly skeptical at the point about the utility of these scene-switching testbed examples. I wonder what is the advantage over a directory full of separate examples? Is app startup time in CI particularly significant? (Not blocking, just curious)

It would be nice for the testbeds to share switch_scene_in_ci.

[mockersf]

screenshots have nicer names:
https://pixel-eagle.com/project/B04F67C0-C054-4A6F-92EC-F599FEC2FD1D/run/6265/compare/6259?screenshot=testbed_2d/screenshot-Bloom.png

[mockersf]

To be honest, I'm slightly skeptical at the point about the utility of these scene-switching testbed examples. I wonder what is the advantage over a directory full of separate examples? Is app startup time in CI particularly significant? (Not blocking, just curious)

In native, not so much. In Wasm, build time is a lot longer. On mobile, even worse

I also think it's nice to have just one command to check for all 2d/3d/...

[rparrett]

Seems to be working well on my end.

[IceSentry]

I'm confused, when does the first scene change gets triggered to start the chain of next scenes?

Is it just that .is_changed() is true when the Scene is added?

[mockersf]

yup, it's true from start