CLI runner support

[realtica]

Hello, how can I run a test file from CLI command?

[tacheometry]

There is no CLI at the moment.

However, the VS Code extension and the Roblox Studio plugin are separate. If you are interested in building something of the sort, you can use the existing Studio extension, and then implement a HTTP server with the API similar to here: extension/src/testResultProviders/httpTestResultProviderGenerator.ts. The related test file should be of big help too.

Hope this info is of help 🙂 Unfortunately, at the moment not focusing on Roblox so much.

[jackTabsCode]

@tacheometry Is there any chance you could document the API surface for me? I'm interested in making a standalone server in Rust to resolve this issue, since I don't use VSCode anymore, but I'm having a hard time parsing the JS server code in this project.

[tacheometry]

Terminology:

• Extension: always refers to the VSCode extension (or your cli)
• Server: HTTP server periodically hosted by the extension
• Plugin: always refers to the Roblox Studio plugin

The Studio source code should also be of use: https://github.com/tacheometry/testez-companion/blob/main/plugin/src/plugin.server.lua - that's where all the requests are made, and it's only ever Studio -> localhost, not the other way around.

Whenever the user wants to run tests in a place, the HTTP server starts listening. After it gets test results or a couple hundred milliseconds pass, it closes.

This has been done this way (opposed to having a server constantly open) to support having multiple VSCode windows and multiple Studio windows open at the same time, since each VSCode window runs its own instance of the Companion extension, and each Studio instance runs its own plugin instance, but you're only ever running tests in a single project at once. If multiple Studio instances are open, the user is prompted to select which one to run tests in.

In a loop, the Studio plugin calls GET to http://127.0.0.1:28859/poll with identifier headers.

The server keeps the requests from each Studio place hanging, while also keeping the place identifiers in memory. After ~900 milliseconds pass since pressing the "run tests" button and starting the server, it responds to each request. By this time, the extension/user should have decided which place to run tests in (by choosing a place-guid) - if there are multiple places, options are presented to the user - otherwise, the first one is auto selected.

• The request with the correct place-guid gets responded with HTTP 200, and a JSON body of this type, describing the fields from testez-companion.toml:

  {
      "testRoots": ["ReplicatedStorage/TestsA", "ReplicatedStorage/TestsB"],
      "testExtraOptions": {}
  }

  testExtraOptions is passed directly from the config file into TestEZ, it is not processed by the plugin/extension.

• The rest of the requests get responded with HTTP 403.

After the Studio plugin gets the OK response, it starts running the tests. Either way, it keeps polling the address with /poll no matter what response it gets (if any).

While the tests run, if any Studio console logs are printed, each log will get POSTed to the /logs endpoint. The schema is linked - the Studio plugin basically sends a JSON body like:

{
    "message": "Hello world",
    "messageType": 0
}

For the messageType see https://create.roblox.com/docs/reference/engine/enums/MessageType. Server returns 200, or 403 if test results are already received. Identifier headers aren't actually sent (so it could be possible for an unrelated Studio instance to send logs), but the plugin is configured to only send logs if it is running tests - and it's only running tests if it's instructed to do so.

Once TestEZ finishes running the tests, it forwards the results to the plugin, which will then POST to /results. The JSON body will always be of the type TestEZ.ReporterOutput, even if TestEZ errors. If TestEZ errors, an extra header caught-testez-error will be set to "true". 200 is returned, or 403 if the place-guid isn't correct.

For an example of what gets sent to /results (in JSON form), see sampleTestEZOutput.ts. This implements the TestEZ.ReporterOutput type.

Finally, now that results have been received, the HTTP server can close and they can be displayed in the interface/CLI...

This is the Studio place I used to test the project, with failing/passing/skipped tests: https://github.com/tacheometry/testez-basic-place. Check it out - should save some time instead of scaffolding one yourself.

Summary:

1. User clicks "Run tests"
2. HTTP server starts at 127.0.0.1:28859
3. A 900ms timeframe begins wherein the server listens for requests at /poll, but doesn't respond yet. Keeps the place info in memory though.
4. StudioPlace1 and StudioPlace2 call /poll with their place GUIDs, names and IDs (the latter are used for disambiguation for the user). Their HTTP requests remain hanging
5. Timeframe ends, and the user is presented with the available places to run tests in
6. User picks StudioPlace1 as the place to run tests in - StudioPlace1's GUID is known
7. The server responds to StudioPlace2's request with a 403, and responds to StudioPlace1's request with a 200 and the extension config
8. StudioPlace1 starts running tests
9. If any console logs are outputted, they are posted to /logs
10. After tests finish, StudioPlace1 sends the results to /results
11. HTTP server closes
12. The extension displays the results

(again) These are the files I definitely recommend looking at:

• extension/src/testResultProviders/httpTestResultProviderGenerator.ts - HTTP server
• extension/src/testResultProviders/httpTestResultProviderGenerator.spec.ts - HTTP server unit tests
• plugin/src/plugin.server.lua - Studio plugin
• extension/src/sampleTestEZOutput.ts - the plugin output after getting TestEZ results - no other keys are present

Hope this helps 👍 If you make the project public I can definitely link it in Companion's README

[jackTabsCode]

Thank you for this write up! I really appreciate your time effort here. I am getting to work on this soon.

[jackTabsCode]

@tacheometry https://github.com/jackTabsCode/testez-companion-cli

It's not finished quite yet (doesn't support logging, and probably should install the plugin for you), but the basic functionality should be there!

[tacheometry]

@jackTabsCode This is great! I added it to the README

[tacheometry]

BTW, don't forget to add a license file 👌

[jackTabsCode]

BTW, don't forget to add a license file 👌

Whoops, that got missed!