From e481137f9c08223dceccf839b3afe03a40b1e51f Mon Sep 17 00:00:00 2001 From: Lucien Greathouse Date: Thu, 2 Nov 2017 16:46:40 -0700 Subject: [PATCH] Port in README and DESIGN files --- DESIGN.md | 66 +++++++++++++++++++++ README.md | 172 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 237 insertions(+), 1 deletion(-) create mode 100644 DESIGN.md diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 0000000..a5e8417 --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,66 @@ +# TestEZ Design Notes + +## Test Flow +Testing using this module is broken down into four steps: + +1. Load tests +2. Create test plan +3. Execute test plan +4. Report results + +Before executing the test plan, we can also modify it to change semantics separately from the code. + +### #1: Load tests +All ModuleScript objects that have a name ending in `.spec` are loaded as tests. On the filesystem, they appear as `.spec.lua` files. + +These tests should return a function that describes the actual test using the `describe`, `it`, and `expect` functions. These functions are automatically injected into the test environment when available. + +A test suite for addition might look like this: + +```lua +return function() + describe("Addition", function() + it("should be commutative", function() + local a, b, c = 5, 8, 11 + expect(a + b + c).to.equal(c + b + a) + end) + + it("should be associative", function() + local a, b, c = 7, 4, 9 + expect((a + b) + c).to.equal(a + (b + c)) + end) + end) +end +``` + +All test assertions should be contained inside the `it` blocks. + +### #2: Create test plan +A tree of tests is constructed out of all the `describe` and `it` calls in the tree. No test code is actually run. + +This allows us to potentially output a tree of the tests in the system without actually running them. It also gives us a mechanism to run only specific tests. + +This step is carried out by `TestPlanner`. It uses `TestPlanBuilder` to hold temporary state relevant only when building the plan, and then returns a `TestPlan` object. + +To debug the test plan tree, use `plan:visualize()` + +### #3: Execute test plan +A tree of test results is created that mirrors the test plan nodes. + +This step is carried out by `TestRunner`. It creates a `TestSession` using the `TestPlan` from the previous step. This object holds state only relevant when building the test results, then returns a `TestResults` object. + +To debug the result tree, use `results:visualize()` + +### #4: Report results +Reporting is handled by a test reporter object, which is just a table with a `report` method on it. It takes a `TestResults` object and outputs it to standard output, creates a GUI, or tells TestService about the results. + +This is pluggable, and could conceivably output in any format. + +## System Architecture +The system is broken down in a way that eliminates global state, provides multiple abstraction layers for each operation, and stores plain data in a way that's agnostic to the operations performed upon it. + +In both the planning and running phases of tests, a stateless module (`TestPlanner` and `TestRunner`) creates a temporary object to hold state about the operation it's trying to perform. These objects (`TestPlanBuilder` and `TestSession`) allow the code to traverse and build trees without passing around extra state between iterations. + +When the modules are done using these builder objects, they call `finalize` on them to receive a `TestPlan` or `TestResults` object. These objects are passed back to the original caller. + +`TestBootStrap` packages up use of the stages of testing into a simpler interface that also automatically locates test modules. When more advanced functionality is desired, it's easy to stop using TestBootstrap and instead call the underlying objects directly. \ No newline at end of file diff --git a/README.md b/README.md index 80dc9d1..5a4119e 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ TestEZ can run within Roblox itself, as well as inside [Lemur](https://github.com/LPGhatguy/Lemur) for testing on CI systems. -## Usage +## Installation ### Method 1: Installation Script (Roblox) * Download the latest release from the [GitHub releases page](https://github.com/Roblox/TestEZ/releases). @@ -28,5 +28,175 @@ You can use [Lemur](https://github.com/LPGhatguy/Lemur) paired together with a r This is the best approach when testing Roblox Lua libraries using existing continuous integration systems like Travis-CI. We use this technique to run tests for [Rodux](https://github.com/Roblox/Rodux). +## Usage + +## Writing Tests +Create `.spec.lua` files (or Roblox objects with the `.spec` suffix) for each module you want to test. These modules should return a function that in turn calls functions from TestEZ. + +A simple module and associated TestEZ spec might look like: + +`Greeter.lua` +```lua +local Greeter = {} + +function Greeter:greet(person) + return "Hello, " .. person +end + +return Greeter +``` + +`Greeter.spec.lua` +```lua +return function() + local Greeter = require(script.Parent.Greeter) + + describe("greet", function() + it("should include the customary English greeting", function() + local greeting = Greeter:greet("X") + expect(greeting:match("Hello")).to.be.ok() + end) + + it("should include the person being greeted", function() + local greeting = Greeter:greet("Joe") + expect(greeting:match("Joe")).to.be.ok() + end) + end) +end +``` + +The functions `describe`, `it`, and `expect` are injected by TestEZ and automatically hook into the current testing context. + +Every module is implicitly scoped according to its path, meaning the tree that the above test represents might be: + +``` +LuaChat + Greeter + greet + [+] should include the customary English greeting + [+] should include the person being greeted +``` + +## Debugging Tests +Often during development, you'll want to only run the test that's concerned with the specific code you're working on. + +TestEZ provides the `SKIP()` and `FOCUS()` functions to either skip or focus the block that the call is contained in. + +This mechanism does not work for `it` blocks, where you can instead is `itSKIP` and `itFOCUS`. This is because the code inside `it` blocks is not run until the test is executed. + +For example, I might want to run the tests targeting a specific method for my `DateTime` module: + +`DateTime.spec.lua` +```lua +return function() + describe("ImportantFeature", function() + FOCUS() + + it("does really important things", function() + -- This callback *will* run! + end) + end) + + describe("Format", function() + it("formats things", function() + -- This callback will never run! + end) + end) +end +``` + +***`FOCUS` and `SKIP` are intended exclusively for development; future versions of TeztEZ will be able to detect this when running in a CI system and fail tests!*** + +## TestEZ API + +### `describe(phrase, callback)` +This function creates a new `describe` block. These blocks correspond to the things that are being tested. + +Put `it` blocks inside of `describe` blocks to describe what behavior should be correct. + +For example: + +```lua +describe("This cheese", function() + it("should be moldy", function() + expect(cheese.moldy).to.equal(true) + end) +end) +``` + +### `it(phrase, callback)` +This function creates a new 'it' block. These blocks correspond to the behaviors that should be expected of the thing you're testing. + +For example: + +```lua +it("should add 1 and 1", function() + expect(1 + 1).to.equal(2) +end) +``` + +### `FOCUS()` +When called inside a `describe` block, `FOCUS()` marks that block as *focused*. If there are any focused blocks inside your test tree, *only* focused blocks will be executed, and all other tests will be skipped. + +When you're writing a new set of tests as part of a larger codebase, use `FOCUS()` while debugging them to reduce the amount of noise you need to scroll through. + +For example: + +```lua +describe("Secret Feature X", function() + FOCUS() + + it("should do something", function() + end) +end) + +describe("Secret Feature Y", function() + it("should do nothing", function() + -- This code will not run! + end) +end) +``` + +**Note: `FOCUS` will not work inside an `it` block. The bodies of these blocks aren't executed until the tests run, which is too late to change the test plan.** + +### `SKIP()` +This function works similarly to `FOCUS()`, except instead of marking a block as *focused*, it will mark a block as *skipped*, which stops any of the test assertions in the block from being executed. + +**Note: `SKIP` will not work inside an `it` block. The bodies of these blocks aren't executed until the tests run, which is too late to change the test plan.** + +### `expect(value)` +Creates a new `Expectation`, used for testing the properties of the given value. + +Expectations are intended to be read like English assertions. These are all true: + +```lua +-- Equality +expect(1).to.equal(1) +expect(1).never.to.equal(2) + +-- Nil checking +expect(1).to.be.ok() +expect(false).to.be.ok() +expect(nil).never.to.be.ok() + +-- Type checking +expect(1).to.be.a("number") +expect(newproxy(true)).to.be.a("userdata") + +-- Function throwing +expect(function() + error("nope") +end).to.throw() + +expect(function() + -- I don't throw! +end).never.to.throw() +``` + +## Inspiration and Prior Work +The `describe` and `it` syntax in TestEZ is based on the [Behavior-Driven Development](https://en.wikipedia.org/wiki/Behavior-driven_development) methodology, notably as implemented in RSpec (Ruby), busted (Lua), Mocha (JavaScript), and Ginkgo (Go). + +The `expect` syntax is based on Chai, a JavaScript assertion library commonly used with Mocha. Similar expectation systems are also used in RSpec and Ginkgo, with slightly different syntax. + ## License TestEZ is available under the [todo] license. See [LICENSE.md](LICENSE.md) for details. \ No newline at end of file