Initial Playwright setup

[benmartin-coforma]

When merged, this PR will resolve ticket CMDCT-4062.

We still need accessibility tests, and more smoke tests. But this is probably sufficient as-is for now.

Hopefully the rest of this comment will prove useful for adding Playwright to SEDS, CARTS, and QMR someday.

Playwright Setup

1. Create the /tests folder. All code related to integration tests will live here.
   1. In that folder, if there isn't already a package.json, create one with yarn init
   2. Within that directory, yarn add @playwright/test @axe-core/playwright
      • BY THE WAY: You will see a warning that axe-core has unmet peer dependency playwright-core. I don't believe this is a problem; axe-core expected us to install playwright but we opted for @playwright/test instead. The former exposes only the playwright library, but the latter also includes the test runner bells and whistles. And in fact, if you inspect the yarn.lock file, you will find that playwright-core was included, as a transient dependency.
   3. Within that directory, yarn playwright install
      • This installs the browser binaries playwright uses to run the tests: chromium, firefox, and webkit. We only actually use chromium at the moment, but the installer doesn't know that.
   4. Copy the tests/playwright.config.ts file from another app that's already using playwright, like MCR.
   5. Add the playwright scripts to this package.json. Something like:
      • "test": "playwright test", — executes the tests by themselves
      • "test-ui": "playwright test --ui", — brings up the test runner interface, allowing you to select which tests to run.
2. Tell the rest of the repo not to be alarmed by the new folder
   1. In the root tsconfig.json, add "./tests/*" to the exclude list.
      • The root tsconfig is used to compile the files in /src, which stand up the app for local debugging. Therefore, it specifies /src as its root directory. If the TS compiler, when using this config, sees any .ts files outside of the root directory, it will complain. So this exclusion hides our new files to prevent that.
   2. In the root .gitignore, add rules for auto-generated tests files.

      tests/test-results/
      tests/playwright-report/
      tests/playwright/.cache/
      tests/playwright/.auth
      

      • You may find that your app has multiple .gitignores in different subdirectories, and you may wish to follow this pattern with a new .gitignore for the /tests directory. I find it's generally simplest to keep .gitignore settings in the root. But it's of little consequence either way.
   3. (Optional) In the root package.json, add scripts which forward to the scripts in the tests/package.json
      • For example: "test:e2e": "cd tests && yarn test && cd -"
        • BY THE WAY: Here, the && is a command separator; the three commands will be run in sequence. And cd - navigates to the previous working directory: in this case, the root directory
      • This is optional because it may convenient to invoke yarn run test:e2e from the root directory, but it is nearly as easy to cd tests yourself first.
3. Progress check: Write a test!
   1. Within the /tests directory, create a subdirectory called /playwright
   2. Within that directory, create a new file called hello.spec.ts:

      import { test, expect } from "@playwright/test";
      
      test.describe("Basic tests", () => {
         test("The site should load", async ({ page }) => {
            await page.goto("/");
            expect(page.url()).toBe("http://localhost:3000/");
         });
      });

   3. Within the /tests directory, yarn test
      • You should see this test execute and pass.
   4. Spin up the Test Runner with yarn test-ui
      • You should see the Playwright UI come up
      • Your test should be listed in the sidebar
      • You should be able to click into the test and execute it
      • You should see your app's login screen in the main pane
   5. Take a lil' break. Have a snack. You're doing great.
4. Set up auth
   1. In .env.tpl, the environment setup file in the root directory, add authentication-related variables

      # These settings are needed for playwright end-to-end tests
      TEST_ADMIN_USER_EMAIL=op://mdct_devs/mcr_secrets/CYPRESS_ADMIN_USER_EMAIL
      TEST_ADMIN_USER_PASSWORD=op://mdct_devs/mcr_secrets/CYPRESS_ADMIN_USER_PASSWORD # pragma: allowlist secret
      TEST_STATE_USER_EMAIL=op://mdct_devs/mcr_secrets/CYPRESS_STATE_USER_EMAIL
      TEST_STATE_USER_PASSWORD=op://mdct_devs/mcr_secrets/CYPRESS_STATE_USER_PASSWORD # pragma: allowlist secret
      

      • Naturally, replace the OnePassword collection name with your app's (for example, hcbs_secrets)
   2. Update the actual .env file with ./run update-env
      • The playwright.config.ts file should already be pointing at the root .env file to load these values into the environment at runtime
   3. Add a new file /tests/playwright/utils/consts, which will pull these values from the environment into the code

      export const adminUser = process.env.TEST_ADMIN_USER_EMAIL!;
      export const adminPassword = process.env.TEST_ADMIN_USER_PASSWORD!; // pragma: allowlist secret
      export const stateUser = process.env.TEST_STATE_USER_EMAIL!;
      export const statePassword = process.env.TEST_STATE_USER_PASSWORD!; // pragma: allowlist secret
      

   4. Add a new file /tests/playwright/utils/auth.setup.ts
      • Copy this from an app already using playwright, making appropriate adjustments
      • BY THE WAY: These setup methods log in as a user, and then save the browser's state to a local file. That state includes whatever cookies, localStorage objects, or tokens are associated with that user's session.
5. Progress check: Write a test that requires admin login!
   1. In your spec file, add this test

      test("Admins should see the admin view", async ({ browser }) => {
         const adminContext = await browser.newContext({
            storageState: "playwright/.auth/admin.json"
         });
         const page = await adminContext.newPage();
         await page.goto("/");
      
         const stateSelection = page.getByLabel('Select state or territory:');
         await expect(stateSelection).toBeVisible();
      
         await adminContext.close();
      });

   2. Run yarn test-ui command again, if you already closed the Test Runner since the last time.
   3. Run the new test. The auth setup should run automatically first, and the test should succeed.
      • HOT TIP: Not seeing your test in the sidebar? It may be hidden by filters in the top-left. Ensure you're looking at "Status: all, Projects: all".
      • JUST IN CASE: The setup won't run automatically each test execution - only each time you start the Test Runner. If you leave the Test Runner open all day, you may need to re-run the setup steps manually (by clicking that lil' play button).
      • BY THE WAY: Playwright knows that the auth setup steps are setup steps because that's what you named the file: auth.setup.ts! Which matches the regex in playwright.config.ts.
   4. This is a good time to delete our initial "The site should load" test. It's not doing anything useful, and it will fail in CI anyway (since we won't be on localhost anymore).
      • If you have the Playwright Test Runner up, you'll see that test disappear instantly from the sidebar. Thanks, hot reload!
   5. Hey, this is great progress! Good work, you.
6. Run the tests during deployment
   1. In .github/workflows/deploy.yml:
      1. Ensure the root-level permissions object includes pages: write and contents: write
      2. Add a test job that invokes playwright. Copy this from another app; no changes should be necessary.
      3. Add an upload-reports job that publishes the playwright results. Copy this too; no changes should be needed here either.
      4. Add test to the needs array for the cleanup job. Github Actions' default behavior is to try to run every job in parallel; the needs are how we specify which jobs should be sequential.
   2. You've been working on a branch, right? Commit your work so far, but don't push it yet!
   3. Create a new branch off of main, named gh-pages, and push that as-is.
      • BY THE WAY: The upload-reports job will actually commit test results to this branch, whenever it executes. The branch name isn't magical, but it is the conventional name for these things.
   4. In the repo settings on github, enable Github Pages, pointing it to the gh-pages branch
      • JUST IN CASE: If you don't have permissions to do this, your app lead will.
   5. In the repo settings on github, under "Secrets and Variables" > "Actions" > "Repository Secrets", add the four environment variables you need (TEST_ADMIN_USER_EMAIL and the rest).
      • Again, your app lead will have permissions to do this.
   6. Switch back to your playwright branch, and now push it.
   7. You should your new actions execute in Github, with happy green console messages as all of the tests pass
      • JUST IN CASE: If it seems your tests can't find the environment variables, double-check deploy.yml. Under the test job, under the Run Playwright Tests step, there will be an env map, connecting the new github secrets to the process.env of the test runner. Are the correct secret names being mapped to the correct environment variable names?
      • HOT TIP: If the tests fail, and you notice that this does not block your PR, you may wish to imitate the test_outcome management in HCBS' deploy.yml. In short: Give the test step an id. Output that step's outcome at the job level. During cleanup, check that outcome, and fail the job if needed.
      • BY THE WAY: That test_outcome song and dance wouldn't normally be needed; Github automatically fails jobs with non-success outcomes. But we had to set the continue-on-error flag for the test step, overriding Github's normal behavior - otherwise, when tests failed, we would not upload reports.
   8. The actions' output should include a link to the playwright report
   9. That link should work, and the playwright report should be gorgeous ✨
      • JUST IN CASE: The Github Pages publish process is super fast, but not instant. If the link doesn't work at first, give it just a minute.
   10. We're in the home stretch now!
7. Clean up old test results
   1. Check out the gh-pages branch
   2. Copy deleteOldFolders.js from the HCBS repo into your repo's root directory
   3. Copy ./github/workflows/delete-pages.yml from the HCBS repo as well
      • BY THE WAY: at the moment (Nov '24), MFP and MCR use a python script. HCBS uses JS. Either should work, but you can't mix and match the script and yml files.
   4. Commit and push!
   5. To test this, wait 31 days. Then look at the gh-pages branch and see if there were any recent commits deleting old test result folders.
      • To speed things up I guess you could edit the MAX_AGE_DAYS variable in that script, to something less than 30.
      • Or maybe create a folder yourself with a timestamp in the past.
      • DON'T FORGET: any changes you make to this workflow or this script will have to be pushed to the gh-pages branch, because that's the branch the action runs on. You can push them to main if you like, but it won't make anything happen.
8. Get Organized
   • Hey, congratulations! You just Made Playwright Happen. 👏
   • You can delete the hello.spec.ts file we created during this guide. The only assertion it makes is that login happened, which will be verified inherently by every test you write from now on.
     • You could even make that assertion within the setup function itself, so that if login ever stops working, we'll have a very clear & direct indication in the test report.
   • You may wish to structure your tests with Page Object Models (POMs). These will introduce a layer of abstraction between the logic you write in the spec files and the nuts and bolts of page URLs, input selectors, and so on.
     • If you are going to have a full regression test suite, this can be a big win. With multiple tests hitting the same pages and clicking the same buttons, it can be nice to give names to those pages and buttons. See MFP and MCR for examples of how this might look.
     • If you are only going to have a "smoke test" style suite with a handful of tests, the extra layer of abstraction may be more trouble than it's worth. See HCBS for examples of how this might look.
   • If you use POMs, you should also encapsulate the auth sessions in Fixtures. This is another type of abstraction that reduces the amount of code needed in spec files. Again, take a look at what's going on in MFP and MCR.
   • If you do not use POMs, you can still reduce auth boilerplate in most cases with the test.use({ storageState }) shorthand. The long form — with browser.newContext(), context.newPage(), and context.close() — will only be necessary when multiple different logins are required within a single test.
     • See https://playwright.dev/docs/auth#advanced-scenarios for more details. Since our app has multiple user roles, we are already in a so-called "advanced scenario", but the docs include examples of various advanced ways to handle these auth contexts.
     • Another quick boilerplate win will be to pull the storage state filepaths into consts.ts
   • HOT TIP: The playwright UI has a little target icon, in the top-left corner of the main pane, that will generate high-quality selectors for any page element you point it to. If you're in the middle of writing a test and you're not sure how to refer to some button or input, this can be a big help!
9. Write more tests. As many as you like. You're in charge now.

[jessabean]

Just swooping in here to say that this is an epic PR description. I'm gonna get some coffee and settle down to read.

[codeclimate]

Code Climate has analyzed commit 7875ad0 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (90% is the threshold).

This pull request will bring the total coverage in the repository to 92.8% (0.0% change).

View more on Code Climate.

[peoplespete]

I left a few comments that might be helpful but if not no worries, nothing remotely close to problematic was here to find!