From efe0ad62d31486e6f7125a31a527dff2366683c4 Mon Sep 17 00:00:00 2001 From: "Alex N. Jose" Date: Wed, 31 May 2023 09:26:09 -0700 Subject: [PATCH] Reorganize incubation repo to allow hosting multiple projects (#10) --- README.md | 104 +++++------------- .../cf-worker-prototype/.gitignore | 0 .../cf-worker-prototype/LICENSE | 0 prototypes/cf-worker-prototype/README.md | 88 +++++++++++++++ .../assets}/images/arch.png | Bin .../assets}/images/filmstrip.png | Bin .../assets}/images/sequence.svg | 0 .../cf-worker-prototype/package-lock.json | 0 .../cf-worker-prototype/package.json | 0 .../cf-worker-prototype/src}/client/index.ts | 0 .../cf-worker-prototype/src}/config.ts | 0 .../cf-worker-prototype/src}/index.ts | 0 .../src}/lib/ab-random-variant.ts | 0 .../src}/lib/ab-variant-selection.ts | 0 .../cf-worker-prototype/src}/lib/constants.ts | 0 .../src}/lib/pre-ua-transform-applicator.ts | 0 .../src}/lib/transformer.ts | 0 .../src}/lib/ua-transform-applicator.ts | 0 .../cf-worker-prototype/tsconfig.json | 0 .../cf-worker-prototype/wrangler.toml | 0 EXPLAINER.md => specs/EXPLAINER.md | 64 +++++------ 21 files changed, 146 insertions(+), 110 deletions(-) rename .gitignore => prototypes/cf-worker-prototype/.gitignore (100%) rename LICENSE => prototypes/cf-worker-prototype/LICENSE (100%) create mode 100644 prototypes/cf-worker-prototype/README.md rename {assets => prototypes/cf-worker-prototype/assets}/images/arch.png (100%) rename {assets => prototypes/cf-worker-prototype/assets}/images/filmstrip.png (100%) rename {assets => prototypes/cf-worker-prototype/assets}/images/sequence.svg (100%) rename package-lock.json => prototypes/cf-worker-prototype/package-lock.json (100%) rename package.json => prototypes/cf-worker-prototype/package.json (100%) rename {src => prototypes/cf-worker-prototype/src}/client/index.ts (100%) rename {src => prototypes/cf-worker-prototype/src}/config.ts (100%) rename {src => prototypes/cf-worker-prototype/src}/index.ts (100%) rename {src => prototypes/cf-worker-prototype/src}/lib/ab-random-variant.ts (100%) rename {src => prototypes/cf-worker-prototype/src}/lib/ab-variant-selection.ts (100%) rename {src => prototypes/cf-worker-prototype/src}/lib/constants.ts (100%) rename {src => prototypes/cf-worker-prototype/src}/lib/pre-ua-transform-applicator.ts (100%) rename {src => prototypes/cf-worker-prototype/src}/lib/transformer.ts (100%) rename {src => prototypes/cf-worker-prototype/src}/lib/ua-transform-applicator.ts (100%) rename tsconfig.json => prototypes/cf-worker-prototype/tsconfig.json (100%) rename wrangler.toml => prototypes/cf-worker-prototype/wrangler.toml (100%) rename EXPLAINER.md => specs/EXPLAINER.md (90%) diff --git a/README.md b/README.md index d2a0993..84c42eb 100644 --- a/README.md +++ b/README.md @@ -1,88 +1,36 @@ -# ab-worker-prototype +# Client A/B testing -A proof of concept implementation of the performant client-side A/B testing specification [as outlined in `EXPLAINER.md`](/EXPLAINER.md), using a CloudFlare Edge Worker. +Client A/B testing refers to the method of performing experimentation related +changes to a web application at runtime, typically within a browser. +This method of experimentation is popular in the industry as it is easier to +deploy, requires minimal to no engineering bandwidth for creating experiments +and is accessible to non-engineering personnel as well. -## Architecture +This incubation's objective is to devise methods of conducting the same outcome +with all its benefits offered, but without the performance penalties. -![architecture-diagram](/assets/images/arch.png) +## Goals -To implement a proof of concept, we will use a CloudFlare Worker to apply the `PRE_UA` transforms. Alternatively, this could be any compute node that can act as a proxy — a connect middleware for an Express.js frontend application, a proxy or an Edge at the boundaries of an Origin, or a CDN with compute capabilities (like CF workers). The proxy should be able to intercept a request to the origin, minimally parse and modify the HTML and apply the transformations. + * Standardize A/B transformation operations to a spec. + * Enable application of experimentation changes at the Origin, + an intermediary CDN/Edge/Proxy or the Browser. + * Minimize or eliminate performance metrics degradation caused by + application of changes in the browser through performant methods and + through exploration of browser-native implementations. -Internally, CloudFlare production is [said to](https://blog.cloudflare.com/html-parsing-1/) use [lol-html](https://github.com/cloudflare/lol-html), a low-latency HTML parser that supports streaming. A non-CloudFlare version of this prototype will have to employ a similar mechanism with same performance characteristics for this to function at scale. +## Explainers and presentations -For the A/B configuration provider, we will make a pseudo-API out of GitHub gists — this would allow us to test multiple sites and variations without redeploying the CloudFlare worker. This will also mimic the real world dependency of an API call federation off the Edge/CDN/Server. We will use stale-while-revalidate (SWR) policy for the configuration fetch, which should be acceptable for most websites’ real A/B tests. + * [Explainer and supported operations](specs/EXPLAINER.md) + * [WICG: A/B testing: updates - March 2023](https://docs.google.com/presentation/d/1WX-E63jL7ZwGf_jNszhfkdxsvzlXLdJdPMSTxK3X0A0/edit?usp=sharing) + * [PerfWG: Client-side A/B testing - March 2022](https://docs.google.com/presentation/d/1-cxHITwVtWJ5x3ev0__XzDtDtJn2cB9CAgN9Mkia3Ag/edit?usp=sharing) + * [Incubation proposal](https://github.com/WICG/proposals/issues/54) -In addition to that, the prototype edge also implements the following essential A/B testing functions: - * Figure out if an A/B test configuration is in place for the requested URI, via consulting the pseudo API. - * Randomly select the request into an experiment based on the configuration, and set a cookie to make the selection sticky. +## Directory structure -## Sequence diagram + * `specs/` folder ontains the latest specification and explainers. + * `sdks/` contains sdks, libraries, etc. + * `prototypes/` hosts current prototypes, demos and sdks. -![Sequence diagram](/assets/images/sequence.svg) +## Contributing -## Demo - -For a demo, we're going to A/B test ToDoMVC, that is built using React as a client-side SPA. - -The following A/B test configuration consists of transformations that create a randomized experiment for 50% of the traffic: - * Change background-color of the page to _`beige`_. Since this is best done on the static markup, we'd like to do have this transformation applied before UA. - * Change heading (H1) to "_a/b test h1_". - * Change the placeholder text of the text box to "_What would you like to do today?_" - * Color the first todo item _`red`_. - -In addition, we'll also inject correct `` tag to make relative URLs work, and `` tag to avoid getting our prototype indexed as a duplicate. - -```javascript -const experimentConfigJson = { - "control": { - "url": "https://todomvc.com/examples/react", - "cache": { - } - }, - "variants": [ - { - "weight": 0.5, - "url": "https://todomvc.com/examples/react", - "transformations": [ - [1 /* PRE_UA */, "head", 3 /* OP_PREPEND */, - ""], - [1 /* PRE_UA */, "head", 3 /* OP_PREPEND */, - ""] - ] - }, - { - "weight": 0.5, - "url": "https://todomvc.com/examples/react", - "transformations": [ - [1 /* PRE_UA */, "head", 3 /* OP_PREPEND */, - ""], - [1 /* PRE_UA */, "head", 3 /* OP_PREPEND */, - ""], - [1 /* PRE_UA */, "head", 4 /* OP_APPEND */, - ""], - [2 /* ON_UA */, "h1", 0 /* OP_CUSTOM_JS */, - "$.innerHTML=\"a/b test h1\";"], - [2 /* ON_UA */, ".new-todo", 0 /* OP_CUSTOM_JS */, - "$.placeholder=\"What would you like to do today?\""], - [2 /* ON_UA */, ".todo-list>li", 0 /* OP_CUSTOM_JS */, - "$.style.color=\"red\""] - ] - } - ] -} -``` - -The prototype deployed on CloudFlare Edge can be accessed at: - * [Control](https://ab-worker.alexnj.workers.dev/?experiment=todomvc-v01.json&force=0) - * [Experiment](https://ab-worker.alexnj.workers.dev/?experiment=todomvc-v01.json&force=1) - * [Select one of them at random](https://ab-worker.alexnj.workers.dev/?experiment=todomvc-v01.json) - -Configuration for running this experiment is [hosted as a gist](https://gist.github.com/alexnj/4c8d9198d16b238e4c7040250f052284#file-todomvc-v01-json). - -### Performance comparison - -#### WebPageTest - -![Filmstrip](/assets/images/filmstrip.png) - -[Comparison with more stats here](https://webpagetest.org/video/compare.php?tests=220128_AiDcKT_e71a33f6dd31af1157630f95377bbb4c,220128_BiDcH4_8af7080f26f4bc2f030ccce7e2695045). \ No newline at end of file +Please see [How to Contribute](CONTRIBUTING.md). diff --git a/.gitignore b/prototypes/cf-worker-prototype/.gitignore similarity index 100% rename from .gitignore rename to prototypes/cf-worker-prototype/.gitignore diff --git a/LICENSE b/prototypes/cf-worker-prototype/LICENSE similarity index 100% rename from LICENSE rename to prototypes/cf-worker-prototype/LICENSE diff --git a/prototypes/cf-worker-prototype/README.md b/prototypes/cf-worker-prototype/README.md new file mode 100644 index 0000000..31998ef --- /dev/null +++ b/prototypes/cf-worker-prototype/README.md @@ -0,0 +1,88 @@ +# ab-worker-prototype + +A proof of concept implementation of the performant client-side A/B testing specification [as outlined in `EXPLAINER.md`](/EXPLAINER.md), using a CloudFlare Edge Worker. + +## Architecture + +![architecture-diagram](assets/images/arch.png) + +To implement a proof of concept, we will use a CloudFlare Worker to apply the `PRE_UA` transforms. Alternatively, this could be any compute node that can act as a proxy — a connect middleware for an Express.js frontend application, a proxy or an Edge at the boundaries of an Origin, or a CDN with compute capabilities (like CF workers). The proxy should be able to intercept a request to the origin, minimally parse and modify the HTML and apply the transformations. + +Internally, CloudFlare production is [said to](https://blog.cloudflare.com/html-parsing-1/) use [lol-html](https://github.com/cloudflare/lol-html), a low-latency HTML parser that supports streaming. A non-CloudFlare version of this prototype will have to employ a similar mechanism with same performance characteristics for this to function at scale. + +For the A/B configuration provider, we will make a pseudo-API out of GitHub gists — this would allow us to test multiple sites and variations without redeploying the CloudFlare worker. This will also mimic the real world dependency of an API call federation off the Edge/CDN/Server. We will use stale-while-revalidate (SWR) policy for the configuration fetch, which should be acceptable for most websites’ real A/B tests. + +In addition to that, the prototype edge also implements the following essential A/B testing functions: + * Figure out if an A/B test configuration is in place for the requested URI, via consulting the pseudo API. + * Randomly select the request into an experiment based on the configuration, and set a cookie to make the selection sticky. + +## Sequence diagram + +![Sequence diagram](assets/images/sequence.svg) + +## Demo + +For a demo, we're going to A/B test ToDoMVC, that is built using React as a client-side SPA. + +The following A/B test configuration consists of transformations that create a randomized experiment for 50% of the traffic: + * Change background-color of the page to _`beige`_. Since this is best done on the static markup, we'd like to do have this transformation applied before UA. + * Change heading (H1) to "_a/b test h1_". + * Change the placeholder text of the text box to "_What would you like to do today?_" + * Color the first todo item _`red`_. + +In addition, we'll also inject correct `` tag to make relative URLs work, and `` tag to avoid getting our prototype indexed as a duplicate. + +```javascript +const experimentConfigJson = { + "control": { + "url": "https://todomvc.com/examples/react", + "cache": { + } + }, + "variants": [ + { + "weight": 0.5, + "url": "https://todomvc.com/examples/react", + "transformations": [ + [1 /* PRE_UA */, "head", 3 /* OP_PREPEND */, + ""], + [1 /* PRE_UA */, "head", 3 /* OP_PREPEND */, + ""] + ] + }, + { + "weight": 0.5, + "url": "https://todomvc.com/examples/react", + "transformations": [ + [1 /* PRE_UA */, "head", 3 /* OP_PREPEND */, + ""], + [1 /* PRE_UA */, "head", 3 /* OP_PREPEND */, + ""], + [1 /* PRE_UA */, "head", 4 /* OP_APPEND */, + ""], + [2 /* ON_UA */, "h1", 0 /* OP_CUSTOM_JS */, + "$.innerHTML=\"a/b test h1\";"], + [2 /* ON_UA */, ".new-todo", 0 /* OP_CUSTOM_JS */, + "$.placeholder=\"What would you like to do today?\""], + [2 /* ON_UA */, ".todo-list>li", 0 /* OP_CUSTOM_JS */, + "$.style.color=\"red\""] + ] + } + ] +} +``` + +The prototype deployed on CloudFlare Edge can be accessed at: + * [Control](https://ab-worker.alexnj.workers.dev/?experiment=todomvc-v01.json&force=0) + * [Experiment](https://ab-worker.alexnj.workers.dev/?experiment=todomvc-v01.json&force=1) + * [Select one of them at random](https://ab-worker.alexnj.workers.dev/?experiment=todomvc-v01.json) + +Configuration for running this experiment is [hosted as a gist](https://gist.github.com/alexnj/4c8d9198d16b238e4c7040250f052284#file-todomvc-v01-json). + +### Performance comparison + +#### WebPageTest + +![Filmstrip](assets/images/filmstrip.png) + +[Comparison with more stats here](https://webpagetest.org/video/compare.php?tests=220128_AiDcKT_e71a33f6dd31af1157630f95377bbb4c,220128_BiDcH4_8af7080f26f4bc2f030ccce7e2695045). diff --git a/assets/images/arch.png b/prototypes/cf-worker-prototype/assets/images/arch.png similarity index 100% rename from assets/images/arch.png rename to prototypes/cf-worker-prototype/assets/images/arch.png diff --git a/assets/images/filmstrip.png b/prototypes/cf-worker-prototype/assets/images/filmstrip.png similarity index 100% rename from assets/images/filmstrip.png rename to prototypes/cf-worker-prototype/assets/images/filmstrip.png diff --git a/assets/images/sequence.svg b/prototypes/cf-worker-prototype/assets/images/sequence.svg similarity index 100% rename from assets/images/sequence.svg rename to prototypes/cf-worker-prototype/assets/images/sequence.svg diff --git a/package-lock.json b/prototypes/cf-worker-prototype/package-lock.json similarity index 100% rename from package-lock.json rename to prototypes/cf-worker-prototype/package-lock.json diff --git a/package.json b/prototypes/cf-worker-prototype/package.json similarity index 100% rename from package.json rename to prototypes/cf-worker-prototype/package.json diff --git a/src/client/index.ts b/prototypes/cf-worker-prototype/src/client/index.ts similarity index 100% rename from src/client/index.ts rename to prototypes/cf-worker-prototype/src/client/index.ts diff --git a/src/config.ts b/prototypes/cf-worker-prototype/src/config.ts similarity index 100% rename from src/config.ts rename to prototypes/cf-worker-prototype/src/config.ts diff --git a/src/index.ts b/prototypes/cf-worker-prototype/src/index.ts similarity index 100% rename from src/index.ts rename to prototypes/cf-worker-prototype/src/index.ts diff --git a/src/lib/ab-random-variant.ts b/prototypes/cf-worker-prototype/src/lib/ab-random-variant.ts similarity index 100% rename from src/lib/ab-random-variant.ts rename to prototypes/cf-worker-prototype/src/lib/ab-random-variant.ts diff --git a/src/lib/ab-variant-selection.ts b/prototypes/cf-worker-prototype/src/lib/ab-variant-selection.ts similarity index 100% rename from src/lib/ab-variant-selection.ts rename to prototypes/cf-worker-prototype/src/lib/ab-variant-selection.ts diff --git a/src/lib/constants.ts b/prototypes/cf-worker-prototype/src/lib/constants.ts similarity index 100% rename from src/lib/constants.ts rename to prototypes/cf-worker-prototype/src/lib/constants.ts diff --git a/src/lib/pre-ua-transform-applicator.ts b/prototypes/cf-worker-prototype/src/lib/pre-ua-transform-applicator.ts similarity index 100% rename from src/lib/pre-ua-transform-applicator.ts rename to prototypes/cf-worker-prototype/src/lib/pre-ua-transform-applicator.ts diff --git a/src/lib/transformer.ts b/prototypes/cf-worker-prototype/src/lib/transformer.ts similarity index 100% rename from src/lib/transformer.ts rename to prototypes/cf-worker-prototype/src/lib/transformer.ts diff --git a/src/lib/ua-transform-applicator.ts b/prototypes/cf-worker-prototype/src/lib/ua-transform-applicator.ts similarity index 100% rename from src/lib/ua-transform-applicator.ts rename to prototypes/cf-worker-prototype/src/lib/ua-transform-applicator.ts diff --git a/tsconfig.json b/prototypes/cf-worker-prototype/tsconfig.json similarity index 100% rename from tsconfig.json rename to prototypes/cf-worker-prototype/tsconfig.json diff --git a/wrangler.toml b/prototypes/cf-worker-prototype/wrangler.toml similarity index 100% rename from wrangler.toml rename to prototypes/cf-worker-prototype/wrangler.toml diff --git a/EXPLAINER.md b/specs/EXPLAINER.md similarity index 90% rename from EXPLAINER.md rename to specs/EXPLAINER.md index e43667d..210deb5 100644 --- a/EXPLAINER.md +++ b/specs/EXPLAINER.md @@ -4,15 +4,15 @@ Client side A/B testing refers to the method of performing experimentation relat ## Overview -The key to the approach being outlined here is considering **_control_** as the base document, and expressing each **_variant_** as a series of transformations being applied onto control. +The key to the approach being outlined here is considering **_control_** as the base document, and expressing each **_variant_** as a series of transformations being applied onto control. -This is analogous to how client side A/B testing is conducted today — except we want to improve it further and make it more performant. How do we do that? +This is analogous to how client side A/B testing is conducted today — except we want to improve it further and make it more performant. How do we do that? 1. Standardize the serialization of changes to a common schema. -2. Whenever possible, optimize for applying a transformation where it makes the best sense (pre-UA, or on-UA). This helps to: +2. Whenever possible, optimize for applying a transformation where it makes the best sense (pre-UA, or on-UA). This helps to: - a. reduce the bytes transferred, + a. reduce the bytes transferred, b. potentially improve cache hit rates at serving c. reduce computation needs on each client. @@ -36,8 +36,8 @@ where each field can be defined as follows flags - A bit field that indicates the type of transformation. - For an initial version, we can support the following flags: + A bit field that indicates the type of transformation. + For an initial version, we can support the following flags: - - - @@ -116,7 +116,7 @@ where each field can be defined as follows @@ -127,7 +127,7 @@ Arguments: @@ -138,7 +138,7 @@ Arguments:
0x1 @@ -45,7 +45,7 @@ where each field can be defined as follows PRE_UA Transform can be applied on the server, or prior to UA parsing. -
It could be done at a CDN/Edge, or at the Origin server itself. +
It could be done at a CDN/Edge, or at the Origin server itself.

Most of the static document transformations would use this flag. @@ -76,27 +76,27 @@ where each field can be defined as follows
selectorA CSS selector that targets the `HTMLElement` for transformation. - Depending on the target platform and capability, only a subset of - CSS selectors might be applicable here (and that needs to be + A CSS selector that targets the `HTMLElement` for transformation. + Depending on the target platform and capability, only a subset of + CSS selectors might be applicable here (and that needs to be documented and revisioned as the support changes).
operationA numeric value indicating the operation to be performed. - For a list of operations and their supported parameters, see the section + A numeric value indicating the operation to be performed. + For a list of operations and their supported parameters, see the section "Operations" below.
- The operations are expected to be idempotent, i.e., repeated - applications of the operations should be possible and not have + The operations are expected to be idempotent, i.e., repeated + applications of the operations should be possible and not have unintended side effects.
payloadA variable number of arguments to support the operation. + A variable number of arguments to support the operation. Should follow the specification of the operation used.
Executes a custom Javascript block of code against the element.

-Arguments: +Arguments: `code`: Javascript code serialized as a string. The applicator code will call this code as a function ($) => code, $ referring to the element selected.

Inserts content right before the element.

-Arguments: +Arguments: `content`: HTML markup

Inserts content right after the element.

-Arguments: +Arguments:

`content`: HTML markup @@ -150,7 +150,7 @@ Arguments:

Inserts content right after the start tag of the element.

-Arguments: +Arguments:

`content`: HTML markup @@ -162,7 +162,7 @@ Arguments:

Inserts content right before the end tag of the element.

-Arguments: +Arguments:

`content`: HTML markup @@ -174,7 +174,7 @@ Arguments:

Replaces the element with the provided content.

-Arguments: +Arguments:

`content`: HTML markup @@ -186,7 +186,7 @@ Arguments:

Replaces the content of the element with provided content.

-Arguments: +Arguments:

`content`: HTML markup @@ -210,7 +210,7 @@ none.

Sets an attribute’s value on the element.

-Arguments: +Arguments:

`name`: Name of the attribute @@ -224,7 +224,7 @@ Arguments:

Redirect the user to a different page or URL.

-Arguments: +Arguments:

`URL`: URL to redirect the page to. @@ -242,12 +242,12 @@ Arguments: ## `PRE_UA` components A `PRE_UA` component is responsible for applying the transformations flagged as `PRE_UA`, i.e., the transforms that make best sense to be applied before User-Agent. This could be done at: -1. the Origin itself, or +1. the Origin itself, or 2. an Edge component at the Origin (like a web server plugin, or a proxy), or -3. a CDN compute node that fronts the Origin as a proxy, or -4. an implementation inside the UA/Browser prior to parsing the document (this has a latency impact). +3. a CDN compute node that fronts the Origin as a proxy, or +4. an implementation inside the UA/Browser prior to parsing the document (this has a latency impact). -The important aspect is that the `PRE_UA` transformations are best done at a step prior to UA’s parsing. +The important aspect is that the `PRE_UA` transformations are best done at a step prior to UA’s parsing. `PRE_UA` component has the following responsibilities: @@ -259,13 +259,13 @@ The important aspect is that the `PRE_UA` transformations are best done at a ste The client-side/`ON_UA` components are responsible for applying `ON_UA` transformations as necessary, and consists of two parts: 1. The remaining `ON_UA` transformations from the experiment configuration. -2. The client side transformation applicator code. +2. The client side transformation applicator code. -## Client-side transform applicator +## Client-side transform applicator The applicator component injected into the `HEAD` of the document has the following parts to it: -* A `MutationObserver` client code that listens for DOM changes. +* A `MutationObserver` client code that listens for DOM changes. * The listening code looks for DOM changes that match the `ON_UA` selectors, as (1) the browser is parsing the document, or (2) the client side Javascript is making changes to the DOM. * One or more identified DOM mutations that match the selectors are queued and deferred for processing until the next repaint, via a `requestAnimationFrame` callback. * The queue is processed, applying each `ON_UA` transformation as needed — via running the matching `DOMElement` through the supplied transformation function. @@ -318,7 +318,7 @@ This block results in approximately 365 bytes of minified code when processed vi # References * [WebPerfWG Open meeting notes 2021-02-04](https://w3c.github.io/web-performance/meetings/2021/2021-02-04/index.html) -* [A/B Testing at the Edge with Servers Workers](https://www.filamentgroup.com/lab/servers-workers.html) +* [A/B Testing at the Edge with Servers Workers](https://www.filamentgroup.com/lab/servers-workers.html) * [Performant A/B Testing with Cloudflare Workers](https://philipwalton.com/articles/performant-a-b-testing-with-cloudflare-workers/) * [A History of HTML Parsing at Cloudflare: Part 1](https://blog.cloudflare.com/html-parsing-1/) * [The Case Against Anti-Flicker Snippets](https://andydavies.me/blog/2020/11/16/the-case-against-anti-flicker-snippets/)