Extend WebClient of HtmlUnit/XLT to allow better header setup

[rschwietzke]

Goal

During testing, one often faces the challenge to deal with HTTP headers. This is often due to debugging needs or due to test restrictions, such as inserting custom headers to control network behavior, removing headers, or changing values.

It is preferred to make that an HtmlUnit change in its WebClient instead of just changing the XltWebClient. That way, even when one bypasses the XltWebClient due to direct access of HtmlUnit's transport classes, the functionality is still in place.

It is important that all headers can be modified easily. Currently, HtmlUnit/WebClient seem to treat some headers differently and overriding them is hard.

When talking about headers, we implicitly also mean cookies, so we want to apply this logic to cookies as well.

Non-Goal

We don't want to rewrite the WebClient nor make it extremely more flexible. These changes are just here to aid any testing in the context of performance or other activities.

Functionality

When using a WebClient for performance testing, one is often using it for more than one request. Header modifications often apply to all requests, including but not limited to indirect requests that are done by HtmlUnit such as frame handling or subresource loading.

So, we want to be able to set up all desired headers upfront to get them into all following requests, but we also want to be able to modify a header for a single request only, which is not affecting any later requests.

The examples below are just first API ideas and not meant to be the final state.

Add Headers

• We want to add a header for good aka once and it will be always there.
• We also want to add a header just for the next request.

In both cases, we want to be able to run a "do only when" handling. In this case, it is likely something like:

• Set header only of not yet set
• Set header always, override when needed

Preferably, we use lambas for that to make it elegant and flexible.

Modify Headers

Similarly to setting a header, we also want to apply logic to an existing header to modify it in any shape or form.

Remove Headers

And of course, we want to remove any header (also repeatably), also based on conditions, such as when a certain value exists, we remove it, otherwise it stays or we completly remove it despite any state.

Only-If/Do-when

For all header modification actions, we would like to be able to decide if and how we modify things. That often might be easily doable be providing a lamba-powered interface.

Cookies

The previous ideas should also apply to cookies. Because cookies are state that might change, we have to be even more careful but also require some more data such as domain we are working on, to make a call if this cookie has to stay, go, added, or modified.

Existing XLT Features

Existing features of XLT, such as the custom user agent, the request header id should be rewritten to use that new functionality to test it as well as check that the API design makes sense.

Limitations

HtmlUnit tries to by JDK 8 compliant, which might be a reason to keep certain nifty functionality in the XLT layer which is JDK 17 compliant or soon JDK 21.

Also, header and cookie handling heavily interfere with the application state and logic. This might break things but this is in the responsibility of the user. XLT and HtmlUnit are tools for testing and other low-level engineering tasks, hence we can imply certain knowledge about this concepts when one interacts with headers and cookies.

So let's conclude with a final warning: Use these new features with caution and at your own risk. If you don't know what you do and why, don't do it, especially when applying global overrides. Also, make sure that you don't break the application state unless you want to do that to provoke certain behavior, in that case, be our guest.