Using Rock in Dream to share middlewares [factor out sublibraries]

[tmattio]

Hi @aantron!

Thanks a lot for your amazing work on Dream, I'm really excited to see a full-featured OCaml web server and can't wait to try it for a side project 😄

I wanted to take the pulse and see how you felt about using Rock (Opium's low-level HTTP and middleware layer) in Dream.

To give a bit of context, we extracted Rock from Opium and made it as minimalist as possible in the hope that it could be re-used by other Web frameworks in the community.
Our motivation was to provide a common middleware layer to all of the frameworks so that users and frameworks could re-use them. (In the spirit of Rack from Ruby)

If that seems to make sense to you, there would be some work needed on our side, as Rock does not support Http2 and Web sockets at the moment, but that's things we wanted to work on anyways.
I've also seen #4 and that's something I wanted to explore as well. All in all, there seem to be some overlaps and I'd love to see a collaboration happen to consolidate OCaml's web ecosystem.

Cheers and thanks again for all your work!

[aantron]

Hi. Thanks!

I've given using Rock some thought. The Dream project itself grew in part out of a collection of private Opium middlewares, and I originally hoped to be able to combine what-became-Dream-style middlewares with both public Opium/Rock middlewares, and the private ones I had already written. However, Dream ended up diverging enough that I found it easier to just rewrite them over Dream, for my own purposes.

What would using Rock in Dream entail? To my mind (and I may well be missing much), Rock consists mainly of

1. An HTTP server adapter (i.e. a wrapper over http/af).
2. A body streaming module.
3. Several data type definitions (e.g. app, middleware).

Dream has these things as well, but

1. The HTTP server adapter also wraps h2, websocketaf, and Lwt_ssl. This is good because a web framework "consumes" this part, so Rock can be built over Dream's HTTP adapter without losing functionality.
2. A body streaming module which AFAICT is lower-level (i.e. single-shot promises and/or continuations, no Lwt_stream.t). This is again good, because a web framework "provides" this part, and Rock can build over this without having to deal with extra assumptions.
3. Similar data type definitions, but which are again lower-level (middlewares as just functions, etc.). Again, Rock should be able to get Rock middlewares from this by putting Dream middlewares into Middleware.t records, etc.

So in summary, it appears to me that Dream varies everything in the "right" direction ("contravariant" in 1, "covariant" in 2 and 3) to be a lower layer. (I want to note however, that I personally think that Dream's is the right layer for end-users to program at in the web framework case, because it introduces fewer concepts, some of which are, I think, not necessary).

So at first glance, if I wanted to make Rock and Dream compatible, I would implement a version of Rock as an adapter to Dream.

The main obstacle is probably that Rock requests and responses are not abstract, but their fields are known, so there would have to be a translation at the point where one enters a Rock middleware, then again on the way out. That unfortunately could mean up to four translations :/

Is there a large set of Rock middleware out there? If not too much, it may be easier to simply port, as Dream is quite similar to Rock in most of the respects that are directly relevant to middleware in particular. In fact, it should be slightly easier to program middleware for Dream than for Rock — this was one of the major goals of the project.

[aantron]

Thinking about it more, Dream really is extremely low-level. It fundamentally has only two main abstract types, request and response.

handler and middleware are just bare functions defined over those, and route is specific to whatever router is being used — Dream just has a built-in one that it offers users to start with (cc @anuragsoni).

The two "real" abstract types are created/consumed by the HTTP layer, which we may make replaceable in #4 (also as originally intended). All of Dream's built-in functionality can be turned off by passing ~builtins:false to the HTTP layer (Dream.run), resulting in an extremely low-level machine that just hides http/af+h2+websocketaf+lwt_ssl (or whatever stack) behind request and response and does nothing else, especially if you choose to never call into any of Dream's higher-level functions.

So Dream may already be somewhat lower-level than Rock. It's just maybe not immediately apparent because of all the other helpers and defaults present in the API, that trigger higher-level behaviors.

[tmattio]

Thanks for the detailed and thoughtful answer :)

My first thought reading this is that we missed something in the design of Rock. Indeed, the sole purpose of Rock is to be as low level as possible while also providing a layer of abstraction that would allow web servers to share building blocks (i.e. middlewares).

If a server such as Dream seems to be at a lower abstraction level than Rock, we probably want to revisit a few things in Rock 😉

A body streaming module which AFAICT is lower-level (i.e. single-shot promises and/or continuations, no Lwt_stream.t). This is again good, because a web framework "provides" this part, and Rock can build over this without having to deal with extra assumptions.

That's a good point - we have been thinking of revisiting the body streaming for a while to not use Lwt_stream. (cc @anuragsoni)

Similar data type definitions, but which are again lower-level (middlewares as just functions, etc.).

Rock is supposed to be only that: a bunch of data type definition (wrapping httpaf ones for requests and responses). I'm not sure what makes Rock's definition higher level, but we can simplify them if needed.

The reason I'm thinking it should be the other way around (that Dream could wrap Rock) is that there's a bunch of high level dependency in Dream that a thin abstraction layer like Rock should not have to impose on users (e.g. graphql, caqti, etc.).
And if we wanted to extract this low-level layer from Dream in another package, we would end up with exactly what Rock aims to be.

All of this is really good feedback for me, and I think the limitations you've seen in Rock have more to do with me not ironing out the implementation rather than Rock's goal not aligning with what you were looking for.

I certainly don't want to push for using Rock in Dream if that does not align with the direction you want to take for the project, so don't hesitate to tell me so :)
But if having a common layer to Opium, Dream, Sihl, etc. seems like a sensible idea to you and it's a matter of implementation constraints, I'll be happy to work on addressing the bottlenecks in Rock (updating the Body handling, supporting h2 and websockets, updating the types?)

[tmattio]

Is there a large set of Rock middleware out there?

As far as I know, there are Opium's: https://github.com/rgrinberg/opium/tree/master/opium/src/middlewares
And Sihl's: https://github.com/oxidizing/sihl/tree/master/sihl/src

But the list will most likely grow as I wanted to provide middlewares for e.g. compression, security, etc.

[aantron]

I think the fastest way to get a layer like this would be to factor out a dream-pure.opam, which already exists as an internal library in src/pure. That would give the "Rock"-like layer.

Separately factoring out internal library src/http into a public dream-httpaf.opam would also make the internal stack replaceable. cc @dinosaure

I programmed Dream to avoid all hard dependencies like the ones you mentioned. The one package dream is just a convenience for end-users, but if you look in the directory structure, Caqti, GraphQL, all middlewares, are all and each separated into their own sublibraries with their own dependencies. Even the sql_sessions and memory_sessions middlewares come from different sublibraries, although a lot more could be done to untangle e.g. cryptography — I didn't bother to untangle it, but I also kept it somewhat tidy so that it could be done later.

The templater, is, of course, also separated. In fact, I wanted to publish it separately for general use, but I didn't want to spend a lot of time designing a general-use templater before even the first release of Dream :)

I do think a common layer would be useful, but it seems much easier to use the pieces of Dream, than to mutate Rock into the same thing :) At least to be usable as a lower level for Dream, Rock would need to make its types abstract, eliminate concepts like applications and services, eliminate S-expressions from contexts, unwrap the middleware records and make middlewares into bare functions, and then implement the h2+ stack. At that point, it would be exactly what Dream is in src/pure and src/http, anyway, modulo having been written slightly differently :)

[tmattio]

I agree, it would definitely be more work to update Rock with the things you have listed. FWIW, it's all things we wanted to do, but if it is already implemented in Dream and distributed as a library, it probably makes sense to use it 🙂

If we do this, however, Rock becomes somehow irrelevant and Opium and other frameworks can simply use the HTTP layer of Dream directly.

I see two things to consider for this:

• Rock also intended to offer an abstraction to build HTTP clients. In this, it is inspired by Sinatra, hence the filter and service types. This does not seem compatible with Dream?
• Rock was developed for other framework maintainers, so adapting to fit the community's needs is part of its purpose. I'm not sure that's a goal you'll want to define for the low-level layer of Dream?

I also quite liked the idea of having a separate project for this low-level layer (as again, the goal was to be framework agnostic), not sure what would be your preference here?

I really appreciate you taking your time to discuss this and brainstorming on the solutions 🙌

[aantron]

• Rock also intended to offer an abstraction to build HTTP clients. In this, it is inspired by Sinatra, hence the filter and service types. This does not seem compatible with Dream?

I haven't looked at clients in any detail yet, but it is on the big to-do list :) At the very least, some middlewares may need to read enough of a response that they are almost clients themselves. In addition, I also considered that Web apps may need to make their own requests, or act like proxies. So, I'm probably not ready to answer this, and perhaps it's not sustainable to eliminate filter and/or service.

Rock was developed for other framework maintainers, so adapting to fit the community's needs is part of its purpose. I'm not sure that's a goal you'll want to define for the low-level layer of Dream?

To the extent it is still compatible with Dream's model (which is very simple), yes — I've documented the initial version of using Dream as a very simple request to response machine already :) And it was always the intent to make Dream portable to different stacks (part of the reason for the abstract types). I probably wouldn't follow the community needs for something that works in a completely different way — but there would naturally then be a different web framework or set of frameworks built around that.

I also quite liked the idea of having a separate project for this low-level layer (as again, the goal was to be framework agnostic), not sure what would be your preference here?

That seems fine in principle. From Dream's point of view, I think the best thing to do would be to actually factor out dream-pure (and factor cryptography out of it) in, say, alpha2, and then see where to go from there.

[aantron]

I really appreciate you taking your time to discuss this and brainstorming on the solutions 🙌

Likewise in return :)

[tmattio]

From Dream's point of view, I think the best thing to do would be to actually factor out dream-pure (and factor cryptography out of it) in, say, alpha2, and then see where to go from there.

That sounds great, there's no rush on my side to do this :)

In this case, once these pieces are factored out, I'll try to use them in Opium instead of Rock and will be able to give some initial feedback. We'll have a better idea of the next steps once we're there I guess 🙂

[anuragsoni]

The main obstacle is probably that Rock requests and responses are not abstract, but their fields are known, so there would have to be a translation at the point where one enters a Rock middleware, then again on the way out. That unfortunately could mean up to four translations :/

If i could do things over again I'd have pushed forward with my intention to make these types abstract as part of the breaking changes done to opium in porting it to httpaf. I feel like I missed an opportunity there in refining certain things when I had the chance to do so 😢

@tmattio That's a good point - we have been thinking of revisiting the body streaming for a while to not use Lwt_stream.
@aantron A body streaming module which AFAICT is lower-level (i.e. single-shot promises and/or continuations, no Lwt_stream.t). This is again good, because a web framework "provides" this part, and Rock can build over this without having to deal with extra assumptions.

This sounds excellent. Single shot promises is what I've been exploring as a streaming option as I wanted to move Rock away from Lwt_stream for a while rgrinberg/opium#218 was a rough start on this, but I haven't finished up explorations here as I got busy with other tasks.

@aantron I do think a common layer would be useful, but it seems much easier to use the pieces of Dream, than to mutate Rock into the same thing :) At least to be usable as a lower level for Dream, Rock would need to make its types abstract, eliminate concepts like applications and services

I'd be interested in hearing more about this point. Re, applications and services, i'm not sure many people (or anyone?) uses them directly in Rock today? I'd expect most user code to be using the Handler and Middleware module which at a cursory glance look similar to Dream's choice of req -> response promise.

@aantron middlewares into bare functions, and then implement the h2+ stack. At that point, it would be exactly what Dream is in src/pure and src/http, anyway, modulo having been written slightly differently :)

FWIW one reason the middlewares in Rock are a record (name + function) is because we allow for some introspection over the CLI which allows a user to query for the middlewares mounted to an app, and the name also shows up during debugging to get some more insights into which middlewares get fired. I kind of like the ability of being able to add some sort of unique identifier to tag a middleware but there are obviously more ways to achive this while keeping middlewares are simple functions :)

So at first glance, if I wanted to make Rock and Dream compatible, I would implement a version of Rock as an adapter to Dream.

👍🏼

Thank you both @tmattio and @aantron for the wonderful discussion. I haven't done a lot of recent work on Opium, but I really like what I see in Dream so far, and its very nice to see http/2, graphql, websockets etc supported in Dream 🎉 . My 2 cents would be to explore options to see how Rock can benefit from Dream. It might be nice to have an Opium library powered by Dream for users that will prefer the builder/cli experience that opium offers, and maybe we can learn from the current experience from opium and make futher refinements :)

I'll also love to help out in any way I can to help round out the web story for OCaml once Dream is ready for external contributors!

[anuragsoni]

I haven't looked at clients in any detail yet, but it is on the big to-do list :) At the very least, some middlewares may need to read enough of a response that they are almost clients themselves. In addition, I also considered that Web apps may need to make their own requests, or act like proxies. So, I'm probably not ready to answer this, and perhaps it's not sustainable to eliminate filter and/or service.

The way I envisioned client support via Rock should be possible in dream too (without needing to have the user learn about filters/services etc). I mostly envisioned being able to write "drivers" that can go from req -> res promiseas from a user's perspective writing a middleware for both server and client should be a fairly similar process? I've mostly used this similarly to write certain logging, timeout, metrics etc tasks that follow a fairly similar pattern for both server handlers and clients.

[aantron]

@anuragsoni Thanks!

Regarding introspection, since Dream has a "first-class" router, and the routes are "fat" objects the way Opium middlewares are "fat," my plan was to add introspection to the router. In Opium to date, routing is done syntactically by middleware-like things (at least visually), so it's natural to "shoehorn" this kind of functionality into middlewares in Opium. But actually routes are fundamentally dual to middlewares (they are +-like, middlewares are *-like). I put a comment about it in the docs, for people who like algebra:

The three handler-related types have a vaguely algebraic interpretation:

• Literal handlers are atoms.
• middleware is for sequential composition (product-like).
• route is for alternative composition (sum-like).

Dream.scope implements a distributive law.

And, the reality is, I think, most apps will have only a few completely global middlewares. Most middlewares will be scoped to a subset of the routes using Dream.scope, so the vast majority of the site's structure will be inside the routing DSL, which, as I already sketched, is introspection-friendly at least for the routes (not the intervening middlewares).

In summary, I am moving introspection from the *-operations to the +-operations, while current Opium can't do that because it conflates the two.

This obviously doesn't apply to Rock, and probably not to the new router PR (but I haven't looked in detail).

Another option I considered was providing an alternative composition operator to the standard @@, that could compose decorated middlewares. That is, you could build an introspection-aware application spine using some kind of @@@ operator, that would take pairs of a middleware and a name — something like that. Obviously, you would lose the benefit of having forced all libraries to have used @@@ pervasively. However, I hope that most library APIs will be relatively shallow, and if we settle on @@@ for instrospection, it will be easy to simply stick on a name in your usage of the library, if the library did not publish a decorated middleware together with an undecorated one.

I personally hope @@@ is unnecessary, though, and that most of the value of introspection comes from introspecting only routes. I already recommend publishing routes and not large middleware stacks for site composition, and sticking to the introspection-friendly DSL in places. For example, in example f-static:

The static route ends with *. This is a subsite route. Generally, you should prefer Dream.scope to *, because Dream.scope will support router introspection, if it is added in the future.

To further show the difference between Dream's +-oriented composition, and Opium's *-oriented composition, as a very small instance of it, note that Opium has a static middleware that filters out some requests that happen to go to the static path. Dream has a static handler that you separately route requests to using the (or a) router.

In any case, I was completely certain that I didn't want to "force" the syntactic overhead that results from middleware introspection on everyone always, by baking introspection in at the basic level. This comes in part from my own experience writing Opium middlewares.

I'll also love to help out in any way I can to help round out the web story for OCaml once Dream is ready for external contributors!

From Dream's side, it should be ready after alpha1 within ~1 week :)

[tmattio]

I'll also love to help out in any way I can to help round out the web story for OCaml once Dream is ready for external contributors!

Same, I feel like Dream nailed down the low-level HTTP features with support for Http2, websockets, etc. and it clears the room for higher level things I have been wanting to explore in Opium (instrumentation dashboard, hot reloading, SQL DSL à la sequoia, etc.)

Would love to work on these in a way that benefits both Dream and Opium 😃

[dangdennis]

You like to say Dream is low-level. @aantron
What’s an example of a high level framework?

Examples like Ruby on Rails and Go Gin is high because they offer additional helpers. But when it comes to creating raw APIs, it seems Dream has provided that. If we compare Dream to Phoenix then, Phoenix is also a convenience wrapper around Plug (composable http layer) with a lot of macros. Where do you see Dream?

[aantron]

What’s an example of a high level framework?

Sihl is an example of a high-level OCaml framework. Although Dream will gradually cover some more of Sihl, I think it won't ever cover all of it directly — maybe only with recommendations to some well-done libraries to use with Dream. For example, I don't want Dream to impose configuration choices on users, at least not yet — we haven't "solved" this in OCaml, so it seems better to have Dream read no external configuration explicitly or implicitly, and leave it to users or Dream-based libraries to figure this out. This also has the side effect of making Dream really self-contained and easy to understand, almost "functional."

Sihl has things like database migrations, some REST helpers for creating multiple endpoints (I haven't kept up with development recently), etc. The database support of Sihl is (was?) integrated with the rest of Sihl in a slightly invasive way. At least, if you'd like to swap it out, or change it in any non-trivial way, it's really not clear how to do it. For example, I needed to use sqlite to go with a sort of my-command host-a-web-ui type of interface, showing a lightweight web interface, integrated into a bigger binary. Sihl became a bit of an obstacle for this. I'd like to avoid any such entanglements in Dream, so even when there is integration right in the Dream core, it's always opt-in, loosely coupled, and for convenience only.

Apart from extra helpers, another thing that often (though not always) separates high-level frameworks and low-level frameworks is that high-level frameworks typically require a large amount of boilerplate to get started, so they have project generators, etc. At least core Dream is really meant to work out of single files (when you start out), and it's supposed to be something so small and clear that you have a very clear mental picture of what your Dream-based program is doing, rather than having, potentially, generated files in your project that you never have looked at, and which are a mystery to you, even long into project development. I'm not completely opposed to project generators — I just don't want them to be the only practical way. I'm happy if people find common patterns and build templates/generators around them, and I'll link to them. I suspect that some good templates/generators will cover almost everything that most high-level frameworks cover. See #42.

High-level frameworks impose a lot of concepts, even "model," "view," and/or "controller." While all those things and others are pretty obvious once you're using them, I also don't want them around as framework-imposed concepts. While developing Dream, I tried to simplify as many concept-kinks as possible. If a project needs explicit concepts, it will develop it (or take them from a generator or template), but I want Dream itself to be dead simple, and you can think of composing it how you want.

Examples like Ruby on Rails and Go Gin is high because they offer additional helpers.

Dream is probably more like Rack, and more like Plug, though I don't deliberately keep it that way. As long as something is general-use, can be implement reasonably simply, and doesn't pull in any substantial new dependencies, it can be part of Dream. An example of this is the current flash messages PR, #62.

On the other hand, htmx (#59) is not general-use (but very cool). It also brings maintenance questions that are unique to itself (AFAIK), so it's better to develop it separately (though in coordination, as necessary, and with links from Dream).

it seems Dream has asked provided that.

I failed to parse this! :)

Where do you see Dream?

I think this post gets the position of Dream about right:

           Higher level ->
+-------------+--------------------+
|    Opium    |        Sihl        |
+-------------+--------------------+
+-----------------+
|      Dream      |
+-----------------+

EDIT: Not to scale!

Dream is a framework, which, in terms of span, actually covers (what I think) are the most useful (lower) parts of Sihl (and I think the rest should be in separate libraries). However, Dream is loosely coupled, and factored out in such a way, and uses opting in so much (rather than generation + opting out), that, when you're not calling into any of that higher-level stuff, you're effectively using something very low-level and raw, so something like Rack, Plug, or the lower levels of Opium. This is despite, again, that Dream actually fully spans Opium (AFAIK).

[aantron]

In summary, by keeping the parts of Dream as orthogonal as possible, Dream is able to have a simple, low-level core with several higher-level opt-in features also available, which don't affect someone who is interested in only the low-level core in any way.

Such a person can then build some other higher-level features over the low-level core, according to their needs.

[aantron]

You like to say Dream is low-level. @aantron

I just realized, based on some messages I wrote in Discord, that I need to challenge this :P

I don't like to say that Dream is low-level. It's just that here, and on the Discuss forum, people saw that Dream has some features that make it appear as high-level, and they compare it with e.g. Opium, with Opium being "low-level." However, as it turns out, the way Dream is factored, its main API is more low-level than the lowest levels of Opium, and it is able to add high-level features to go with that, in a non-entangled way. So I'm forced to respond to that, to clarify :)

For example, in this issue, I was arguing that it is straightforward to implement the low-level layer of Opium, Rock, as an abstraction over Dream — but not the other way. This is because Dream is simpler than Rock.

[aantron]

If it was up to me, we would drop all these "level" comparisons, as they are clearly not useful — I personally am not using them, and we can see how many words it takes to shoehorn the Dream vs. X comparison into the level mental model.

Compared to an "average" framework, Dream is just extremely thoroughly factored into separate, orthogonal, non-interfering concepts, a very small number of them, and that's why it is at once "lower level" and has high-level features. It does more with less and with much less mental load. Undoubtedly, there are others like it — I don't claim uniqueness for Dream. I'm just comparing to some average I imagined, based on some kind of idea of docs of Laravel, Django, etc.

We can implement "less factored" frameworks on top of Dream by implementing their (IMO) somewhat confused extra concepts over Dream. We couldn't implement Dream (easily) over frameworks that have such extra concepts, because we'd have to do work just to tuck them away. That's half of the (unhelpful) "lower level."

The other half of "lower level," again because of the thorough factoring, Dream's core is like literally just an absolutely minimalistic abstraction layer over http/af, h2, and websocket/af, or any other web server library we could choose, and by simply not opting into all the extra stuff (because of the high orthogonality), you can use only that abstraction without triggering anything else.

[aantron]

Where do you see Dream?

So in a new summary, I see Dream as a very clearly-factored and easy-to-use Web framework. I haven't made any decisions about it specifically because of "levels' — I literally did not think about levels during its development. I do make decisions about it based on clarity, factoring, dependencies, future-proofing, and maintainability.

For the latter, Dream can again falsely give the impression that it is low-level because I am avoiding certain "high-level" contributions directly to it (while, confusingly, still offering certain high-level features). The reason this combination could be confusing, given that the contributions are on the same "level," is because I am not using a levels mental model :) I just need a distributed team of people at this point in Dream, including some maintainers, other than me, of related projects.

[aantron]

Another example, from memory: Rust Rocket (also Java Spring, and others) have routing based on decorators (IIRC).

Dream (and Opium, and many others) have TOC-style centralized routing specification.

Decorator-based routing can be implemented on top of TOC by just writing route specifications from each decorator into a ref, and then building the TOC somewhere in a hidden way.

TOC-based routing can't be so easily implemented on top of decorators, as you have to force people into some kind new convention, like "write all your decorators on little wrapper functions in one place."

So Rocket-style routing can be implemented for Dream, if someone makes a library with

val drop_handler_as_a_route_into_a_ref : Dream.method_ -> string -> Dream.handler -> unit
val read_the_hidden_ref_and_make_a_router : unit -> Dream.handler

(probably with some helpers for scopes and middlewares).

If someone does that, I will link to it.

Dream-style routing may exist somewhere in Rocket (I haven't looked), but it's not what they recommend in the docs, and can't be easily implemented based on what they do recommend.

I don't think that makes Dream inherently lower-level than Rocket.

But Dream is simpler than Rocket, because there is no hidden initialization state for the router, because a Dream app can statically analyzed in a visual way, and if you don't want that, you can build the Rocket way on top of it.

[aantron]

Ok to continue the super spam thread:

there is no hidden initialization state for the router

...there is no the router either. You can have many routers, multiple Web apps linked into one process (who knows why, but maybe someone wants to deploy their 1000 microservices in one binary because it's easier to deal with a blob).

You can even use a completely different routing library with Dream. Dream just offers "a" router so you don't have to hunt for one out of the box. But if you like something else better, you can replace it for your app's TOC (or decorators implemented on top of another routing library).

[aantron]

Closing this for now.

The current status is that once

• Dream itself has settled a bit, and
• there is a second framework that uses a similar model,

we can factor out type Dream.request, Dream.response, into something that can be shared across frameworks.

This kind of sharing is already happening to an extent with the Dream repo, with the regular and Mirage Dream (#119) being somewhat different implementations that agree on the request/response type, and therefore the implementation-agnostic middlewares (most of the ones in Dream).

We'll wait for some external motivation to do the actual factoring-out :)

[aantron]

I've completed an initial factoring-out of the Dream core, which can be seen in a near-final state here:

dream/src/pure/dream_pure.mli

Lines 8 to 366 in 8652178

	type 'a message
	type client
	type server
	type request = client message
	type response = server message
	type 'a promise = 'a Lwt.t
	type handler = request -> response promise
	type middleware = handler -> handler
	type buffer =
	(char, Bigarray.int8_unsigned_elt, Bigarray.c_layout) Bigarray.Array1.t
	type stream
	type method_ = [
	| `GET
	| `POST
	| `PUT
	| `DELETE
	| `HEAD
	| `CONNECT
	| `OPTIONS
	| `TRACE
	| `PATCH
	| `Method of string
	]
	val method_to_string : [< method_ ] -> string
	val string_to_method : string -> method_
	val methods_equal : [< method_ ] -> [< method_ ] -> bool
	val normalize_method : [< method_ ] -> method_
	type informational = [
	| `Continue
	| `Switching_Protocols
	]
	type successful = [
	| `OK
	| `Created
	| `Accepted
	| `Non_Authoritative_Information
	| `No_Content
	| `Reset_Content
	| `Partial_Content
	]
	type redirection = [
	| `Multiple_Choices
	| `Moved_Permanently
	| `Found
	| `See_Other
	| `Not_Modified
	| `Temporary_Redirect
	| `Permanent_Redirect
	]
	type client_error = [
	| `Bad_Request
	| `Unauthorized
	| `Payment_Required
	| `Forbidden
	| `Not_Found
	| `Method_Not_Allowed
	| `Not_Acceptable
	| `Proxy_Authentication_Required
	| `Request_Timeout
	| `Conflict
	| `Gone
	| `Length_Required
	| `Precondition_Failed
	| `Payload_Too_Large
	| `URI_Too_Long
	| `Unsupported_Media_Type
	| `Range_Not_Satisfiable
	| `Expectation_Failed
	| `Misdirected_Request
	| `Too_Early
	| `Upgrade_Required
	| `Precondition_Required
	| `Too_Many_Requests
	| `Request_Header_Fields_Too_Large
	| `Unavailable_For_Legal_Reasons
	]
	type server_error = [
	| `Internal_Server_Error
	| `Not_Implemented
	| `Bad_Gateway
	| `Service_Unavailable
	| `Gateway_Timeout
	| `HTTP_Version_Not_Supported
	]
	type standard_status = [
	| informational
	| successful
	| redirection
	| client_error
	| server_error
	]
	type status = [
	| standard_status
	| `Status of int
	]
	val status_to_string : [< status ] -> string
	val status_to_reason : [< status ] -> string option
	val status_to_int : [< status ] -> int
	val int_to_status : int -> status
	val is_informational : [< status ] -> bool
	val is_successful : [< status ] -> bool
	val is_redirection : [< status ] -> bool
	val is_client_error : [< status ] -> bool
	val is_server_error : [< status ] -> bool
	val status_codes_equal : [< status ] -> [< status ] -> bool
	val normalize_status : [< status ] -> status
	val request :
	?method_:[< method_ ] ->
	?target:string ->
	?version:int * int ->
	?headers:(string * string) list ->
	stream ->
	stream ->
	request
	val method_ : request -> method_
	val target : request -> string
	val version : request -> int * int
	val with_method_ : [< method_ ] -> request -> request
	val with_version : int * int -> request -> request
	val response :
	?status:[< status ] ->
	?code:int ->
	?headers:(string * string) list ->
	stream ->
	stream ->
	response
	val status : response -> status
	val header : string -> 'a message -> string option
	val headers : string -> 'a message -> string list
	val all_headers : 'a message -> (string * string) list
	val has_header : string -> 'a message -> bool
	val add_header : string -> string -> 'a message -> 'a message
	val drop_header : string -> 'a message -> 'a message
	val with_header : string -> string -> 'a message -> 'a message
	val with_all_headers : (string * string) list -> 'a message -> 'a message
	val body : 'a message -> string promise
	val with_body : string -> response -> response
	val read : request -> string option promise
	val with_stream : 'a message -> 'a message
	val write : response -> string -> unit promise
	val flush : response -> unit promise
	val close_stream : response -> unit promise
	(* TODO This will need to read different streams depending on whether it is
	passed a request or a response. *)
	val client_stream : 'a message -> stream
	val server_stream : 'a message -> stream
	val with_client_stream : stream -> 'a message -> 'a message
	val next :
	stream ->
	data:(buffer -> int -> int -> bool -> bool -> unit) ->
	close:(int -> unit) ->
	flush:(unit -> unit) ->
	ping:(buffer -> int -> int -> unit) ->
	pong:(buffer -> int -> int -> unit) ->
	unit
	val write_buffer :
	?offset:int -> ?length:int -> response -> buffer -> unit promise
	module Stream :
	sig
	type reader
	type writer
	type read =
	data:(buffer -> int -> int -> bool -> bool -> unit) ->
	close:(int -> unit) ->
	flush:(unit -> unit) ->
	ping:(buffer -> int -> int -> unit) ->
	pong:(buffer -> int -> int -> unit) ->
	unit
	(** A reading function. Awaits the next event on the stream. For each call of a
	reading function, one of the callbacks will eventually be called, according
	to which event occurs next on the stream. *)
	type write =
	close:(int -> unit) ->
	(unit -> unit) ->
	unit
	(** A writing function. Pushes an event into a stream. May take additional
	arguments before [~ok]. *)
	val reader : read:read -> close:(int -> unit) -> reader
	(** Creates a read-only stream from the given reader. [~close] is called in
	response to {!Stream.close}. It doesn't need to call {!Stream.close} again
	on the stream. It should be used to free any underlying resources. *)
	val empty : reader
	(** A read-only stream whose reading function always calls its [~close]
	callback. *)
	val string : string -> reader
	(** A read-only stream which calls its [~data] callback once with the contents
	of the given string, and then always calls [~close]. *)
	val pipe : unit -> reader * writer
	(** A stream which matches each call of the reading function to one call of its
	writing functions. For example, calling {!Stream.flush} on a pipe will cause
	the reader to call its [~flush] callback. *)
	val writer :
	ready:write ->
	write:(buffer -> int -> int -> bool -> bool -> write) ->
	flush:write ->
	ping:(buffer -> int -> int -> write) ->
	pong:(buffer -> int -> int -> write) ->
	close:(int -> unit) ->
	writer
	val no_reader : reader
	val no_writer : writer
	val stream : reader -> writer -> stream
	(* TODO Consider tupling the arguments, as that will make it easier to pass the
	result of Stream.pipe. *)
	val close : stream -> int -> unit
	(** Closes the given stream. Causes a pending reader or writer to call its
	[~close] callback. *)
	val read : stream -> read
	(** Awaits the next stream event. See {!Stream.type-read}. *)
	val read_convenience : stream -> string option promise
	(** A wrapper around {!Stream.read} that converts [~data] with content [s] into
	[Some s], and [~close] into [None], and uses them to resolve a promise.
	[~flush] is ignored. *)
	val read_until_close : stream -> string promise
	(** Reads a stream completely until [~close], and accumulates the data into a
	string. *)
	val ready : stream -> write
	val write : stream -> buffer -> int -> int -> bool -> bool -> write
	(** A writing function that sends a data buffer on the given stream. No more
	writing functions should be called on the stream until this function calls
	[~ok]. The [bool] arguments are whether the message is binary and whether
	the [FIN] flag should be set. They are ignored by non-WebSocket streams.
	Note: [FIN] is provided as part of the write call, rather than being a
	separate stream event (like [flush]), because the WebSocket writer needs to
	immediately know when the last chunk of the last frame in a message is
	provided, to transmit the [FIN] bit. If [FIN] were to be provided as a
	separate event, the WebSocket writer would have to buffer each one chunk, in
	case the next stream event was [FIN], in order to be able to decide whether
	to set the [FIN] bit or not. This is awkward and inefficient, as it
	introduces an unnecessary delay into the writer, as if the next event is not
	[FIN], the next data chunk might take an arbitrary amount of time to be
	generated by the writing user code. *)
	val flush : stream -> write
	(** A writing function that asks for the given stream to be flushed. The meaning
	of flushing depends on the implementation of the stream. No more writing
	functions should be called on the stream until this function calls [~ok]. *)
	val ping : stream -> buffer -> int -> int -> write
	(** A writing function that sends a ping event on the given stream. This is only
	meaningful for WebSockets. *)
	val pong : stream -> buffer -> int -> int -> write
	(** A writing function that sends a pong event on the given stream. This is only
	meaningful for WebSockets. *)
	end
	val no_middleware : middleware
	val pipeline : middleware list -> middleware
	type websocket = stream
	val websocket :
	?headers:(string * string) list ->
	(websocket -> unit promise) ->
	response promise
	val send : ?kind:[< `Text | `Binary ] -> websocket -> string -> unit promise
	val receive : websocket -> string option promise
	val close_websocket : ?code:int -> websocket -> unit promise
	val is_websocket : response -> (websocket -> unit promise) option
	module Formats :
	sig
	val html_escape : string -> string
	val to_base64url : string -> string
	val from_base64url : string -> string option
	val to_percent_encoded : ?international:bool -> string -> string
	val from_percent_encoded : string -> string
	val to_form_urlencoded : (string * string) list -> string
	val from_form_urlencoded : string -> (string * string) list
	val from_cookie : string -> (string * string) list
	val to_set_cookie :
	?expires:float ->
	?max_age:float ->
	?domain:string ->
	?path:string ->
	?secure:bool ->
	?http_only:bool ->
	?same_site:[ `Strict | `Lax | `None ] ->
	string -> string -> string
	val split_target : string -> string * string
	val from_path : string -> string list
	val to_path : ?relative:bool -> ?international:bool -> string list -> string
	val drop_trailing_slash : string list -> string list
	val make_path : string list -> string
	val text_html : string
	val application_json : string
	end
	type 'a local
	val new_local : ?name:string -> ?show_value:('a -> string) -> unit -> 'a local
	val local : 'a local -> 'b message -> 'a option
	val with_local : 'a local -> 'a -> 'b message -> 'b message
	val fold_locals : (string -> string -> 'a -> 'a) -> 'a -> 'b message -> 'a
	(* TODO Delete once requests are mutable. *)
	val first : 'a message -> 'a message
	val last : 'a message -> 'a message
	val sort_headers : (string * string) list -> (string * string) list

It is a fairly minimalistic set of helpers for three types, request, response, and stream. The other types are functions between these, or type abbrevations for standard types.

This work was prompted by writing the client, a primitive form of the core of which was in the repo here:

dream/src/hyper.mli

Lines 9 to 11 in f69b956

	val send :
	?connection_pool:connection_pool ->
	request -> response promise

The new client uses exactly the same types as Dream. The new client can be used with or without Dream. The "obvious" way to use Dream and the client is to communicate using HTTP with the outside world or with each other.

However, in addition, because body stream endpoints are carefully arranged, and both (1) the internal runner of the client and (2) a Dream server have the same type

request -> response promise

• A Dream server can trivially pass a request it has received to the client library, creating a proxy, and same with the response in the other direction. The client's HTTP body sending code will automatically cause reading of the HTTP body by the server, and vice versa with the response. The Dream server can modify the request and response.
• A Dream client can use an in-process Dream server as a runner, creating a no-network tester. Again, the streams work out in such a way that everything "just works" without any HTTP layers interpreting the body streams, etc.

Getting the client and server to be this composable forced me to think hard and delete several concepts, such as "apps" and "global" variables (per-server global state). I also moved many fields out of requests/responses, so they have become simpler. They are now in message-local variables on the server.

I'm going to make a few further simplifications, like more fully substituting the Stream module for the WebSocket helpers, and converting to mutable requests and responses (#21), which will allow some more simplifications.

[aantron]

Or, in very bad pictures, where <===> represents ordinary function calls:

Normal usage of the client:

+--------+
| client | <===> HTTP
+--------+

Normal usage of the server:

                            +--------+
                 HTTP <===> | server |
                            +--------+

If your client and server communicate over HTTP:

+--------+                  +--------+
| client | <===> HTTP <===> | server |
+--------+                  +--------+

If you just use the server to implement the client's runner, you short-circuit the HTTP and get a no-network tester:

+--------+       +--------+
| client | <===> | server |
+--------+       +--------+

The "dual" of this is a proxy (note the order of client and server is flipped):

           +--------+       +--------+
HTTP <===> | server | <===> | client | <===> HTTP
           +--------+       +--------+