HTTPRequestAsync operator

HTTPRequestAsync operator proposal

Goal

A new operator that supports all HTTP 1.1 request methods (GET, POST, PUT, DELETE, HEAD) and is designed to work well with HTTP(S) (REST) services in asynchronous manner.

Issues

• Current native functions support HTTP requests in synchronous manner only.
• Streaming pipelines as well as other streams applications are required to perform HTTP/REST calls to external services (such as alerting / getting updated models from WML) and yet, avoid back pressure.

Functionality concepts

• The asynchronous call approach can make HTTP/REST requests possible without blocking the thread. The responses will be handled as callbacks, in asynchronous manner when they arrive.

• The Operator makes an HTTP(S) request for each tuple arriving on its input port.

• The request is placed on a queue(s) and the calling thread is released

• Each queue has a working thread that sends out the HTTP request with a callback to handle the response, keeping the input tuple data.

• When the response arrives, it will call the callback with the response parameters (including the input tuple data)

• The operator will provide an output section to allow output manipulations

param
    url : "http://httpbin.org/" + action ;
    // url can be any expression (in this example the action is an attribute of an input tuple)
    method : GET ;
    //valid methods are GET, POST, PUT, DELETE, HEAD
    body : '{"key" : "value"}' ;
    // body should be aligned with content type parameter
    headers : {"Connection" : "close"} ;
    // HTTP request header key values
    contentType : json ;
    // the supported types are: json, xml, urlencoded,html, plain (text) 
    timeout : 10.0;
    // set an overall timeout for HTTP requests
    userAgent : "IBM Streaming Pipeline";
    // specifies the caller user agent 
    workerQueues : 1;
    // specifies the number of worker queues required (the default is 1)

output
    body = Body(),        //response body as rstring 
    headers = Headers(),  //response headers as map<rstring,rstring>
    message = Message(),  //response message as rstring
    status = Status();    //response HTTP code as uint16

Output port schema:

• All output tuple's attributes will be auto completed by the same input tuple attribute names
• Additionally, custom output functions can be used to set the output tuple with the Http response fields

Error Handling:

There is an optional error output port

• The error output port allows any input attribute to be included (auto completed), additionally the error "message" and "status"
• Assignments: no output section is available for error output port