PUT Requests are timing out at spoa-mirror agent.

[aslam-m]

We noticed that PUT requests are timing out at spoa-mirror agent. GET, POST are working fine in our setup as expected. Based on your Readme doc, its mentioning below lines under "Know Bugs & Limitations". Can you provide some instructions or any document references - how to add PUT request support also in spoe-mirror agent?

Appreciate your help in advance. Thanks

Text From Your Readme -
"HTTP methods that certainly work are GET, HEAD and POST. Other methods can be
added if necessary"

Sample PUT request from our SPOE-Mirror logs for your reference -

[ 1][ 363.550820] <1:26> <-- Sending data
[ 1][ 363.550835] <1:26> 4/4/4 byte(s) send frame length
[ 1][ 363.550842] <1:26> 8/8/8 byte(s) send frame data
[ 1][ 363.550845] <1:26> Frame of 8 bytes sent: <6700000001f23801> <g.....8.>
[ 1][ 373.554094] "PUT http://xyz.com/res?force=false HTTP/???" 0 0/0 10003.358 Operation timed out after 10002 milliseconds with 0 bytes received
[ 1][ 373.554133] <1:26> --> Receiving dat

[lmcdasm]

Hello there..

Has anyone else there hit this issue.

POST and GETs are working, however i have PUT requests that need to service and i get the Timeout as well.

on my side im using https in the --mirrorurl but i dont think it makes a difference since POSTs are working (however the upstream call is a PUT not a POST server).

Some information

haproxy.cfg bits:

mirror
command /usr/local/bin/spoa-mirror --runtime 0 --mirror-url https://mymirror.with.realcert.com

This is in my https-in frontend:

#---------------------------------------------------------------------
frontend https-in
bind *:80
bind *:443 ssl crt /etc/ssl/my.good.cert.pem strict-sni alpn h2,http/1.1
mode http
http-request redirect scheme https unless { ssl_fc }
http-response set-header Strict-Transport-Security max-age=31536000
option http-buffer-request
filter spoe engine mirror config /etc/haproxy/mirror.conf

Mirror.conf
[mirror]
spoe-agent mirror
log global
messages mirror
use-backend mirroragents
timeout hello 500ms
timeout idle 5s
timeout processing 5s

spoe-message mirror
## #HACKING AROUND HERE
http-request set-var(txn.fc_sni) "cli-2-qa.na.onecloud.hosting.cerence.net"
http-request set-header Host cli-2-qa.na.onecloud.hosting.cerence.net
http-request set-uri https://%[req.hdr(Host)]
###
args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
event on-frontend-http-request

Any tips would be great to get PUT working
~
~

[baylanger]

RTFM

[baylanger]

Any tips would be great to get PUT working

Best tip : if you look at the source code, this project is using haproxy SPOE protocol and curl libraries. It's not rocket science.

[zaga00]

Hello all,

as for the PUT HTTP method, it could be added. If you want you can look at the following url https://github.com/RallySoftware/spoa-mirror/tree/empty-put-body where the user @argreway solved it for himself. The patch is very simple, I just need to have more time for this project because I have some other commitments.

Best regards.

[lmcdasm]

Hey there.. @baylanger - so friendly.. i had started to look at the code, and had spent a bit of time today, noticed that there was some work and a issue being done and added... sorry for asking.. I do have it working very nicely with POST/GET...

@zaga00 - thanks for the link.. ill take a stab it ..
Cheers,

[zaga00]

Hello @aslam-m and @lmcdasm,

The HTTP PUT method is implemented so you can test if it works for you now.

Best regards.

[lmcdasm]

Hey there.

Amazing.. i will try it out right now .. thanks so much!

[aslam-m]

@zaga00 @lmcdasm

Fantastic!! I will try out too.
This will help in my other projects as the current project is already closed.

Thanks a lot!

[lmcdasm]

Hello @zaga00

this works great now, thanks alot.

A question for you.. if now i want to change parts of the frame body (specifically the HTTP Header parts encapsulated - i.e.
HOST, URL (IP:PORT and PATH), im trying to figure what is the better way.

(a) - can i do this with a set var in the spoe section, something like set var arg8:Host:"my.new.host.com"?
(b) - should i go into the curl put functions and put a hook in there (perhaps a passable argument).
(c) - i have a golang-gin setup that basically does this mirror/splitting. and i saw the reference golang-spop setup (with the iprep) and your code, would it make more sense to get the frame and then channel it into the setup i have today?

the end goal is really to have a mirror spillter that allows me to change the HTTP header and body parts on the way "out" to the second (or multiple) destinations (like goduplicator ).

the PUT option works great and thanks for this!, I can see that the HOST and SNI from the original call is kept (as a mirror would expect) rather than rewrite with the --mirrorurl. I am working towards a method of being able to split /mirror and alter just a couple header aspects.

Any inputs/thoughts would be appreciated.
D

[lmcdasm]

Just a note - for this issue - the problem for me is solved - any other notes are about "extended/new features" - but the PUT timeout issue is resolved (tested at 1200/tx/sec and it was stable).

[zaga00]

Hello @lmcdasm,

HTTP headers are sent via SPOP and added to the hdrs list of the mirror structure in the spoa_msg_arg_hdrs() function. cURL sets HHTP headers using that list in the mir_curl_set_headers() function. In one of these two functions, the recognition and replacement of certain HTTP headers can be added. For some other simpler way I don't know, possibly something can be tried via fetching HAProxy samples so that instead of req.hdrs_bin in the SPOA configuration something more complex and complicated can be used.

Best regards.

[lmcdasm]

Thanks again @zaga00

you have helped me alot and appreciate your input!

Cheers