Linux: zipdeploy API times out in synchronous mode #2776

phawxby · 2018-05-25T07:16:21Z

Due to some pretty severe issues with VSTS deployments and performance we've using CircleCI to assemble our application, compress to a zip and deploy via the REST API. Unfortunately due to this we're having to deploy with all the node modules bundled inside and we've bypassed the npm install phase of the deployment script. This results in deploying a 50Mb zip file.

This all works, we have our application deploying and running. We're using the non-asynchronous API because we can use that detect when the application is deployed and due to fan-out out build pipeline isn't getting choked, the region deployments happen in parallel.

However the API lies, it's doing weird things instead of holding the connection until deployment is complete.

At a little over 4 minutes we've had this.

#!/bin/bash -eo pipefail
if [ "${CIRCLE_BRANCH}" == "master" ]; then
  curl -i -X POST -u SNIP --data-binary @"/tmp/drop.zip" https://SNIP.scm.azurewebsites.net/api/zipdeploy
fi
HTTP/1.1 100 Continue



HTTP/1.1 502 Bad Gateway

Content-Type: text/html

Server: Microsoft-IIS/10.0

Date: Thu, 24 May 2018 08:18:08 GMT

Content-Length: 1477



<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>

<title>502 - Web server received an invalid response while acting as a gateway or proxy server.</title>

<style type="text/css">

<!--

body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}

fieldset{padding:0 15px 10px 15px;} 

h1{font-size:2.4em;margin:0;color:#FFF;}

h2{font-size:1.7em;margin:0;color:#CC0000;} 

h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;} 

#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;

background-color:#555555;}

#content{margin:0 0 0 2%;position:relative;}

.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}

-->

</style>

</head>

<body>

<div id="header"><h1>Server Error</h1></div>

<div id="content">

 <div class="content-container"><fieldset>

  <h2>502 - Web server received an invalid response while acting as a gateway or proxy server.</h2>

  <h3>There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server.</h3>

 </fieldset></div>

</div>

</body>

</html>

At a little over 4 minutes we've also had this

#!/bin/bash -eo pipefail
if [ "${CIRCLE_BRANCH}" == "development" ]; then
  curl -i -X POST -u SNIP --data-binary @"/tmp/drop.zip" https://SNIP.scm.azurewebsites.net/api/zipdeploy
fi
HTTP/1.1 100 Continue



HTTP/1.1 504 Gateway Time-out

Content-Length: 260

Content-Type: text/html

ETag: "5b054e03-104"

Server: nginx

Set-Cookie: ARRAffinity=5d10da0e40ca983507b2d5977c4a4d6507be25813895d6f7c215841f05c3dce8;Path=/;HttpOnly;Domain=SNIP.scm.azurewebsites.net

Date: Thu, 24 May 2018 15:46:33 GMT



<html>

    <head>

        <title> Server Error </title>

    </head>

    <body>

        <font color =\"#aa0000\">

            <h2>Server Error.</h2>

        </font>



        There was an unexpected error in the request processing.

    </body>

</html>

At 3 minutes we've had this.

#!/bin/bash -eo pipefail
if [ "${CIRCLE_BRANCH}" == "development" ]; then
  curl -i -X POST -u SNIP --data-binary @"/tmp/drop.zip" https://SNIP.scm.azurewebsites.net/api/zipdeploy
fi
HTTP/1.1 100 Continue



curl: (56) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
Exited with code 56

Ultimately deployment ends up taking 10-20 minutes even on the premium tier Kudu hosts.

A few extra notes:

We're going with CircleCI over VSTS because CircleCI is is reporting Lint and Test results in ~40 seconds, VSTS is still checking out code at the point CircleCI finishes. Ultimately it takes nearly 5 minutes.
CircleCI gives us vastly more configuration options. We're using a 4core/8gb host for the test phase, there's no easy way to do this in VSTS
Kudu deployment is extremely slow, 10-20 minutes in most cases.
We're having a nightmare of a time with service principals, hence not using the Azure CLI for deployment

The text was updated successfully, but these errors were encountered:

davidebbo · 2018-05-25T14:34:13Z

It is by design that any request to App Service will time out after 230 seconds. Please use the async API instead. It lets you poll the API to know when it is done.

Also, please try using the new Run-From-Zip, which should make your deployment much faster. See this announcement for details. Basically, in your Azure App Settings, set WEBSITE_RUN_FROM_ZIP to 1, and then deploy as normal using zipdeploy.

phawxby · 2018-05-25T15:08:28Z

Ah, 230 isn't documented anywhere I've seen but I think that's counter to the what is expected. I'll investigate using the async API as an alternative.

Run-From-Zip sounds like exactly what we need, however we're using a Linux host and being experimental I'm nervous about relying on it for a production site. I think we're slowly being pushed to AWS for hosting, Linux and Node feel like second class citizens on the Azure platform.

davidebbo · 2018-05-25T15:10:13Z

I've renamed the issue to better capture what it is about.

weltan · 2018-05-29T23:01:18Z

@davidebbo: I believe I'm running into similar issues when using /api/wardeploy to deploy to Azure using the Linux Tomcat 9.0 (preview) web app. I'm able deploy a .war file from my development box via curl -X POST ... /api/wardeploy (I still get a 502, but the deployment works), but getting a 502 would (probably?) prevent me from deploying reliably in VSTS.

To be fair, the .war file is ~280 MB, so maybe that's a factor?

I noticed that your Run-From-Zip announcement doesn't apply to Linux (yet). Do you have any ideas on how to avoid the 502 in the Linux ecosystem?

davidebbo · 2018-05-29T23:07:32Z

@weltan I'm on the WIndows side, so I'll let the Linux experts comment further. @rramachand21, who can look at this?

weltan · 2018-05-31T00:54:30Z

@rramachand21 should I open an issue on a separate repo, or is here fine? Didn't want to hijack the issue.

weltan · 2018-06-05T19:17:23Z

hi @rramachand21, just following up here: any word on timeout issues related with the api/wardeploy and api/zipdeploy giving a 502 for timeouts for larger artifacts on Linux?

BenWalters · 2018-12-18T22:06:49Z

@rramachand21 @davidebbo
Any update on this topic? I'm struggling for deployment options with my Linux WebApp setup. ZipDeploy seems to do the job but I have similar issues to the above - getting 502's instead of a success message. Would be good to have a solution for this.

davidebbo · 2018-12-18T22:10:17Z

@BenWalters I am no longer working on this project, so I will let @rramachand21 or others comment further.

phawxby · 2018-12-18T22:18:51Z

You guys could be suffering from this too.
Azure/azure-cli#8048

BenWalters · 2018-12-18T22:41:39Z

@phawxby Looks like it could resolve the issue. We aren't using ZipDeploy directly in the CLI, we've been using a ZipDeploy VSTS marketplace task. So think we'd need to implement the CLI tasks to get this running. Will have a look, thanks for pointing it out!

jvano · 2024-04-29T10:18:11Z

Hi

If the problem persists and is related to running it on Azure App Service, please open a support incident in Azure:
https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request

This way we can better track and assist you on this case

Thanks,

Joaquin Vano
Azure App Service

davidebbo changed the title ~~Rest deployment API cannot be trusted~~ Linux: zipdeploy API times out in synchronous mode May 25, 2018

jvano closed this as completed Apr 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Linux: zipdeploy API times out in synchronous mode #2776

Linux: zipdeploy API times out in synchronous mode #2776

phawxby commented May 25, 2018 •

edited

Loading

davidebbo commented May 25, 2018

phawxby commented May 25, 2018

davidebbo commented May 25, 2018

weltan commented May 29, 2018

davidebbo commented May 29, 2018

weltan commented May 31, 2018

weltan commented Jun 5, 2018 •

edited

Loading

BenWalters commented Dec 18, 2018

davidebbo commented Dec 18, 2018

phawxby commented Dec 18, 2018

BenWalters commented Dec 18, 2018

jvano commented Apr 29, 2024

Linux: zipdeploy API times out in synchronous mode #2776

Linux: zipdeploy API times out in synchronous mode #2776

Comments

phawxby commented May 25, 2018 • edited Loading

davidebbo commented May 25, 2018

phawxby commented May 25, 2018

davidebbo commented May 25, 2018

weltan commented May 29, 2018

davidebbo commented May 29, 2018

weltan commented May 31, 2018

weltan commented Jun 5, 2018 • edited Loading

BenWalters commented Dec 18, 2018

davidebbo commented Dec 18, 2018

phawxby commented Dec 18, 2018

BenWalters commented Dec 18, 2018

jvano commented Apr 29, 2024

phawxby commented May 25, 2018 •

edited

Loading

weltan commented Jun 5, 2018 •

edited

Loading