Curl error thrown for http POST to /session

[Capn-Precious]

The following errors are happening in a production & staging environment when we attempt to screenshot with panther, set size for the window, or do anything that utilizes panther for web scraping. It seems like the connection drops or is lost between panther and php-webdriver.

Errors:

Facebook\WebDriver\Exception\Internal\WebDriverCurlException: Curl error thrown for http POST to /session with params: 
{
    "capabilities": {
        "firstMatch": [
            {
                "browserName": "chrome",
                "goog:chromeOptions": {
                    "args": [
                        "--no-sandbox",
                        "--headless",
                        "--disable-dev-shm-usage",
                        "--hide-scrollbars"
                    ]
                }
            }
        ]
    },
    "desiredCapabilities": {
        "browserName": "chrome",
        "platform": "ANY",
        "goog:chromeOptions": {
            "args": [
                "--no-sandbox",
                "--headless",
                "--disable-dev-shm-usage",
                "--hide-scrollbars"
            ]
        }
    }
}
...
Operation timed out after 180000 milliseconds with 0 bytes received in /app/vendor/php-webdriver/webdriver/lib/Exception/Internal/WebDriverCurlException.php:20

Current Workarounds:

• implemented exponential backoff when the WebDriverCurlException is thrown
• implementing longer timeouts for connection, request, & read.

Versions::

• php: 8.3
• symfony: 6.4
• symfony/panther: 2.1.1
• php-webdriver/webdriver: 1.15.1
• chrome: latest
• chromedriver: latest, stable for chrome

[fnagel]

Same issue here but with a single domain only, so I always think this is a server issue. Can you give more insights on your workarounds so I can test?

Probably related issues: #155

[Capn-Precious]

@fnagel So whenever we attempted to interact with chromedriver/panther, I setup exponential backoff for it to retry 3 times before moving on (this did not end up working at all). Like you said, most likely a server issue.

For example, when creating the chrome client with panther, I set timeouts:

        $this->pantherClient = PantherClient::createChromeClient(
            arguments: [
                '--no-sandbox',
                '--headless',
                '--disable-dev-shm-usage',
                '--hide-scrollbars',
            ],
            options: [
                'connection_timeout_in_ms' => 300000,
                'request_timeout_in_ms' => 300000,
                'read_timeout' => 300,
            ],
        );
    }

One thing I can add is that chrome is constantly making modification to how chrome/chromedriver operates so there are plenty of bugs that reflect this since chrome 126.

The only actual workaround that I didn't post here was that I set the chromedriver version to a working one in our docker container which is the only thing that works.

[fnagel]

@Capn-Precious Thanks for the insight! Still struggling a little, but its already much better than before with longer timeouts. Sadly I have no control of the Chrome version (its "oldschool" hosting, not Docker or similar).

[Capn-Precious]

@fnagel No problem! I'm not even sure if they intend to fix how headless performs in the shared chrome path anytime soon with the number of bugs that continue to pop-up with newer versions.

They do have a way to utilize the old headless binary independently from the new chrome but it looks to only work for puppeteer. I haven't really gotten the chance to dive into that one yet though

For reference: https://developer.chrome.com/docs/chromium/headless