Server crashing

[jefferya]

The number of TCP connections to Apache web server with a status of "CLOSE_WAIT" will increase until the Apache web server max connection limit is reached and the site is no longer responsive.

netstat -tanp | grep ':443 ' | grep httpd | awk '{print $6}' | sort | uniq -c | sort -n
      1 LISTEN
     18 ESTABLISHED
    526 CLOSE_WAIT


netstat -tanp | grep ':443 ' | grep httpd | awk '{print $6}' | sort | uniq -c | sort -n
      1 LISTEN
     19 ESTABLISHED
    557 CLOSE_WAIT

This is normal

netstat -tanp | grep ':443 ' | grep httpd | awk '{print $6}' | sort | uniq -c | sort -n
      1 CLOSE_WAIT
      1 FIN_WAIT2
      1 LISTEN
     15 ESTABLISHED

netstat -tanp | grep ':443 ' | grep httpd | awk '{print $6}' | sort | uniq -c | sort -n
      1 ESTABLISHED
      1 FIN_WAIT2
      1 LISTEN

Temporary fix, restart httpd

sudo service httpd restart

Hypothesis, something on the server holds the server side of the connection open after the client side has sent a close. https://stackoverflow.com/questions/15912370/how-do-i-remove-a-close-wait-socket-connection. The server goes from functioning normally for several days (properly closing connections) to holding open connections. Is there a certain page or module causing the problem as the problem is not continuously visible.

[SusanBrown]

Re: your strategy of Temporary fix, restart httpd sudo service httpd restart Would it be possible to script a restart every couple of days? From: Jeffery Antoniuk ***@***.***> Date: Wednesday, February 14, 2024 at 7:28 PM To: cwrc/beta.cwrc.ca ***@***.***> Cc: Subscribed ***@***.***> Subject: [cwrc/beta.cwrc.ca] Server crashing (Issue #52) CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to ***@***.*** The number of TCP connections to Apache web server with a status of "CLOSE_WAIT" will increase until the Apache web server max connection limit is reached and the site is no longer responsive. netstat -tanp | grep ':443 ' | grep httpd | awk '{print $6}' | sort | uniq -c | sort -n 1 LISTEN 18 ESTABLISHED 526 CLOSE_WAIT netstat -tanp | grep ':443 ' | grep httpd | awk '{print $6}' | sort | uniq -c | sort -n 1 LISTEN 19 ESTABLISHED 557 CLOSE_WAIT Temporary fix, restart httpd sudo service httpd restart Hypothesis, something on the server holds the server side of the connection open after the client side has sent a close. https://stackoverflow.com/questions/15912370/how-do-i-remove-a-close-wait-socket-connection. The server goes from functioning normally for several days (properly closing connections) to holding open connections. Is there a certain page or module causing the problem as the problem is not continuously visible. — Reply to this email directly, view it on GitHub<#52>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAEFJIH53QUPILEGTATUISLYTVJBNAVCNFSM6AAAAABDJK66ROVHI2DSMVQWIX3LMV43ASLTON2WKOZSGEZTKNBRHA3TMNY>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jefferya]

I've been working on this assumption that general load was the primary cause for the last 6 months but I've not found a causal relationship. A yet unidentified specific type of load might be related to the cause (my hypothesis, a defect/bug in the webserver version, Drupal 7, or CWRC modules).

The LEAF version doesn't seem to be impacted.

I'm adding an automated restart of the web server .