Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Apache Proxy to Docker Container Causes 408 After a Short Delay #672

Open
abegosum opened this issue Dec 28, 2021 · 3 comments
Open

Apache Proxy to Docker Container Causes 408 After a Short Delay #672

abegosum opened this issue Dec 28, 2021 · 3 comments

Comments

@abegosum
Copy link

Current Behavior

I'm attempting to reverse-proxy via Apache on the host machine to a netbox docker container. The initial request seems to work, but after some delay (I typically wait 5 mins for testing), the Apache server will ONLY respond with 408 timeouts. This is unusual, considering I frequently use Apache as a reverse proxy without issue.

Below is a sanitized version of my Apache vhost configuration.

<VirtualHost *:443>
    ServerName netboxlab.mydomain.org
    SSLEngine on
    SSLCertificateFile /path/to/cert.pem
    SSLCertificateKeyFile /path/to/key.pem
    SSLCertificateChainFile /path/to/chain.pem

    ProxyPreserveHost Off
    SSLProxyEngine Off
    ProxyRequests Off
    ProxyPass '/' 'http://127.0.0.1:8000/'
    ProxyPassReverse '/' 'http://127.0.0.1:8000/'

    CustomLog "/logs/apache2/netbox/logs/access_log" combined
    ErrorLog "/logs/apache2/netbox/logs/error_log"
</VirtualHost>

The container logs seem to indicate that Apache is no longer proxying requests.

Things I've tried that don't fix the issue:

  • Setting proxytimeout to 120
  • Setting ping on proxypass to 5
  • Setting keepalive on proxypass to on
  • Pointing proxypass and proxyreversepass to localhost
  • Pointing proxypass and proxyreversepass to the hostname of the machine (with that added to the ALLOWED_HOSTS variable in the docker container)
  • Changed the MPM module from prefork to worker mode

My environment:

Apache Version: 2.4.6
CentOS Version: 7.9.2009
Netbox-Docker Project Tag: 1.5.1
Netbox Image Tag (in docker-compose.override.yml): v3.1

Env vars overridden in .env.prod: DB_PASSWORD, SECRET_KEY, SKIP_SUPERUSER (set to 'true'), POSTGRES_PASSWORD

Expected Behavior

I expect the proxy to continue working after the initial load.

Docker Compose Version

docker-compose version 1.29.2, build 5becea4c

Docker Version

Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:41 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:44:05 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

The git Revision

58a1579

The git Status

# HEAD detached at 1.5.1
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	.env.prod
nothing added to commit but untracked files present (use "git add" to track)

Startup Command

docker-compose up -d

NetBox Logs

Attaching to netbox-docker_netbox_1
netbox_1               | ↩️ Skip creating the superuser
netbox_1               | 🧬 loaded config '/etc/netbox/config/configuration.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/extra.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/logging.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/plugins.py'
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/000_users.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/010_groups.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/015_object_permissions.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/020_custom_fields.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/020_tags.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/020_tenant_groups.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/030_regions.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/030_tenants.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/040_sites.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/050_manufacturers.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/060_device_types.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/070_rack_roles.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/075_locations.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/080_racks.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/090_device_roles.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/100_platforms.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/130_cluster_types.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/135_cluster_groups.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/135_clusters.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/140_clusters.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/140_devices.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/150_rirs.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/160_aggregates.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/165_cluster_groups.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/175_route_targets.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/180_vrfs.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/190_prefix_vlan_roles.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/200_vlan_groups.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/210_vlans.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/220_prefixes.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/230_virtual_machines.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/240_virtualization_interfaces.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/250_dcim_interfaces.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/260_ip_addresses.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/270_primary_ips.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/280_custom_links.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/280_providers.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/290_circuit_types.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/290_webhooks.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/300_circuits.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/320_services.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/330_power_panels.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/340_power_feeds.py
netbox_1               | ✅ Initialisation is done.
netbox_1               | ⏳ Waiting for control socket to be created... (1/10)
netbox_1               | 2021/12/28 18:21:41 [warn] 9#9 Unit is running unprivileged, then it cannot use arbitrary user and group.
netbox_1               | 2021/12/28 18:21:41 [info] 9#9 unit started
netbox_1               | 2021/12/28 18:21:41 [info] 17#17 discovery started
netbox_1               | 2021/12/28 18:21:41 [notice] 17#17 module: python 3.9.5 "/usr/lib/unit/modules/python3.unit.so"
netbox_1               | 2021/12/28 18:21:41 [info] 9#9 controller started
netbox_1               | 2021/12/28 18:21:41 [notice] 9#9 process 17 exited with code 0
netbox_1               | 2021/12/28 18:21:41 [info] 19#19 router started
netbox_1               | 2021/12/28 18:21:41 [info] 19#19 OpenSSL 1.1.1l  24 Aug 2021, 101010cf
netbox_1               | ⚙️ Applying configuration from /etc/unit/nginx-unit.json
netbox_1               | 2021/12/28 18:21:42 [info] 23#23 "netbox" application started
netbox_1               | ✅ Unit configuration loaded successfully
netbox_1               | 2021/12/28 18:21:44 [notice] 9#9 process 15 exited with code 0
netbox_1               | 2021/12/28 18:21:48 [info] 32#32 "netbox" application started
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:50 +0000] "GET / HTTP/1.1" 200 86116 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:50 +0000] "GET /static/netbox-external.css HTTP/1.1" 200 286568 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:50 +0000] "GET /static/netbox-light.css HTTP/1.1" 200 493648 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:50 +0000] "GET /static/netbox-dark.css HTTP/1.1" 200 788892 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:50 +0000] "GET /static/netbox.js HTTP/1.1" 200 374756 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:51 +0000] "GET /static/netbox_logo.svg HTTP/1.1" 200 4719 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:51 +0000] "GET /static/netbox_icon.svg HTTP/1.1" 200 835 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:51 +0000] "GET /static/netbox-print.css HTTP/1.1" 200 1623714 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:51 +0000] "GET /static/materialdesignicons-webfont-KSYPMDN6.woff2?v=5.9.55 HTTP/1.1" 200 325244 "http://localhost:8000/static/netbox-external.css" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:54 +0000] "GET /login/ HTTP/1.1" 200 8228 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:22:08 +0000] "POST /login/ HTTP/1.1" 302 0 "http://localhost:8000/login/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:22:09 +0000] "GET / HTTP/1.1" 200 142800 "http://localhost:8000/login/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 🧬 loaded config '/etc/netbox/config/configuration.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/extra.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/logging.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/plugins.py'
netbox_1               | 2021/12/28 18:24:09 [notice] 9#9 process 32 exited with code 0
netbox_1               | 2021/12/28 20:01:49 [notice] 9#9 process 18 exited with code 0
netbox_1               | 🧬 loaded config '/etc/netbox/config/configuration.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/extra.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/logging.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/plugins.py'
netbox_1               | 2021/12/28 20:01:49 [notice] 9#9 process 19 exited with code 0
netbox_1               | 2021/12/28 20:01:49 [notice] 9#9 process 23 exited with code 0

Content of docker-compose.override.yml

version: '3.4'
  
x-servicebase: &servicebase
  restart: always

x-netboxbase: &netboxbase
  <<: *servicebase
  image: netboxcommunity/netbox:v3.1
  env_file:
    - env/netbox.env
    - .env.prod

services:
  netbox:
    <<: *netboxbase
    ports:
      - 8000:8080
  netbox-worker:
    <<: *netboxbase
  netbox-housekeeping:
    <<: *netboxbase

  postgres:
    <<: *servicebase
    env_file:
      - env/postgres.env
      - .env.prod

  redis:
    <<: *servicebase
  redis-cache:
    <<: *servicebase
@abegosum
Copy link
Author

While not a solution, I have identified a workaround.

The 408 errors seem to be related to some interaction between the Python app and the Apache proxy connections. Apache typically reuses these connections for efficiency's sake, but something about that reuse is causing the error. At least in our environment, disabling reuse of proxy connections has caused the issue to go away:

ProxyPass '/' 'http://127.0.0.1:8000/' disablereuse=on

@nawlbergs
Copy link

this is happening for me as well... after a few minutes... refreshing my browser just spins... Restarting container does nothing and I have to restart docker desktop to get it working again. (oddly after restart... it shows my postgres db still running)... i kill that then restart container and it works again.. but it was consistently going into a non-responsive state after a few minutes.

@marcquark
Copy link
Contributor

Could be related to #854 - perhaps the workaround posted there is helpful?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nawlbergs @abegosum @marcquark