Author: Mike Lake
Version of this Doc: 2015.03.23
Description
New Version Release at Nectar
Software Required
System Setup at Nectar
Two Programs - choice.py and process_choices.py
Running the Application
Running Regression Tests
Running under Microsoft Windows
Problems Encountered
Programming References
This describes the python scripts that run the web based "Discrete Choice Experiments". This was written for for Deborah Street and Leonie Burgess. Mike Lake wrote the choice.py and Emily Bird (from Maths) wrote the main functions in process_choices.py.
This is how to do a new release at the Nectar hosted site.
-
Update the version number for the main program:
choice.py
e.g. version = '2013.10.23'The version numbers for the other programs remain as-is; i.e.
choice_common.pyandprocess_choices.py -
Checkin changes and push to github.
-
Run the script
./release_to_nectar.sh
This will just make a tarball of the required files and scp the tarball to Nectar's/home/ec2-user/public_html/. It will then echo to the screen the command for logging into Nectar. -
Login to Nectar (this is done automatically by the above script)
$ ssh -i ~/.ssh/keys/mikes_nectar [email protected]Your home directory will be
/home/ec2-user/
The tarball will be in public_html/$ cd public_html $ tar xvf choice_release_2014.01.07.tar <== This will create directory choice_2014.01.07/ $ rm choice_release_2014.01.07.tar $ sudo systemctl stop nginx.service $ rm choice <== This is a symlink to a release. $ ln -s choice_release_2014.01.07 choice <== Create new symlink. $ sudo systemctl start nginx.service $ sudo systemctl restart uwsgi.service -
Check http://XXX.XXX.XXX.XXX/choice It should show the correct version number at the bottom left of the page.
If you just want an archive of the current repo then:
$ git archive HEAD --prefix=choice/ --output ../choice.tar
The following required software will be installed onto a "Fedora release 19" release.
First "sudo yum update" then install the following Fedora packages:
Fedora Package Provides
-------------- --------
python Python 2.7.5 will already be installed.
python-bottle Provides micro web framework
python-devel Required for a pip install of subprocess32-3.2.6
numpy Provides numerical array maths
gcc Required for a pip install of subprocess32-3.2.6
nginx Provides web server
uwsgi Provides /usr/sbin/uwsgi and other files in /etc/ and /usr/share/doc/uwsgi
uwsgi-plugin-python Only provides /usr/lib64/uwsgi/python_plugin.so
uwsgi-plugin-common Pulled in as a dependeny of uwsgi-plugin-python
Note: on my laptop I have installed:
mod_wsgi <-- this is for Apache
/etc/httpd/conf.modules.d/10-wsgi.conf
/usr/lib64/httpd/modules/mod_wsgi.so
/usr/share/doc/mod_wsgi
/usr/share/doc/mod_wsgi/LICENCE
/usr/share/doc/mod_wsgi/README
sympy: The python package called sympy is available from from http://sympy.org.
This provides symbolic maths for python. From that site download and install sympy-0.7.2.tar.gz
Note: Do not install sympy from the yum repos. That will install version 0.7.3
which has a problem. Download 0.7.2 and install using setup.py as below.
sympy-sympy-0.7.2$ python setup.py build
sympy-sympy-0.7.2$ sudo python setup.py install
subprocess32-3.2.6: "subprocess" is the native one for Python 2.7 but I have installed "subprocess32" (via "pip install") which is a backport from Python 3.0 which supports a timeout arg.
Added via pip install: subprocess32-3.2.6
$ python setup.py build
$ sudo python setup.py install
OK
Note: if you have not installed python-devel via yum then when you do the "python setup.py build" step you will get an error "Python.h: No such file or directory". This package also needs gcc so ensure gcc yum package is installed.
Mike made some global changes to nginx configuration files. The changes are documented in these files. Search for MRL strings for changes.
/etc/nginx/nginx.conf Changed document root, changed log format to a simpler string.
/etc/nginx/conf.d/default.conf Contains hello and choice apps.
/usr/share/nginx/html/index.html Changed to just say "Nothing here."
/home/ec2-user/public_html Created public_html directory.
Changed permissions of /home/ec2-user/ so nginx can see the public_html dir.
from: drwx------
to: drwxr-xr-x
Check nginx configuration:
$ sudo /usr/sbin/nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
$
Change ownership and mode of choice python files:
choice$ chown ec2-user:nginx *.py
choice$ chmod ug+x *.py
Make sure that nginx starts at boot time:
$ sudo systemctl enable nginx.service
ln -s '/usr/lib/systemd/system/nginx.service' '/etc/systemd/system/multi-user.target.wants/nginx.service'
$
TODO deployment: nginx+uwsgi
TODO nohup python app.py & - run in background with ability to logout from console.
In /etc/nginx/nginx.conf
# TODO logs outside of http or server do ???
#access_log /var/log/nginx/access.log main;
#error_log /var/log/nginx/error.log;
#error_log /var/log/nginx/error.log notice;
#error_log /var/log/nginx/error.log info;
Package uwsgi will contain /usr/sbin/uwsgi and some doc files.
$ rpmquery -ql uwsgi | grep -v doc
/etc/uwsgi.d <-- was empty, added choice.ini & hello.ini /etc/uwsgi.ini /run/uwsgi /usr/lib/systemd/system/uwsgi.service /usr/sbin/uwsgi
File: /etc/uwsgi.ini
[uwsgi] uid = uwsgi <-- drops privs from root to these after starting. TODO uwsgi or nginx gid = uwsgi pidfile = /run/uwsgi/uwsgi.pid emperor = /etc/uwsgi.d stats = /run/uwsgi/stats.sock emperor-tyrant = false cap = setgid,setuid
File: /etc/uwsgi.d/choice.ini
[uwsgi] uid = nginx gid = nginx socket = 127.0.0.1:9000 plugin = python wsgi-file = /home/ec2-user/public_html/choice/choice.py chdir = /home/ec2-user/public_html/choice/ process = 3
File: /etc/uwsgi.d/hello.ini
[uwsgi] uid = nginx gid = nginx socket = 127.0.0.1:9090 plugin = python wsgi-file = /home/ec2-user/public_html/hello/hello.py process = 1
This is what we should see:
uwsgi is running:
$ ps ax | grep uwsgi
325 ? Ss 1:11 /usr/sbin/uwsgi --ini /etc/uwsgi.ini
341 ? S 0:43 /usr/sbin/uwsgi --ini choice.ini
343 ? S 0:43 /usr/sbin/uwsgi --ini hello.ini
381 ? S 0:00 /usr/sbin/uwsgi --ini hello.ini
407 ? S 0:01 /usr/sbin/uwsgi --ini choice.ini
$
~/$ ps axjf | grep wsgi
1 325 325 325 ? -1 Ss 996 1:11 /usr/sbin/uwsgi --ini /etc/uwsgi.ini
325 341 325 325 ? -1 S 996 0:43 \_ /usr/sbin/uwsgi --ini choice.ini
341 407 325 325 ? -1 S 996 0:01 | \_ /usr/sbin/uwsgi --ini choice.ini
325 343 325 325 ? -1 S 996 0:43 \_ /usr/sbin/uwsgi --ini hello.ini
343 381 325 325 ? -1 S 996 0:00 \_ /usr/sbin/uwsgi --ini hello.ini
~/$
$ sudo netstat -tlpn | grep uwsgi tcp 0 0 127.0.0.1:9090 0.0.0.0:* LISTEN 343/uwsgi tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 341/uwsgitcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 8509/uwsgitcp 0 0 127.0.0.1:9001 0.0.0.0:* LISTEN 8507/uwsgi$
Here is how to start the uwsgi service:
$ sudo systemctl start uwsgi.service
$ ps axjf | grep wsgi 0:00 /usr/sbin/uwsgi --ini /etc/uwsgi.ini 0:00 \_ /usr/sbin/uwsgi --ini /etc/uwsgi.ini
Make sure uWSGI starts at boot time:
$ sudo systemctl enable uwsgi.service ln -s '/usr/lib/systemd/system/uwsgi.service' '/etc/systemd/system/multi-user.target.wants/uwsgi.service' $
Note: /etc/uwsgi/ directory containing emperor.ini and vassals/ is not part of the uwsgi package.
At Nectar we are running a systemd based init system which replaces the older SysV init system. See /etc/init.d/README. Files are in: /etc/systemd/system and some of these are symlinks to files in /usr/lib/systemd/system
Basic usage:
systemctl start nginx.service systemctl stop nginx.service systemctl status nginx.service systemctl list-unit-files
If you use the older SysV system command: $ service uwsgi status it will redirect it with a message "Redirecting to /bin/systemctl status uwsgi.service"
The Choice application page should now be visible at: http://130.56.248.113/choice/
This is the web form submission process.
Start here
+--- Users Web Browser ---+
| |
| Javascript validation | <-- Note 1
| for basic checking |
+-------------------------+
|
User submits form
|
+--- choice.py: web application ---+
| |
| gets inputs from web browser | +--- process_choices.py ----+
| inputs_validation() | | |
| | | get_expected_io() |
| write_input_files() ------------------> | read_input_files() |
| | | ... processing ... |
| | | ... processing ... |
| read_output_files() <------------------ | write_output_files() |
| | +---------------------------+
| creates HTML pages with data |
+----------------------------------+
|
Results sent back to user
|
+--- Users Web Browser ---+
| |
| results |
+-------------------------+
This is roughly what each program does.
choice.py: |
process_choices.py: |
Note 1: To turn off the javascript validation edit views/head.tpl and rename the src="/static/validate.js" to something "wrong" so it won't load.
This is just for testing - not production!
The script choice.py needs to have this at the end:
run(host='localhost', port=8080, reloader=True)
Start the script:
choice/$ ./choice.py Bottle server starting up (using WSGIRefServer())... Listening on http://localhost:8080/ Hit Ctrl-C to quit.
Go to: http://localhost/choice
Note: The port=8080 is superfluous as the default port is actually 8080. If you try to run on port 80 you will get a "socket.error: [Errno 13] Permission denied error" as you don't have permission to bind that that port - even if apache or nginx are not running.
This is how to run for production.
The script choice.py needs to have this:
run(server=FlupFCGIServer, port=9000, host='localhost', reloader=True)
Start the script:
choice/$ ./choice.py Bottle server starting up (using FlupFCGIServer())... Listening on http://127.0.0.1:9000/ Hit Ctrl-C to quit.
or
$ nohup python ./choice.py & <-- Runs COMMAND, ignoring hangup signals. nohup: ignoring input and appending output to "nohup.out"
Go to: http://localhost/choice/
[ec2-user@choice2 ~]$ ps ax | grep choice
2:11 avahi-daemon: running [choice2.local]
1:31 python ./choice.py
3:14 /usr/bin/python ./choice.py
[ec2-user@choice2 ~]$
If you get this web page below then you forgot to start ./choice.py
Possibly Busy The page you are looking for is temporarily unavailable. Please try again later.
Note: For Mikes laptop only make sure nginx is running and not apache:
$ sudo service httpd stop Redirecting to /bin/systemctl stop httpd.service choice/$ sudo service nginx start Redirecting to /bin/systemctl start nginx.service $
Run the script to show usage. For each test there should be no output if the test passes.
$ ./regression_test.sh $
The web server and its app (i.e. choice.py) can't be run under Windows but the program process_choices.py can be run under Windows. This means that Emily Bird can run and debug her code on Windows. For details on this see below.
1. Copy the following files over to the Windows box:
process_choices.py and choice_common.py
2. Copy the directory test with its test files to the Windows box.
Then run it like this:
C:\choice_emily> C:\Python27\python.exe process_choices.py test\check_main_1 check main C:\choice_emily>
Note: Regression tests also can't be run under Windows.
Error is:
2014/04/29 13:10:04 [error] 10640#0: *9 connect() failed (111: Connection refused) while connecting to upstream, client: 113.197.9.114, server: , request: "GET /hello/ HTTP/1.1", upstream: "uwsgi://127.0.0.1:9090", host: "130.56.248.113"
This error Means nginx is trying to pass along the app to port 9090 but nothing seems to be listening there.
But we do seem to have uwsgi listening OK - see below:
10848 ? Ss 0:00 /usr/sbin/uwsgi --ini /etc/uwsgi.ini
10850 ? S 0:00 /usr/sbin/uwsgi --ini /etc/uwsgi.ini
$ sudo /usr/sbin/uwsgi --ini /etc/uwsgi.ini [uWSGI] getting INI configuration from /etc/uwsgi.ini setting capability setgid [6] setting capability setuid [7] *** Starting uWSGI 1.9.19 (64bit) on [Thu May 1 14:19:04 2014] *** compiled with version: 4.8.2 20131017 (Red Hat 4.8.2-1) on 12 November 2013 18:02:19 os: Linux-3.13.9-100.fc19.x86_64 #1 SMP Fri Apr 4 00:51:59 UTC 2014 nodename: uts-choice.novalocal machine: x86_64 detected number of CPU cores: 1 current working directory: /home/ec2-user writing pidfile to /run/uwsgi/uwsgi.pid detected binary path: /usr/sbin/uwsgi setgid() to 996 setuid() to 996 *** WARNING: you are running uWSGI without its master process manager *** your processes number limit is 1024 *** starting uWSGI Emperor *** [emperor-tyrant] invalid permissions for vassal choice.ini [emperor-tyrant] invalid permissions for vassal hello.ini [emperor-tyrant] invalid permissions for vassal choice.ini [emperor-tyrant] invalid permissions for vassal hello.ini ^C[emperor] *** RAGNAROK EVOKED *** $
2015.03.20 Estimating main effects only.
This shows an nginx error (see screenshot) which says:
"The page you are looking for is temporarily unavailable".
1. Test via command line:
choice/$ ./process_choices.py $dir $operation $effects
time ./process_choices.py test/2015.03.20 check main
Time taken: 2.5 minutes
Creates: out_bmat.dat OK
out_cinv.dat OK
out_cmat.dat OK
out_correln.dat OK
out_lmat.dat OK
out_msg.dat OK
2. Test via http://localhost:8080/:
Make sure TEST exists.
Enter matrices then...
[submit]
temp/$ ps ax | grep python
1010 ? S 0:00 /usr/bin/python /usr/bin/denyhosts.py --daemon --config=/etc/denyhosts.conf
29614 pts/0 S+ 0:00 python ./choice.py
29615 pts/0 Sl+ 0:01 /bin/python ./choice.py
30337 pts/0 R+ 0:18 python ./process_choices.py temp check main
30394 ? Sl 0:01 /usr/bin/python -Es /usr/sbin/setroubleshootd -f
30856 pts/1 S+ 0:00 grep --color=auto python
temp/$
All works fine. It just times out with nginx.
PS. Saved the output files to the test directory test/2015.03.20
3. Test via nginx:
Make sure TEST does NOT exist.
$ uwsgi -- /etc/uwsgi/vassals/choice.ini
User sees this:
504 Gateway Time-out
Server nginx error logs shows this:
2015/04/07 14:56:30 [error] 4818#0: *13 upstream timed out (110: Connection timed out)
while reading response header from upstream, client: 127.0.0.1,
server: localhost, request: "POST /choice/process/ HTTP/1.1", upstream:
"uwsgi://127.0.0.1:9001", host: "localhost", referrer: "http://localhost/choice/"
This occurs at 60 seconds, even if I have in /etc/nginx.conf
# These timeouts don't help in preventing "504 Gateway Time-out"
#keepalive_timeout 65;
#keepalive_timeout 200;
#uwsgi_connect_timeout 200;
#uwsgi_send_timeout 200;
# This timeout is the one required!
uwsgi_read_timeout 200;
Now ....
$ uwsgi -- /etc/uwsgi/vassals/choice.ini
...
spawned uWSGI worker 1 (and the only) (pid: 10925, cores: 1)
[pid: 10925|app: 0|req: 1/1] 127.0.0.1 () {44 vars in 747 bytes} [Tue Apr 7 15:28:00 2015] POST /choice/process/ =>
generated 6536801 bytes in 148932 msecs (HTTP/1.1 200) 2 headers in 84 bytes (2 switches on core 0)
and
No nginx timeout !!!! Good
See the TODO.txt for timeout information.
The problem was that the uwsgi process could not be started.
[ec2-user@choice2 hello]$ uwsgi --http-socket :9090 --plugin python --wsgi-file hello.py
open("/usr/lib64/uwsgi/python_plugin.so"): No such file or directory [core/utils.c line 3639]
!!! UNABLE to load uWSGI plugin: /usr/lib64/uwsgi/python_plugin.so: cannot open shared object file: No such file or directory !!!
uwsgi: unrecognized option '--wsgi-file'
getopt_long() error
[ec2-user@choice2 hello]$
Installed uwsgi-plugin-python
[ec2-user@choice2 hello]$ uwsgi --http-socket :9090 --plugin python --wsgi-file hello.py *** Starting uWSGI 1.9.19 (64bit) on [Tue Jan 7 12:21:26 2014] *** compiled with version: 4.8.2 20131017 (Red Hat 4.8.2-1) on 12 November 2013 18:02:19 os: Linux-3.10.10-200.fc19.x86_64 #1 SMP Thu Aug 29 19:05:45 UTC 2013 Now lynx http://localhost:9090 shows OK
This is what you should have:
[ec2-user@choice2 public_html]$ ps ax | grep choice 7235 pts/0 S 0:00 python ./choice.py 7236 pts/0 Sl 0:00 /usr/bin/python ./choice.py [ec2-user@choice2 public_html]$
If you get this error on Nectar: Error: 502 Bad Gateway then chances are that nginx is runing but the choice.py app is not. Try running it manually from the command line to see any errors. Don't background it so you can see what the problem is.
$ python ./choice.py <-- no & to see what the problem is.
Note: Get the sympy version with "import sympy; sympy.__version__"
Python Sympy Numpy Laptop 2.7.3 0.7.1 1.6.2 Windows 2.7.3 0.7.2 1.7.1 Nectar 2.7.5 0.7.3 1.7.1 <-- has a sympy problem Nectar 2.7.5 0.7.2 1.7.1 <-- works OK LiveShell 2.7.5 0.7.3 1.6.1
On laptop running sympy 0.7.1
mlake$ python Python 2.7.3 (default, Jul 24 2012, 10:05:38) [GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2 >>> import sympy >>> m=[1,2] >>> sympy.Matrix(m) [1] [2] >>>
On Nectar server running sympy 0.7.3
[ec2-user@choice2$ python Python 2.7.5 (default, Aug 22 2013, 09:31:58) [GCC 4.8.1 20130603 (Red Hat 4.8.1-1)] on linux2 >>> import sympy >>> m=[1,2] >>> sympy.Matrix(m) Matrix([ [1], [2]]) >>>
Solved: Removed the yum package sympy (0.7.3) and installed sympy 0.7.2 from the tarball source package sympy-sympy-0.7.2.tar. The problem has now gone.
How (not) to set a timeout on a computation in Python Set timeout for a shell command in python Monica Lent: set-up-nginx-and-uwsgi Good definitions here Bottle + UWSGI + Nginx Quickstart How to serve WSGI using Apache Adding the Emperor to systemd Running multiple python apps with nginx and uwsgi in emperor mode - March 2012 Using the uWSGI Emperor with systemd NginX + uWSGI + Bottle WSGI Explorations in Python Good overview PythonFlup HttpUwsgiModule bottle-virtualenv-uwsgi-nginx-installation-on-ubuntu-12-04-1-lts/ how-to-gracefully-restart-django-running-fcgi-behind-nginx Mod_wsgi_notes bottle-uwsgi-nginx-quickstart customizing-nginx-web-logs running-python-through-fastcgi-for-nginx Webpy-deployment-through-Nginx-Fastcgi-Spawn-fcgi-and-Flup setting-up-nginx-django-uwsgi-a-tutorial-that-actually-works nginx-uwsgi-django-flask-deployment