Skip to content

Installation Instructions

Jump to bottom Edit New page
anenadic edited this page Aug 29, 2013 · 45 revisions

The following installation instructions contain steps to stand up your own ServiceCatalogue application/service using the ServiceCatalographer software and refer to Ubuntu 12.04. Unless you are using Ubuntu, you will have to adapt them to suit your operating system.

User accounts

  • You will need a user with superuser privileges to install the necessary libraries
  • www-data: non-privileged user that Apache2 switches to when running is used to run the ServiceCatalogue application. To switch to it, you can do e.g. sudo su -lp www-data

Required software

Running the ServiceCatalogue application requires several pieces of software:

  • ServiceCatalographer software (from this repository)
  • Apache2 + Phusion Passenger (mod_rails/mod_rack) to run the application in production mode
  • memcached server (for caching pages, etc.)
  • mySQL server (for the ServiceCatalogue database)
  • Solr search server (to power the application's search engine; requires Java)
  • WSDLUtils service for parsing WSDLs (requires PHP)
  • "workers" for various background jobs (e.g. service availability monitoring, etc.)

Managed packages

Some of the prerequisite packages required:

  • Java (needed to run the Solr search server)
  • apache2
  • openssl
  • git-core
  • mysql-server
  • memcached
  • cronolog
  • php5 (needed for the WSDLUtils service)
  • php5-cgi
  • php-xml-parser
  • php-xml-serializer

Ruby

It is suggested to use RVM to manage your Ruby installations.

RVM

Get it from https://rvm.io/rvm/install/. Use multi-user install into /usr/local. Add you superuser and user www-data to group rvm. Run rvm requirements.

Ruby

rvm install 1.9.3

Rubygems

rvm rubygems

Bundler

gem install bundler

Gems

gem install passenger gem install rails --version '3.2.13'

Make sure that Rails installation is system-wide, as Phusion Passenger needs to be able to locate Rails.

Apache2 and Phusion Passenger

There are different ways to host Ruby on Rails application as a production service; we choose to use Apache2 together with the Apache module Phusion Passenger (a.k.a mod_rails or mod_rack).

We also chose to host the app via HTTPS; the Apache2 settings given below are configured to silently forward any HTTP requests to HTTPS. Note that you will need a proper (not self-signed) certificate to run the application.

If you want just to try out the ServiceCatalogue application or develop it - you can run it in the development mode for which you do not need Apache2, Passenger nor certificates. The app will run in a built-in Web server that comes with Rails (on port 3000 by default).

Make sure you have passenger gem installed (see above).

Install Passenger module for Apache:

rvmsudo passenger-install-apache2-module

Create /etc/apache2/mods-available/passenger.load file (an example shown below - modify to suit your system):

LoadModule passenger_module /usr/local/rvm/gems/ruby-1.9.3-p448/gems/passenger-3.0.19/ext/apache2/mod_passenger.so`

Create /etc/apache2/mods-available/passenger.conf file (an example shown below - modify to suit your system):

PassengerRoot /usr/local/rvm/gems/ruby-1.9.3-p448/gems/passenger-3.0.19
PassengerRuby /usr/local/rvm/wrappers/ruby-1.9.3-p448/ruby
PassengerMaxPoolSize 50
PassengerPoolIdleTime 300

Create /etc/apache2/sites-available/catalogue-ssl file (an example shown below - modify to suit your system):

<VirtualHost *:80>
ServerName www.mycatalogue.org
ServerAdmin [email protected]
Redirect permanent / https://www.mycatalogue.org/ # Do not forget the trailing /
</VirtualHost>
 
<VirtualHost *:443>
ServerName www.mycatalogue.org
ServerAdmin [email protected]
 
# This is the Catalogue Rails app's location.
DocumentRoot /var/webapps/mycatalogue/public
PassengerMinInstances 5
<Directory /var/webapps/mycatalogue/public>
Options -MultiViews
AllowOverride None
Order allow,deny
Allow from all
</Directory>
 
# Allow direct access to WSDLUtils via Apache, avoiding Passenger.
Alias /WSDLUtils /var/www/WSDLUtils
<Directory /var/www/WSDLUtils>
PassengerEnabled off
Order allow,deny
#Allow from 127.0.0.0/255.0.0.0 ::1/128
# Can't use localhost for this due to SSL!
Allow from all
</Directory>
 
ErrorLog /var/log/apache2/error.log
 
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn
 
CustomLog /var/log/apache2/ssl_access.log combined
 
SSLEngine on
SSLCertificateFile /etc/apache2/ssl/mycatalogue.crt
SSLCertificateKeyFile /etc/apache2/ssl/mycatalogue.key
SSLCertificateChainFile /etc/apache2/ssl/mycatalogue.ca-bundle
 
# Deal with crappy browsers
BrowserMatch "MSIE [2-6|2-6]"
nokeepalive ssl-unclean-shutdown
downgrade-1.0 force-response-1.0
# MSIE 7 and newer should be able to use keepalive
BrowserMatch "MSIE [7-9|7-9]" ssl-unclean-shutdown
</VirtualHost>

Enable SSL module (if it is not enabled yet):

sudo a2enmod ssl

Enable Passenger module (if it is not enabled yet):

sudo a2enmod passenger

Disable the Apache default site settings:

sudo a2dissite default

Enable the site settings for your site (the ServiceCatalogue app):

sudo a2ensite mycatalogue-ssl

HTTPS

You can generate a new private key and CSR (certificate Signing Request) for your Apache2 server with, e.g.: sudo openssl req -newkey rsa:2048 -days 3650 -nodes -keyout mycatalogue.key -out mycatalogue.csr

You will probably also need to add the domain name in /etc/hosts: 127.0.0.1 localhost www.mycatalogue.org

MySQL

If you do not have it already, install MySQL by running the following command:

sudo apt-get install mysql-server

On Ubuntu, once the installation is complete, the MySQL server should start automatically. You can run the following command from a terminal prompt to check whether the MySQL server is running:

sudo netstat -tap | grep mysql

When you run this command, you should see the following line or something similar:

:tcp 0 0 localhost:mysql *:* LISTEN 2556/mysqld

If the server is not running correctly, you can type the following command to start it:

sudo service mysql restart

MySQL user accounts

Create a mySQL user for the ServiceCatalogue app, e.g. cat. This user will need access to the ServiceCatalogue's production, development and test databases (explained below). Make sure to note the names you give to the databases.

Note that you only need the production database if you are not planning on developing or testing the app.

Actions:

CREATE USER 'cat'@'localhost' IDENTIFIED BY '<type_some_password_here>';
 
create database catalogue_test;
create database catalogue_dev;
create database catalogue_prod;
 
GRANT ALL PRIVILEGES ON catalogue_test.* TO 'cat'@'localhost';
GRANT ALL PRIVILEGES ON catalogue_dev.* TO 'cat'@'localhost';
GRANT ALL PRIVILEGES ON catalogue_prod.* TO 'cat'@'localhost';

ServiceCatalogue software

Presumably you have already cloned this repository (say to var/webapps/servicecatalogue). It not - do it now.

git clone git://github.com/myGrid/servicecatalographer.git

Run bundle install

bundle install --deployment

The --deployment option tells Bundler to install all gems local to the application.

Copy the template configuration files and edit suit your environment (configuring the app is explained in more details below):

cp config/preinitializer.rb.pre config/preinitializer.rb
cp config/memcache.yml.pre config/memcache.yml
cp config/sunspot.yml.pre config/sunspot.yml
cp config/database.yml.pre config/database.yml
cp config/initializers/biocat_local.rb.pre config/initializers/biocat_local.rb
cp config/initializers/mail.rb.pre config/initializers/mail.rb
cp public/stylesheets/colour-override.css.pre public/stylesheets/colour-override.css

Copy and edit the service categories:

cp data/service_categories.yml.pre data/service_categories.yml

Create some required files and directories:

mkdir tmp
mkdir tmp/cache tmp/pids tmp/sessions tmp/sockets

Once mySQL database has been installed and databases needed by the ServiceCatalogue created as explained above, do:

  • Firstly, temporarily set ENABLE_SEARCH to false in config/initializers/biocat_local.rb
  • Run bundle exec rake db:migrate RAILS_ENV=production or bundle exec rake db:schema:load RAILS_ENV=production
  • Run rake db:migrate RAILS_ENV=production

For development environment, you can just simply omit the "RAILS_ENV=..." bit.

Now make sure that you recursively change the ownership of the servicecatalogue folder (where you checkout the code from GitHub) to www-data and that you run the application as that user:

sudo chown -R www-data:www-data servicecatalogue

Make sure to turn log rotation on (in config/application.rb) as log files may fill the disk space.

WSDLUtils

WSDLUtils is a PHP application that does the WSDL parsing for the ServiceCatalogue.

To get it, do:

git clone https://github.com/myGrid/WSDLUtils.git

Make sure to point your Apache configuration to the location of the WSDLUtils installation.

Also check the README.txt file you just downloaded.

Running the ServiceCatalogue application

Configuration

  • config/preinitializer.rb

Not much to do here - log rotation and cron job parameters.

  • config/memcache.yml

Configure the address and port of the memcached server.

  • config/sunspot.yml

Configure the address and port of the Sunspot Solr search server.

Make sure you also enable search in config/initializers/biocat_local.rb.

  • config/database.yml

Configure your database connection and the name of databases you created previously.

  • config/initializers/biocat_local.rb

Enable search via Solr by setting ENABLE_SEARCH = true.

Configure the support email address where user feedback will go to as well as any questions from the users.

Alternative methods of authentication can be configured via RPXNow site/plugin configuration parameters to allow users to log in via their accounts at FaceBook, Twitter, Google, etc. This requires the application administrator to register the app with RPXNow and generate a secret app key.

  • config/initializers/mail.rb

Need to ensure there is something in the :domain field and that starttls is enabled. See the SES docs for more information.

Also, make sure that ActionMailer::Base.perform_deliveries is set to true.

ActionMailer::Base.perform_deliveries = true`
ActionMailer::Base.delivery_method = :smtp
ActionMailer::Base.smtp_settings = {
:address              => 'gmail.com',
:port                 => 587,
:domain               => "gmail.com",
:user_name            => "[email protected]",
:password             => "pie5ahBo0aih",
:authentication       => :plain,
:enable_starttls_auto => true
}
  • public/stylesheets/colour-override.css

If you want to change the colours and style used by the BioCatalogue app, this is the place to do it.

Starting and stopping

You will need to start the Apache2, memcached and mySQL services.

As supersuser:

sudo service apache2 start
sudo service memcached start
sudo service mysql start

As user www-data (which is the non-privileged user Apache2 switches to when running):

export RAILS_ENV=production
rake sunspot:solr:start
rake asset:packager:build_all
ruby script/worker start
ruby script/worker start
touch tmp/restart.txt

You have to run rake asset:packager:build_all only once, not every time you want to start the app. You should run it each time you change the ServiceCatalogue app code though, e.g. when updating from GitHub.

Once running, to restart the app as user www-data do:

touch tmp/restart.txt

This tells Apache2/Phusion Passenger to restart the application.

To stop the ServiceCatalogue, as user www-data do:

export RAILS_ENV=production
bundle exec rake sunspot:solr:stop
bundle exec ruby script/worker stop

Log files

The app's logs files are located in log/production*.log off the ServiceCatalogue application directory.

The Solr search server's log is in log/sunspot-solr-production.log off the ServiceCatalogue application directory.

Apache2's logs are in /var/log/apache2/ - Passenger is logging there too.

Maintenance jobs

It is suggested to run the following maintenance jobs (using cron):

  • Update the search query suggestions (daily)
  • Service monitoring (daily)
  • Expire old sessions >3 days (daily)
  • Update the catalogue's stats (weekly)
An example `www-data`'s crontab file:
37 4 * * * /usr/local/bin/catalogue-maintenance search
07 5 * * * /usr/local/bin/catalogue-maintenance monitoring
37 5 * * * /usr/local/bin/catalogue-maintenance sessions
07 6 * * 0 /usr/local/bin/catalogue-maintenance statistics

An example catalogue-maintenance script:

#!/bin/bash
#
# Usage:
#   catalogue-maintenance [search|monitoring|sessions|statistics]
 
APP_DIR=/var/webapps/servicecatalogue
RENV=production
 
# Stop if not run as www-data!
if [ `whoami` != "www-data" ]
then
    echo "You are not www-data! Exiting..."
    exit 1
fi
 
# Find all the ruby stuff we need.
. /etc/profile.d/rvm.sh
 
if [ -z `which bundle` ]
then
    echo "Could not find 'bundle'! Exiting..."
    exit 1
fi
 
cd ${APP_DIR}
 
case $1 in
    monitor|monitoring)
    echo "Updating URL monitoring..."
    bundle exec rake servicecatalogue:submit:update_urls_to_monitor RAILS_ENV=${RENV}
    bundle exec rake servicebiocatalogue:submit:check_url_status RAILS_ENV=${RENV}
    ;;
    search)
    echo "Updating search suggestions..."
    bundle exec rake servicecatalogue:submit:update_search_query_suggestions RAILS_ENV=${RENV}
    ;;
    session|sessions)
    echo "Cleaning up old sessions..."
    bundle exec rake servicecatalogue:remove_expired_sessions RAILS_ENV=${RENV}
    ;;
    stats|statistics)
    echo "Updating site statistics..."
    bundle exec rake servicecatalogue:submit:refresh_stats RAILS_ENV=${RENV}
    ;;
    *)
    echo -e "Please specify a task (one of):\n\tmonitoring\n\tsearch"
    exit 1
    ;;
esac
 
echo "Done."
exit 0
Add a custom footer
Add a custom sidebar
Clone this wiki locally