How to Run Decision Engine

Decision engine uses postgresql database back-end.

Install postgresql

Default postgresql installed on RH7 is 9.2 which is outdated. Suggest to remove it and install 11 instead :

1. Remove old postgresql

yum erase -y postgresql*

2. Install postgresql 11

yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
yum install -y postgresql11 postgresql11-server 
# optional, also: postgresql11-devel 

3. Enable postgresql

systemctl enable postgresql-11

4. Init db

 /usr/pgsql-11/bin/postgresql-11-setup initdb

5. edit /var/lib/pgsql/11/data/pg_hba.conf like so:

[root@fermicloud371 ~]# diff  /var/lib/pgsql/11/data/pg_hba.conf~ /var/lib/pgsql/11/data/pg_hba.conf 
80c80
< local   all             all                                     peer
---
> local   all             all                                     trust
82c82
< host    all             all             127.0.0.1/32            ident
---
> host    all             all             127.0.0.1/32            trust
84c84
< host    all             all             ::1/128                 ident
---
> host    all             all             ::1/128                 trust

(that is setting authentication method is trust)

6. start database

systemctl start postgresql-11

7. create decisionengine

createdb -U postgres decisionengine

8. create database schema

psql -U postgres decisionengine -f <full path to>/decisionengine/src/decisionengine/framework/dataspace/datasources/postgresql.sql

9. configure database connection. Check

/etc/decisionengine/decision_engine.conf

for above steps it would look like this:

cat decision_engine.conf
{
    'logger' : {'log_file': '/var/log/decisionengine/decision_engine_log',
                'max_file_size': 200*1000000,
                'max_backup_count': 6,
            },

    'dataspace': {
        'reaper_start_delay_seconds': 1818,
        'retention_interval_in_days': 370,
        'datasource' : {
           'module' : 'decisionengine.framework.dataspace.datasources.postgresql',
	   'name' : 'Postgresql',
	   'config' : {
	      'user' : 'postgres',
	      'blocking' : True,
	      'host' : 'localhost',
	      'database' : 'decisionengine',
	      'maxconnections' : 100,
	      'maxcached' : 10,
            },
	   },
    },
}

To use the database you have to add it to the environment:

export PG_VERSION=11
export PATH="/usr/pgsql-${PG_VERSION}/bin:~/.local/bin:$PATH"

Install decision engine

These instructions are to be executed by root for a system installation.

1. Make sure that required yum repos and some required packages (python3, gcc, ...) are there and up to date:

yum install -y http://ftp.scientificlinux.org/linux/scientific/7x/repos/x86_64/yum-conf-softwarecollections-2.0-1.el7.noarch.rpm
yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# gcc, swig and make needed for dependencies (jsonnet)
yum -y install python3 python3-pip python3-setuptools python3-wheel \
    gcc gcc-c++ make \
    python3-devel swig openssl-devel git rpm-build
python3 -m pip install --upgrade --prefix=/usr pip
python3 -m pip install --upgrade --prefix=/usr setuptools wheel setuptools-scm[toml]

2. Setup the decision engine yum repositories

wget -O /etc/yum.repos.d/ssi-hepcloud.repo http://ssi-rpm.fnal.gov/hep/ssi-hepcloud.repo
wget -O /etc/yum.repos.d/ssi-hepcloud-dev.repo http://ssi-rpm.fnal.gov/hep/ssi-hepcloud-dev.repo

3. Install the decision engine (add --enablerepo=ssi-hepcloud-dev for the latest development version)

yum install decisionengine
yum install decisionengine-standard-library

4. Install the required Python packages (these are taken from setup.py)

# from decisionengine
python3 -m pip install jsonnet tabulate toposort
python3 -m pip install wheel DBUtils sqlalchemy
python3 -m pip install "psycopg2-binary >= 2.8.6; platform_python_implementation == 'CPython'
python3 -m pip install "psycopg2cffi >= 2.9.0; platform_python_implementation == 'PyPy'"
# from decisionengine_modules
python3 -m pip install packaging
python3 -m pip install boto boto3 google_auth google-api-python-client
python3 -m pip install gcs-oauth2-boto-plugin
python3 -m pip install htcondor
# This is not in pypi
python3 -m pip install https://test-files.pythonhosted.org/packages/f4/a5/17a14b4ef85bc412a0ddb771771de3f562430328b0d83da6091a4131bb26/bill_calculator_hep_mapsacosta-0.0.10-py3-none-any.whl

decisionengine --help should print the help message

Start decision engine

systemctl start decision-engine