Running the worker on Linux

Read Running the worker before installing the worker.

Minimal worker setup

To run the worker on Ubuntu 20.04 install the required packages and clone fishtest with git:

sudo apt update
sudo apt install --yes python3 python3-pip python3-requests git build-essential && pip3 install updater
git clone https://github.com/official-stockfish/fishtest.git

Run the worker with:

python3 fishtest/worker/worker.py YOUR_USERNAME YOUR_PASSWORD --concurrency NUMBER_OF_CORES

Use single quotes wrapping your username and password if your username or password has special characters.

Worker setup script for Ubuntu

Use this script:

• To create a separate user to run the worker for security reasons
• To setup a python virtual environment
• To clone fishtest with git
• To write a bash script to start the worker in the cli
• (Optional) To configure a systemd unit to run the worker as a service; write a bash script to configure the worker concurrency; to enable the auto startup

Run the script with:

sudo bash setup_worker.sh 2>&1 | tee setup_worker.sh.log

Click to view

#!/bin/bash
# setup_worker.sh
# to setup a fishtest worker on Ubuntu 20.04, simply run: 
# sudo bash setup_worker.sh 2>&1 | tee setup_worker.sh.log

# print CPU information
cpu_model=$(grep "^model name" /proc/cpuinfo | sort | uniq | cut -d ':' -f 2)
n_cpus=$( grep "^physical id" /proc/cpuinfo | sort | uniq | wc -l)
online_cores=$(grep "^bogo" /proc/cpuinfo | wc -l)
n_siblings=$(grep "^siblings" /proc/cpuinfo | sort | uniq | cut -d ':' -f 2)
n_cpu_cores=$(grep "^cpu cores" /proc/cpuinfo | sort | uniq | cut -d ':' -f 2)
total_siblings=$((${n_cpus} * ${n_siblings}))
total_cpu_cores=$((${n_cpus} * ${n_cpu_cores}))
printf "CPU model : ${cpu_model}\n"
printf "CPU       : %3d  -  Online cores    : %3d\n" ${n_cpus} ${online_cores}
printf "Siblings  : %3d  -  Total siblings  : %3d\n" ${n_siblings} ${total_siblings}
printf "CPU cores : %3d  -  Total CPU cores : %3d\n" ${n_cpu_cores} ${total_cpu_cores}

# read the fishtest credentials and the number of cores to be contributed
echo
echo "Write your fishtest username:"
read usr_name
echo "Write your fishtest password:"
read usr_pwd
echo "Write the number of cores to be contributed to fishtest:"
echo "(max suggested 'Total CPU cores - 1')"
read n_cores

# install required packages
apt update && apt full-upgrade -y && apt autoremove -y && apt clean
apt install -y python3 python3-venv git build-essential 

# new linux account used to run the worker
worker_user='fishtest'
# create user for fishtest
useradd -m -s /bin/bash ${worker_user}

# add the bash variable for the python virtual env
sudo -i -u ${worker_user} << 'EOF'
echo export VENV=${HOME}/fishtest/worker/env >> .profile
EOF

# download fishtest
sudo -i -u ${worker_user} << EOF
git clone --single-branch --branch master https://github.com/official-stockfish/fishtest.git
cd fishtest
git config user.email "[email protected]"
git config user.name "your_name"
EOF

# fishtest worker setup and first start only to write the "fishtest.cfg" configuration file
sudo -i -u ${worker_user} << EOF
python3 -m venv \${VENV}
\${VENV}/bin/python3 -m pip install --upgrade pip setuptools wheel
\${VENV}/bin/python3 -m pip install requests

\${VENV}/bin/python3 \${HOME}/fishtest/worker/worker.py ${usr_name} ${usr_pwd} --concurrency ${n_cores} --only_config --no_validation && echo "concurrency successfully set" || echo "Restart the script using a proper concurrency value"
EOF

# write a script to start the worker from a cli with the option to change the number of cores
cat << EOF0 > worker_start.sh
#!/bin/bash
if [ \${#} -gt 1 ]; then
  echo "usage: bash \${0} [<n_cores>]"
  echo "       <n_cores>: new number of cores (optional)"
  exit 1
elif [ \${#} -eq 1 ]; then
  new_param="--concurrency \${1}"
fi

sudo -i -u ${worker_user} << EOF
\\\${VENV}/bin/python3 \\\${HOME}/fishtest/worker/worker.py \${new_param}
EOF
EOF0
chown ${SUDO_USER}:${SUDO_USER} worker_start.sh

echo
echo "Setup fishtest-worker as a service"
echo "Stop here with Windows Subsystem for Linux"
read -p "Press <Enter> to continue or <CTRL+C> to exit ..."

# install fishtest-worker as systemd service
# start/stop the worker with:
# sudo systemctl start fishtest-worker
# sudo systemctl stop fishtest-worker
# check the log with:
# sudo journalctl -u fishtest-worker.service
# the service uses the worker configuration file "fishtest.cfg"

# get the worker_user $HOME
worker_user_home=$(sudo -i -u ${worker_user} << 'EOF'
echo ${HOME}
EOF
)

cat << EOF > /etc/systemd/system/fishtest-worker.service
[Unit]
Description=Fishtest worker
After=multi-user.target

[Service]
Type=simple
StandardOutput=file:${worker_user_home}/fishtest/worker/worker.log
StandardError=inherit
ExecStart=${worker_user_home}/fishtest/worker/env/bin/python3 ${worker_user_home}/fishtest/worker/worker.py
User=${worker_user}
WorkingDirectory=${worker_user_home}/fishtest/worker

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload

# write a script to change the default number of cores
# useful for the fishtest-worker service
cat << EOF0 > worker_config.sh
#!/bin/bash
if [ \${#} -ne 1 ]; then
  echo "usage: \${0} <n_cores>"
  echo "       <n_cores>: new number of cores (mandatory)"
  exit 1
fi

sudo -i -u ${worker_user} << EOF
\\\${VENV}/bin/python3 \\\${HOME}/fishtest/worker/worker.py --concurrency \${1} --only_config --no_validation && echo "concurrency successfully set" || echo "Restart the script using a proper concurrency value"
EOF
EOF0
chown ${SUDO_USER}:${SUDO_USER} worker_config.sh

echo
echo "Start fishtest-worker service"
read -p "Press <Enter> to continue or <CTRL+C> to exit ..."
systemctl start fishtest-worker.service

echo
echo "Enable fishtest-worker service auto start"
read -p "Press <Enter> to continue or <CTRL+C> to exit ..."
systemctl enable fishtest-worker.service

How to choose the number of cores

Option --concurrency refers to the number of cores dedicated to the worker. The safest max setting is to use the number of physical cores, leaving one core for the OS. If you do not know the number of physical cores on your system, execute:

lscpu | awk '/^Core\(s\) per socket:/ {cores=$NF}; /^Socket\(s\):/ {sockets=$NF}; END{print cores*sockets}'

if lscpu is not installed a portable way to report the total (physical + hyperthreaded) core is

getconf _NPROCESSORS_ONLN

Running the worker on RHEL/CentOS

If the default version of GCC should not support C++11, you can install the developer toolset, which adds a newer version of GCC. It installs side-by-side seamlessly, simply follow the instruction for your OS:

https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/

Then, create a file called launch_worker.sh, with these commands:

#!/bin/bash
# usage: fishtest.sh  [<n_cores> <username> <password>]
# <n_cores>: number of cores to be used in fishtest. Suggested max value = n. physical cores-1
# <username>: username on fishtest (to be enclosed in quote if contains special character)
# <password>: password on fishtest (to be enclosed in quote if contains special character)
# The three parameters are mandatory only for the first execution

if [ $# -gt 0 ]; then
  sudo -i -u fishtest << EOF
source scl_source enable devtoolset-7
python3 fishtest-master/worker/worker.py --concurrency $1 "$2" "$3"
EOF
else
  sudo -i -u fishtest << EOF
source scl_source enable devtoolset-7
python3 fishtest-master/worker/worker.py
EOF
fi

And use that to launch the worker with bash fishtest.sh <n_cores> <username> <password>

Running the worker on Alpine Linux

Type the following commands to create a dedicated user, configure fishtest and install all the required machinery.

apk add g++ git make python3
adduser -S fishtest
su fishtest -s /bin/sh
cd
git clone https://github.com/official-stockfish/fishtest.git
python3 fishtest/worker/worker.py
^c

Then create a /etc/init.d/fishtest file with the following content:

#!/sbin/openrc-run

name="fishtest"
command="python3"
command_args="/home/fishtest/fishtest/worker/worker.py"
command_background=true
command_user="fishtest"
pidfile="/run/${RC_SVCNAME}.pid"

depend() {
        need net
}

Mark it as executable, and rc-update add fishtest to enable it at startup