Note
These instructions are for installing the Runestone servers using Docker to automate much of the work involved in setting up the environment needed to run a server. If you would prefer to install all the components directly on a server yourself, see the Manual Installation instructions.
Using Docker, we can bring up the server without needing to install dependencies directly on the host. Instead, the software required to run the server will be installed in four separate Docker containers (the Runestone application, redis cache, postgres DB, and Jobe code runner).
Contents
To build a Docker application with the server and all its dependencies:
Note
You will need to enter the root user password several times during the following steps. Look for the Password:
prompt, then enter your password. Note that no characters will be echoed when you type your password -- this is a normal security precaution built into Unix.
Open a Terminal. In the Terminal:
Edit
/etc/paths
; add/opt/homebrew/bin
as the first line in that file.Execute
brew install python docker md5sha1sum
.Run the newly-installed Docker Desktop application.
Note
Expect to wait a few minutes the first time you start the Docker Desktop. Don't proceed until the Docker Desktop's initialization is complete.
If running on Windows, either:
Install Ubuntu on WSL2. Next, install then run Docker Desktop; see the note above on the initialization process.
OR
Install Ubuntu on VirtualBox or on some other virtualization software. You do not need to install the Docker Desktop.
Note
VirtualBox
To run in VirtualBox, configure the network settings: use NAT, select Advanced, Port forwarding, add:
Name Protocol Host IP Host Port Guest IP Guest Port SSH TCP 22 22 HTTP TCP 80 80 HTTPS TCP 443 443
Next, run the Ubuntu
program to open a Linux terminal (if using WSL2) or boot up Linux and open a terminal (for VirtualBox or other virtualization software).
Finally, follow the directions for Linux below.
If running on Linux, you must use Ubuntu 20.04 (although any version of Ubuntu 18.0+ should work). Installation on older versions of Ubuntu or other Linux distributions may require adjustments. I am a fan of digital ocean droplets and I would recommend starting out with a Docker droplet from their Marketplace. This will save you a few steps. You can of course use AWS or Linode or any other provider, but I know DO the best. Sizing Virtual Machines: You will want to make sure that you have at least 1 GB of RAM as parts of the install process do use quite a bit of memory. If you are going to use this for several classes I would recommend 2 GB. If you are teaching courses that will involve compiling Java, C, or C++ then I would recommend a configuration that includes 4 cpus as well as 4 GB of RAM.
Install curl
by opening a terminal then typing:
sudo apt install -y curl
Warning
If this fails, run sudo apt update
then retry the command above.
Note
On OS X, avoid placing your files in the Documents folder, since security features introduced in OS X 12.4 require you to give Docker additional permissions.
Download the bootstrap script. To do this, open a terminal in Ubuntu or OS X then type:
curl -fLO https://raw.githubusercontent.com/RunestoneInteractive/RunestoneServer/master/docker/docker_tools.py
Warning
On Windows using Ubuntu under WSL2: if you see the error message "Docker Desktop not detected..." when running either command below, but you are running the Docker Desktop, then click the gear (settings) icon in Docker Desktop, select Resources then WSL Integration, and make sure the switch next to Ubuntu is turned on.
The next step, which installs required dependencies for the remainder of the process, depends on the two mutually exclusive use cases below. Remember which use case you select (either production or development); many of the following steps vary based on your use case. Note if you are just kicking the tires on a Runestone Server you should use production not test or development. The test mode is for our testing of the server by running 100's of unit tests.
If your use case is running the server, execute:
python3 docker_tools.py init
OR
If, in addition to running the server, your use case is to change the way Runestone works or change/add to the way interactive exercises behave, then:
- Fork the RunestoneServer, RunestoneComponents, and BookServer repositories. If you've already forked these repositories, fetch the latest updates from these upstream repositories.
- In the terminal, run:
python3 docker_tools.py init --clone-rs <your Github userid>
This process may take a few minutes to complete. When it does:
- Reboot your computer to update your group membership.
- Run the Docker Desktop if using WSL2 on Windows or using OS X.
- Open a terminal.
In the terminal, type:
cd rsdocker/RunestoneServer
Note
All future commands should be run in the rsdocker/RunestoneServer
directory unless instructions specify otherwise.
The next command depends on the use case you chose in the previous step.
For the production use case, execute:
docker-tools build
OR
For the development use case, execute:
docker-tools build --single-dev --clone-all <your Github userid>
Note
The build will take a long time (10-20 minutes in many cases). In particular, the last line (rm -rf $RUNESTONE_PATH
) may seem to hang, but simply takes a long time to complete.
Most basic configuration can be done via two files you will need to create. These files are read every time the server is restarted - to see the effects of any changes you will need to stop the containers and restart them.
For the development use case, you do not need to modify any of the default environment variables.
OR
For the production use case, you will need to modify these variables. To do so, edit the .env
file, which Docker will read automatically as it loads containers. A sample .env
file is provided as ./.env
(copied from docker/.env.prototype <.env.prototype> on the first build). See comments in the file for details.
For the development use case, you do not need to modify any of the default Python settings.
OR
For the production use case, you will need to modify these settings to obtain an HTTPS certificate, send the lost password e-mails, etc. These options will be in the file models/1.py
(which is automatically created on the first build).
Warning
You will NOT want to check either .env
or models/1.py
into source control, since these contain passwords. The .gitignore
file is set to ignore both of them.
Once your environment is ready to go, you can use docker compose
to bring the containers up. This command will create four containers to run different parts of the application stack (the Runestone server, redis cache, postgres DB, jobe code testing environment).
For the production use case, execute:
docker compose up -dThis runs the container in the background (detached mode). Use
docker compose logs --follow
to view logging data as the container starts up and runs; open another terminal to execute instructions from the following steps.
OR
For the development use case, execute:
docker compose upThis displays logging data from the container in the terminal. To stop the container, press ctrl-c. Open another terminal to execute the following commands.
The first time you run the command will take a lot longer as it downloads containers then installs software into the various containers. You may ignore the red message jobe error
that appears during this process. After it is complete, you can go to http://localhost/ to see the application (if you configured a hostname, substitute it for localhost). If everything so far is set up correctly, you should see a welcome/login page. Continue in the instructions to add book(s), course(s) and a user account.
The rsmanage
command will run many useful commands inside the container for you. With rsmanage
you can:
- Add a course -
rsmanage addcourse
- Add a user -
rsmanage adduser
- Get information about a course
rsmanage courseinfo
- Build a book -
rsmanage build bookname
- Get a database shell in the current database -
rsmanage db
...and many other things. Just type rsmanage
for a list of things it can do. For a list of options just type rsmanage
and the subcommand you want followed by --help
; for example, rsmanage build --help
.
If at this stage you type rsmanage --help and you get a command not found it is likely that pip installed the command in a different location than the one in your PATH. It could be in ~/.local/bin
or /usr/local/bin
or /opt/local/bin
. You can add that location to your PATH or you can just type the full path to the command. For example, ~/.local/bin/rsmanage
No books are installed by default; you must add books using the following process. To add a book, you need to add its source code to the RunestoneServer/books/
directory. Typically, that means cloning its source code. For example, to add
thinkcspy:
rsmanage build --clone https://github.com/RunestoneInteractive/thinkcspy.git thinkcspy
After cloning a book, you may need to add it to the database. Most of the standard books are already there, but you can use rsmanage addcourse
to add it if needed.
Note
PreTeXt authors, see Publishing to Runestone Academy. After that, you can build a pretext book just like building a Runestone book rsmanage build --ptx coursename
The following information applies only authoring books using the Runestone.
Warning
It is important that the folder name for the book matches the project_name
set in its pavement.py
.
This is not always automatically the case. For example, the ThinkCPP
repository will normally be cloned into ThinkCPP but it has the project_name
set to thinkcpp
.
If there is a mismatch, you will want to rename the folder you cloned the code into so that it
matches the project_name
.
Note
Most Runestone books set master_url
to get_master_url()
in their pavement.py
file. However, if the book
you are adding does not, it is critical that the master_url
variable in that file is set correctly.
If you are running docker and doing your development on the same machine then http://localhost
will work.
If you are running docker on a remote host then make sure to set it to the name of the remote host.
To add a course based on a book, run the rsmanage addcourse
script. If you run it just like
that it will prompt you for all of the necessary details. Probably the most important thing
to point out is that if this is a new book the first time you add it you want to make sure that the basecourse and the course-name are the same. If you are creating your own course but want it
based on an existing book then make sure to use the correct base course name.
rsmanage addcourse
It will ask for:
Course Name: The short name to identify this course/section (do NOT include any spaces). e.g. yourname-cs1-fall2021
Base Course: The name of the book to use. This MUST match the project_name defined
in pavement.py of the book. e.g. thinkcspy
Your institution: The human readable name of your institution. e.g. Some State U
Then you will be asked whether to allow users to access the course without logging in (defaults to yes) and whether to allow pair programming (default is no).
You do not have to restart the server to make use of the course.
Note
Some of the default books already have "default" courses with the same name as the book. If you try to create
a course with a name like thinkcspy
you will be told that the course name is the same as the book.
To add an initial instructor account to the course you have created, you can either create a new user or add an existing user as an instructor to the course. You may use the web interface or the terminal -- the command-line interface, or CLI -- to do this.
Web interface: browse to https://<your domain name>/runestone/designer/index
(production use case) or https://localhost/runestone/designer/index
(development use case). Use this web page to create a new course.
CLI: To add a new user, use the rsmanage adduser
subcommand; it asks for what class to add the user to and whether or not
they should be made an instructor.
rsmanage adduser
Or, if you already have an account that you want to add as an instructor to the new course, you can use the
rsmanage
command to execute addinstructor which will prompt you for a username and course name:
rsmanage addinstructor
Neither of these will require restarting the server.
Once you have logged in as an instructor, you can bulk add students through the web interface. After logging in to your running server as an instructor, browse to the Instructor's Page, then click on the Manage Students tab.
The containerized application is configured to automatically start as soon as Docker / the Docker Desktop is started. Therefore, on OS X or Windows (when using WSL2): after a reboot or after manually shutting down the Docker Desktop, remember to start the Docker Desktop application.
Before using docker-tools
or rsmanage
:
- Run the Docker Desktop if using WSL2 on Windows or using OS X.
- Open a terminal.
- At the terminal, execute
cd rsdocker/RunestoneServer
.
To re-build an image:
# See the possibilities
docker-tools build --help
# Actually run the build (add options as desired)
docker-tools build
To force a rebuild, make sure the containers are stopped by executing docker compose stop
, then rerun the build
command. The build process caches results from previous builds and should complete much more rapidly. However, the
cache can cause issues if you modify a file that the system is checking for changes. If you need to force a
complete rebuild, use:
docker-tools build -- --no-cache
You can shell into the container to look around, or otherwise test. When you enter,
you'll be in the web2py folder, where runestone/
is an application under applications/
. From the RunestoneServer/
directory do:
docker-tools shell
Remember that the folder under web2py/applications/runestone
is bound to your host,
so do not edit files from inside the container otherwise they will have a change
in permissions on the host.
Data is stored on a Docker containerized application in two distinct places:
- Volumes, such as the Runestone Server path (
$RUNESTONE_PATH
), the BookServer path, and the Runestone Components path. - Layers in a docker image -- which is everything not stored in the volumes listed above.
Anything written to layers after the Docker build process will be lost. For example, if you shell into the container then apt install
a package, these changes will be lost if the container is stopped, its configuration changed, etc. This is the nature of Docker. See the docs for more information.
To install a VNC client on Linux, execute sudo apt install gvncviewer
. Next, run gvncviewer localhost:0 &
. This allows you to open a terminal in the container, see Chrome as Selenium tests run, etc.
Execute sudo apt install openssh-server
to install a SSH server. This allows easy access from VSCode, as well as usual SSH access.
If you make changes to the Runestone Components, you must rebuild the bundle of JavaScript bundle produced by webpack using npm run build
, then re-build the book (or page of a book) which uses the component you're editing via a runestone build
or pretext build
. The unit tests do this automatically; for development, it's easiest to make changes to the test then re-run the test to guarantee the correct builds are done.
If you make changes to the BookServer, you'll need to stop then restart the BookServer. To do this, use docker-tools start-servers
/ docker-tools stop-servers
.
If you make changes to the Runestone server, most changes will be immediately applied. However, changes in the modules
folder require a stop / start sequence to apply these changes.
You can run the unit tests in the container using the docker-tools test
command.
If you want to test the script, the easiest thing
to do is add a command to the docker compose
to disable it, and then run commands
interactively by shelling into the container.
Bring up the containers and then shell inside. Once inside, you can then issue commands
to test the entry point script - since the other containers were started
with docker compose
everything in them is ready to go.
File permissions can seem a little strange when you start this container on Linux. Primarily because both
nginx and Gunicorn run as the www-data
user. So you will suddenly find your files under RunestoneServer
owned by www-data
. The container's entry point script updates permissions to allow both you and the
container enough privileges to do your work.
Note
PreTeXt authors, see Publishing to Runestone Academy. The following information applies only authoring books using the Runestone.
If you are writing your own book you will want to get that book set up properly in the Runestone system. You need to do the following:
- Run the command
rsmanage addcourse
. Use the project name you configured inpavement.py
as the name of BOTH the course and the basecourse when it asks. - Now that your course is registered, rebuild it using the command
rsmanage build <book_name>
command. If this is a PreTeXt book then build with the commandrsmanage build --ptx <book_name>
where thebook_name
should match the document-id specified in the docinfo section of the pretext book. Often found inbookinfo.ptx
but sometimes as a peer of<book>
in themain.ptx
file for the book.
If you modify the dependencies of a non-Poetry project (such as the Runestone Components or rsmanage), then poetry update
will not see these updates. To force an update, manually delete the *.egg-info
directory before running poetry update
. Note you must be in shelled in to the running docker container to run poetry update
.