-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- add email to author frontmatter\n- add ascii logo text file\n- version 0.1.1 declared
- Loading branch information
Christian Ramirez
committed
Dec 26, 2024
1 parent
23a20f5
commit c3b775c
Showing
5 changed files
with
47 additions
and
27 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
.-') _ ('-. .-. .-') | ||
( OO) ) ( OO ).-.\ ( OO ) | ||
,(_)----. ,--. / . --. / ;-----.\ | ||
| | | |.-') | \-. \ | .-. | | ||
'--. / | | OO ).-'-' | | | '-' /_) | ||
(_/ / | |`-' | \| |_.' | | .-. `. | ||
/ /___(| '---.' | .-. | | | \ | | ||
| || | | | | | | '--' / | ||
`--------'`------' `--' `--' `------' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,21 +1,23 @@ | ||
--- | ||
title: Computing Resources | ||
subtitle: Technical guides for commonly used software tools | ||
author: Christian S. Ramirez | ||
authors: | ||
- name: Christian S. Ramirez | ||
email: [email protected] | ||
--- | ||
# Secure Jupyter | ||
## Why? | ||
HTTPS encrypts data between your browser and JupyterLab to prevent eavesdropping/tampering, while password authentication ensures only authorized users can access your notebooks and execute code on your system. | ||
## Setting up your HTTPS and `jupyter_server_config.py` | ||
### Make jupyter password and `certfile` directory | ||
``` | ||
```bash | ||
mkdir -p ~/.jupyter/certfiles | ||
jupyter lab --generate-config | ||
jupyter lab password | ||
``` | ||
### Create your private (.key) and public key (.pem) | ||
Make sure to replace `HOSTNAME` with the machine you are on (i.e. z014). | ||
``` | ||
```bash | ||
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \ | ||
-keyout ~/.jupyter/certfiles/jupyter.key \ | ||
-out ~/.jupyter/certfiles/jupyter.pem \ | ||
|
@@ -24,7 +26,7 @@ chmod 600 ~/.jupyter/certfiles/jupyter.key | |
``` | ||
### Add config options to your `jupyter_server_conifg.py` | ||
Make sure to replace `HOSTNAME` with the machine you are on (i.e. z014) and `USERNAME` with your username. | ||
``` | ||
```python | ||
c.ServerApp.certfile = '/Users/USERNAME/.jupyter/certfiles/jupyter.pem' | ||
c.ServerApp.keyfile = '/Users/USERNAME/.jupyter/certfiles/jupyter.key' | ||
c.ServerApp.ip = 'HOSTNAME' | ||
|
@@ -37,11 +39,11 @@ c.ServerApp.notebook_dir = '/data/GROUP/USERNAME' | |
Now start jupyter lab using your preferred method. [See Getting Started](getting_started.md) if you are unsure. Navigate to `https://127.0.0.1:8888/lab`. If this is you first time doing this, your browser may return a self-signed certificate warning. Ignore this error, and connect to the url. Now you can login using the password you made at the beginning. | ||
|
||
Defining our config options in `jupyter_server_config.py` allows you be less declarative when starting jupyter in the terminal. For example, when using singularity we can start jupyter lab with: | ||
``` | ||
```bash | ||
singularity exec --writable -B /data,/zata /zata/zippy/$(whoami)/bin/bioinformatics jupyter lab | ||
``` | ||
As opposed to: | ||
``` | ||
```bash | ||
singularity exec --writable -B /data,/zata /zata/zippy/$(whoami)/bin/bioinformatics jupyter lab --port=8888 --ip=HOSTNAME --no-browser --notebook-dir=/data/GROUP/$(whoami) | ||
``` | ||
# Screen | ||
|
@@ -123,7 +125,7 @@ Micromamba is a standalone version of Mamba which is an alternative to Conda. Re | |
This wrapper script simplifies the usage of `python -m venv` for creating native python virtual environments. This is especially handy if you are using Homebrew on MacOS since, by default, Homebrew disallows the usage of pip in the global environment. | ||
### Wrapper Script | ||
Add the following to your `.zshrc` or `.bashrc`: | ||
``` | ||
```bash | ||
# Usage | ||
# $ mkvenv myvirtualenv # creates venv with default python3 | ||
# $ mkvenv myvirtualenv 3.8 # creates venv with python3.8 | ||
|
@@ -188,7 +190,7 @@ Make sure to run `source ~/.bashrc` or `source ~/.zshrc` after adding the script | |
Please refer to the [getting started page](https://hpc.umassmed.edu/doc/index.php?title=Getting_started) on the HPC wiki at `hpc.umassmed.edu`. Make sure that you are connected to the UMMS network (on-premises) or [[VPN|Getting Started#computing-access##vpn-setup]]. | ||
## Useful bash aliases | ||
Add the following to you bashrc. | ||
``` | ||
```bash | ||
alias gpu='bsub -q gpu -gpu "num=1:gmodel=TeslaV100_SXM2_32GB" -W 1440 -n 10 -R "rusage[mem=8000]" -R "span[hosts=1]" -Is bash' | ||
alias cpu='bsub -q interactive -n 10 -R "rusage[mem=8000]" -R "span[hosts=1]" -Is bash' | ||
``` | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,8 +2,9 @@ | |
title: Getting Started | ||
subtitle: Onboarding tutorials for new lab members | ||
authors: | ||
- name: Christian S. Ramirez | ||
- name: Greg Andrews | ||
- name: Christian S. Ramirez | ||
email: [email protected] | ||
--- | ||
# Computing Access | ||
|
||
|
@@ -21,7 +22,7 @@ If on windows, you will have to install [WSL](https://learn.microsoft.com/en-us/ | |
[WSL basic commands](https://learn.microsoft.com/en-us/windows/wsl/basic-commands) | ||
|
||
From your powershell start your linux VM with: | ||
``` | ||
```bash | ||
wsl.exe -d ubuntu | ||
``` | ||
## Update ssh config file | ||
|
@@ -56,7 +57,9 @@ Host z010 z011 z012 z013 z014 | |
User USERNAME | ||
``` | ||
## Logging in | ||
`ssh [email protected]` | ||
```bash | ||
ssh [email protected] | ||
``` | ||
|
||
If it's your first time, you will be prompted to scan the QR code with any two-factor authentication app. You should also be prompted to change your password. | ||
```{important} | ||
|
@@ -67,7 +70,7 @@ I personally recommend using Microsoft Authenticator, since this is the 2FA app | |
From this point forward, every time you login, you will provide your password + two-factor code with no spaces. | ||
``` | ||
From bastion, you can then `ssh` into any of the ZLab servers: | ||
``` | ||
```bash | ||
ssh HOSTNAME | ||
``` | ||
Replace `HOSTNAME` with one of the hostnames defined in your ssh config file (i.e. z011, z012 ... z014). | ||
|
@@ -76,7 +79,7 @@ Replace `HOSTNAME` with one of the hostnames defined in your ssh config file (i. | |
## Install conda | ||
Conda is a great way to quickly install software and create separate environments for projects requiring different, and potentially conflicting, pieces of software. The conda (Miniforge3) installation instructions have been adapted from default installation instructions so it is available across all our machines. | ||
|
||
``` | ||
```bash | ||
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" | ||
bash Miniforge3-$(uname)-$(uname -m).sh -p /zata/zippy/$(whoami)/Miniforge3 | ||
source ~/.bashrc | ||
|
@@ -86,7 +89,7 @@ source ~/.bashrc | |
While this is optional, you may want to use the `mamba` dependency resolver which tends to be faster than native `conda`. You can still use conda if you are prefer, but in the following examples will use `mamba`. You may need to run `mamba init` if this is your first time using `mamba` and then update your bashrc file with `source ~/.bashrc`. | ||
|
||
Create your first conda environment: | ||
``` | ||
```bash | ||
mamba create -n myenv jupyterlab numpy pandas matplotlib bedtools | ||
``` | ||
## Singularity | ||
|
@@ -97,12 +100,12 @@ Containers are useful in bioinformatics because they ensure reproducibility by p | |
The remote docker image `clarity001/bioinformatics` contains a full suite of bioinformatics software that you will most likely need. | ||
|
||
Create the destination folder and build the singularity sandbox container: | ||
``` | ||
```bash | ||
mkdir -p /zata/zippy/$(whoami)/bin/ | ||
singularity build --sandbox /zata/zippy/$(whoami)/bin/bioinformatics docker://clarity001/bioinformatics | ||
``` | ||
To start an interactive shell in the *writable* container (optional): | ||
``` | ||
```bash | ||
singularity shell --writable -B /data,/zata /zata/zippy/$(whoami)/bin/bioinformatics | ||
``` | ||
# JupyterLab | ||
|
@@ -113,21 +116,21 @@ In the following examples: | |
|
||
## Start a JupyterLab server using singularity | ||
Using the sandbox container you just built, start the JupyterLab server with the following command: | ||
``` | ||
```bash | ||
singularity exec --writable -B /data,/zata /zata/zippy/$(whoami)/bin/bioinformatics jupyter lab --port=8888 --ip=HOSTNAME --no-browser --notebook-dir=/data/GROUP/$(whoami) | ||
``` | ||
## Start a JupyterLab server using docker | ||
First see [rootless docker setup](rootless_docker_setup) | ||
``` | ||
```bash | ||
docker container run -it --rm -p 8888:8888 --mount type=bind,src=/data,target=/data --mount type=bind,src=/zata,target=/zata clarity001/bioinformatics jupyter-lab --port=8888 --ip=* --no-browser --allow-root --notebook-dir=/data/GROUP/$(whoami)/ | ||
``` | ||
## Start a JupyterLab server in a conda environment | ||
Activate your environment from the previous example: | ||
``` | ||
```bash | ||
conda activate myenv | ||
``` | ||
Start your jupyterlab server: | ||
``` | ||
```bash | ||
jupyter-lab --port=8888 --ip=HOSTNAME --no-browser | ||
``` | ||
## Accessing your JupyterLab server | ||
|
@@ -148,15 +151,15 @@ You can then open your notebook server in your favorite web browser by navigatin | |
## Common issues with JupyterLab | ||
### Unable to delete files | ||
This is an issue that has come up before. To fix, you first need to find you jupyter config file. It will most likely be located at `/home/USERNAME/.jupyter`. First, change into that directory: | ||
``` | ||
```bash | ||
cd /home/$(whoami)/.jupyter | ||
``` | ||
If you cannot find `jupyter_server_config.py` in that directory, run: | ||
``` | ||
```bash | ||
jupyter server --generate-config | ||
``` | ||
Edit `jupyter_server_config.py` with your preferred text editor: | ||
``` | ||
```bash | ||
vim jupyter_server_config.py | ||
``` | ||
Look for the config option `c.FileContentsManager.delete_to_trash`. If it is set to `True`, change it to `False`. Now you should be able to delete files in JupyterLab | ||
|
@@ -165,7 +168,7 @@ Look for the config option `c.FileContentsManager.delete_to_trash`. If it is set | |
## Why use rootless Docker? | ||
Rootless Docker enables users to run containers without administrator privileges, making it ideal for shared HPC environments where granting root access to multiple users would pose security risks. | ||
## Setup | ||
``` | ||
```bash | ||
mkdir -p ~/.config/docker/ | ||
echo '{"data-root":"/rootless/docker/'$(whoami)'/docker"}' > ~/.config/docker/daemon.json | ||
dockerd-rootless-setuptool.sh install | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,9 @@ | ||
--- | ||
title: Learning Resources | ||
subtitle: Curated collection of bioinformatics learning materials | ||
author: Christian S. Ramirez | ||
authors: | ||
- name: Christian S. Ramirez | ||
email: [email protected] | ||
--- | ||
# Foundation Skills | ||
## Programming Essentials (Python) | ||
|
@@ -110,4 +112,4 @@ project_name/ | |
├── scripts/ # Workflow-specific scripts | ||
└── logs/ # Log files | ||
``` | ||
This is not an absolute standard to follow by any means. The above directory tree is meant more as a guideline for how to approach project organization. | ||
This is not an absolute standard to follow. The above directory tree is meant more as a guideline for how to approach project organization. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,6 @@ | ||
# Welcome | ||
# Welcome to the Zlab! | ||
```{literalinclude} ascii_logo.txt | ||
:start-line: 0 | ||
:lineno-match: | ||
``` | ||
This wiki is intended for onboarding new students into the lab, by providing tutorials and guides about common software used in our lab and research. |