Skip to content

Commit

Permalink
Add resources for ope-notebook-culler cronjob
Browse files Browse the repository at this point in the history
File formatting to make git happy

Formatting
  • Loading branch information
dystewart committed Jan 23, 2024
1 parent d2bbdac commit 8a34990
Show file tree
Hide file tree
Showing 6 changed files with 129 additions and 1 deletion.
38 changes: 38 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,44 @@
This repository is a collection of useful scripts and tools for TAs and professors to manage students workloads.

## get_url.py
=======
## Cronjobs

### ope-notebook-culler

This cronjob runs once every 24 hours at 7am, removing all notebooks & pvcs from the rhods-notebooks namespace. To add resources to the rhods-notebooks namespace:

1. Ensure you are logged in to your OpenShift account via the CLI and you have access to rhods-notebooks namespace.
2. Switch to rhods-notebooks namespace:
```
oc project rhods-notebooks
```

3. From cronjobs/ope-notebook-culler/ directory run:
```
oc apply -k .
```

This will deploy all the necessary resources for the cronjob to run on the specified schedule.

Alternatively, to run the script immediately:

1. Ensure you followed the steps above
2. Verify the cronjob ope-notebook-culler exists
```
oc get cronjob ope-notebook-culler
```

3. Run:
```
kubectl create -n rhods-notebooks job --from=cronjob/ope-notebook-culler ope-notebook-culler
```

This will trigger the cronjob to spawn a job manually.

## Scripts

### get_url.py

This script is used to retrieve the URL for a particular notebook associated with one student. To execute this script:

Expand Down
68 changes: 68 additions & 0 deletions cronjobs/notebok-culler/cronjob.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
kind: CronJob
apiVersion: batch/v1
metadata:
name: nb-culler
labels:
component.opendatahub.io/name: nb-culler
opendatahub.io/component: 'true'
opendatahub.io/modified: 'false'
spec:
schedule: '0 7 * * *'
startingDeadlineSeconds: 200
concurrencyPolicy: Replace
suspend: false
jobTemplate:
metadata:
labels:
component.opendatahub.io/name: nb-culler
opendatahub.io/component: 'true'
spec:
template:
metadata:
labels:
component.opendatahub.io/name: nb-culler
opendatahub.io/component: 'true'
parent: nb-culler
spec:
restartPolicy: Never
serviceAccountName: ope-notebook-culler
schedulerName: default-scheduler
terminationGracePeriodSeconds: 30
securityContext: {}
containers:
- name: oc-cli
image: >-
registry.redhat.io/openshift4/ose-cli@sha256:25fef269ac6e7491cb8340119a9b473acbeb53bc6970ad029fdaae59c3d0ca61
command: ["/bin/bash", "-c", "--"]
args:
- |
#threshold to stop running notebooks. Currently set to 24 hours
cutoff_time=86400
current_time=$(date +%s)
notebooks=$(oc get notebooks -n ope-rhods-testing-1fef2f -o jsonpath='{range .items[?(@.status.containerState.running)]}{.metadata.name}{" "}{.metadata.namespace}{" "}{.status.containerState.running.startedAt}{"\n"}{end}')
if [ -z "$notebooks" ]; then
echo "No running notebooks found"
exit 0
fi
# Loop through each notebook
while read -r nb ns ts; do
timestamp=$(date -d $ts +%s)
difference=$((current_time - timestamp))
if [ $difference -gt $cutoff_time ]; then
echo "$nb is more than 24 hours old, stopping the notebook"
oc patch notebook $nb -n $ns --type merge -p '{"metadata":{"annotations":{"kubeflow-resource-stopped":"'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'"}}}'
fi
done <<< "$notebooks"
resources:
limits:
memory: 800Mi
requests:
memory: 400Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
serviceAccount: ope-notebook-culler
dnsPolicy: ClusterFirst
successfulJobsHistoryLimit: 7
failedJobsHistoryLimit: 7

7 changes: 7 additions & 0 deletions cronjobs/notebok-culler/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- cronjob.yaml
- rolebinding.yaml
- serviceaccount.yaml
namespace: rhods-notebooks
11 changes: 11 additions & 0 deletions cronjobs/notebok-culler/rolebinding.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nb-culler
subjects:
- kind: ServiceAccount
name: nb-culler
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: edit
4 changes: 4 additions & 0 deletions cronjobs/notebok-culler/serviceaccount.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: ope-notebook-culler
2 changes: 1 addition & 1 deletion docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ RUN dnf -y update
RUN dnf -y install jq
WORKDIR /shell-scripts
COPY src/* /shell-scripts
CMD ["/bin/sh"]
CMD ["/bin/sh"]

0 comments on commit 8a34990

Please sign in to comment.