Skip to content

Commit

Permalink
Merge pull request #24 from Anaconda-Platform/lambda
Browse files Browse the repository at this point in the history
Enable AWS Lambda
  • Loading branch information
AlbertDeFusco authored Jan 10, 2022
2 parents e21fa71 + e154dae commit db772b4
Show file tree
Hide file tree
Showing 19 changed files with 828 additions and 12 deletions.
4 changes: 4 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,7 @@ $(TEST_IMAGES):
IMAGE_NAME=$(IMAGE_PREFIX)-$(patsubst test-%,%,$@)-candidate test/run-multi-env default py3
IMAGE_NAME=$(IMAGE_PREFIX)-$(patsubst test-%,%,$@)-candidate test/run-multi-env py2 py2
IMAGE_NAME=$(IMAGE_PREFIX)-$(patsubst test-%,%,$@)-candidate test/run-multi-env py3 py3
IMAGE_NAME=$(IMAGE_PREFIX)-$(patsubst test-%,%,$@)-candidate test/run-env-vars "" "CMD_VAR=cmd_value\nPROJECT_VAR=project_value\n"
IMAGE_NAME=$(IMAGE_PREFIX)-$(patsubst test-%,%,$@)-candidate test/run-env-vars "env" "PROJECT_VAR=project_value"
IMAGE_NAME=$(IMAGE_PREFIX)-$(patsubst test-%,%,$@)-candidate test/run-lambda-py
IMAGE_NAME=$(IMAGE_PREFIX)-$(patsubst test-%,%,$@)-candidate test/run-lambda-api
127 changes: 127 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,3 +97,130 @@ Now run the `hello-world` Docker image
When you visit http://localhost:8086 you will see

![](localhost.png)


## Project commands

By default, the s2i assemble step will prepare the docker image for the default command of the project. Within
anaconda-project the default command for a project is the command in the anaconda-project.yml file called `default`.
If a command named `default` is not present then the first command in the anaconda-project.yml file is chosen.

When building the docker image you can choose a non-default command to run with the `-e CMD=<command>` flag. The command
can be either a command defined in the anaconda-project.yml file or an executable in the path of the default `env_spec`
for the project.

## Docker run commands

When the project docker image is built the default runtime command is set to `/usr/libexec/s2i/run`. This script
runs the default command or the one specified at build time. In addition you can execute arbitrary commands from the
project conda environment. Here's an example using the `hello-world` example above.

First the image is built using the default command and you'll see that `docker run` initiates the Panel/Bokeh server.

```
>s2i build https://github.com/AlbertDeFusco/hello-world.git conda/s2i-anaconda-project-alpine hello-world
---> Copying project...
---> Preparing environment for command default...
...
>docker run -p 8086:8086 hello-world
2022-01-10 15:28:01,101 Starting Bokeh server version 2.3.3 (running on Tornado 6.1)
```

Instead the user can run a custom command, for example we'll confirm that these custom commands are executed in the
default Conda environment for the project and all of the appropriate environment variables.

```
>docker run hello-world python -c 'import sys;print(sys.executable)'
/opt/app-root/src/envs/default/bin/python
```

Finally all of the the variables define in the anaconda-project.yml file will be set for use by custom runtime
commands. In this example `BOKEH_ALLOW_WS_ORIGIN: "*"` is defined in the anaconda-project.yml file.

```
>docker run hello-world env
HOSTNAME=f7d56ddd45b4
PWD=/opt/app-root/src
TZ=US/Central
HOME=/opt/app-root/src
LANG=en_US.UTF-8
SHLVL=0
PYTHONDONTWRITEBYTECODE=1
CONDA_VERSION=4.9.2
LC_ALL=en_US.UTF-8
ANACONDA_PROJECT_VERSION=0.10.0
PATH=/opt/app-root/src/envs/default/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PIP_NO_CACHE_DIR=1
CONDA_ROOT=/opt/conda
PROJECT_DIR=/opt/app-root/src
CONDA_PREFIX=/opt/app-root/src/envs/default
CONDA_DEFAULT_ENV=/opt/app-root/src/envs/default
BOKEH_ALLOW_WS_ORIGIN=*
```

## Entrypoints

the `s2i-anaconda-project` builder images provide three entrypoint scripts.

* `/entrypoint.sh`:
* The default entrypoint that execs `anaconda-project run <cmd>`, where `<cmd>` is the command chosen at build time
or runtime.
* `/lambda-api.sh`:
* A cURL-based Lambda runtime that can be used for any command that doesn't utilize the AWS lambda runtime packages.
This entrypoint receives the event data from AWS and sends it to the command as the first argument
* `/lambda.sh`:
* This entrypoint expects that your project utilize the runtime package (`awslambdaric` for Python) to communicate
with AWS Lambda.

### AWS Lambda
To execute the command chosen at build time or a custom command as an AWS Lambda function first build the project image
following the steps above. To execute your image as an AWS Lambda function all you need to do is set the appropriate
entrypoint at runtime.

To test your docker image locally, the AWS Lambda Runtime Emulator has been installed in the builder image. When you
run the docker image locally using either the `/lambda-api.sh` or `/lambda.sh` entrypoint the emulator will launch
and you can make POST requests to the container.

Here's an example where the anaconda-project command does not utilize a Lambda runtime so the `/lambda-api.sh`
entrypoint is enabled.

```
name: test-lambda
packages:
- jq
commands:
default:
unix: jq -ncM '$in | .Key = "output"' --argjson in
```

Here the image is built and run

```
>s2i build -c . conda/s2i-anaconda-project-alpine test-lambda
---> Copying project...
---> Preparing environment for command default...
...
>docker run -p 9000:8080 --entrypoint /lambda-api.sh test-lambda
10 Jan 2022 10:12:36,958 [INFO] (rapid) exec '/usr/libexec/s2i/run-lambda' (cwd=/opt/app-root/src, handler=/usr/libexec/s2i/run)
```

In another terminal make a request to the container

```
>curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'
{"Key":"output"}
```

In the `docker run` terminal you'll see

```
10 Jan 2022 10:12:36,958 [INFO] (rapid) exec '/usr/libexec/s2i/run-lambda' (cwd=/opt/app-root/src, handler=/usr/libexec/s2i/run)
10 Jan 2022 10:13:14,750 [INFO] (rapid) extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory
10 Jan 2022 10:13:14,751 [WARNING] (rapid) Cannot list external agents error=open /opt/extensions: no such file or directory
START RequestId: ad48ef4a-b9a5-4dee-86ca-16a49bca67ab Version: $LATEST
{"Key":"output"}
END RequestId: ad48ef4a-b9a5-4dee-86ca-16a49bca67ab
REPORT RequestId: ad48ef4a-b9a5-4dee-86ca-16a49bca67ab Init Duration: 0.48 ms Duration: 2229.52 ms Billed Duration: 2230 ms Memory Size: 3008 MB Max Memory Used: 3008 MB
```
16 changes: 11 additions & 5 deletions alpine.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,23 +17,29 @@ ENV PATH=/opt/conda/bin:$PATH
ENV PYTHONDONTWRITEBYTECODE=1
ENV HOME=/opt/app-root/src
ENV TZ=US/Central

COPY ./etc/condarc /opt/conda/.condarc
ENV PIP_NO_CACHE_DIR=1

### Set timezone
RUN apk add --no-cache tzdata \
&& cp /usr/share/zoneinfo/${TZ} /etc/localtime \
&& echo ${TZ} > /etc/timezone

### Install and configure miniconda
RUN apk add --no-cache --virtual wget tar bash \
COPY ./etc/condarc /opt/conda/.condarc
RUN apk add --no-cache --virtual wget tar bash curl \
&& wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-py37_4.9.2-Linux-x86_64.sh -O miniconda.sh \
&& sh miniconda.sh -u -b -p /opt/conda \
&& rm -f miniconda.sh \
&& ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh \
&& conda install anaconda-project=0.10.0 anaconda-client conda-repo-cli conda-token tini --yes \
&& conda clean --all --yes \
&& chmod -R 755 /opt/conda
&& chmod -R 755 /opt/conda

### AWS Lambda Runtime Emulator
ADD https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie /opt/aws/
RUN chmod +x /opt/aws/aws-lambda-rie

COPY ./entrypoints/ /
COPY ./s2i/bin/ /usr/libexec/s2i

RUN mkdir -p /opt/app-root && \
Expand All @@ -52,6 +58,6 @@ EXPOSE 8086

WORKDIR $HOME

ENTRYPOINT ["tini", "-g", "--"]
ENTRYPOINT ["/entrypoint.sh"]

CMD ["/usr/libexec/s2i/usage"]
11 changes: 11 additions & 0 deletions entrypoints/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#!/bin/bash

if [ -e /opt/app-root/src/.assembled ]; then
if [[ $1 == "/usr/libexec/s2i/run" ]]; then
exec tini -g -- "$@"
else
exec tini -g -- anaconda-project run "$@"
fi
else
exec tini -g -- "$@"
fi
21 changes: 21 additions & 0 deletions entrypoints/lambda-api.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/bin/bash

ARGS=${@:-"/usr/libexec/s2i/run"}

if [ -e /opt/app-root/src/.assembled ]; then
if [[ $ARGS == "/usr/libexec/s2i/run" ]]; then
if [ -v AWS_LAMBDA_RUNTIME_API ]; then
exec /usr/libexec/s2i/run-lambda "$ARGS"
else
exec /opt/aws/aws-lambda-rie /usr/libexec/s2i/run-lambda "$ARGS"
fi
else
if [ -v AWS_LAMBDA_RUNTIME_API ]; then
exec /usr/libexec/s2i/run-lambda anaconda-project run "$ARGS"
else
exec /opt/aws/aws-lambda-rie /usr/libexec/s2i/run-lambda anaconda-project run "$ARGS"
fi
fi
else
exec "$ARGS"
fi
21 changes: 21 additions & 0 deletions entrypoints/lambda.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/bin/bash

ARGS=${@:-"/usr/libexec/s2i/run"}

if [ -e /opt/app-root/src/.assembled ]; then
if [[ $ARGS == "/usr/libexec/s2i/run" ]]; then
if [ -v AWS_LAMBDA_RUNTIME_API ]; then
exec "$ARGS"
else
exec /opt/aws/aws-lambda-rie "$ARGS"
fi
else
if [ -v AWS_LAMBDA_RUNTIME_API ]; then
exec anaconda-project run "$ARGS"
else
exec /opt/aws/aws-lambda-rie anaconda-project run "$ARGS"
fi
fi
else
exec "$ARGS"
fi
2 changes: 2 additions & 0 deletions s2i/bin/assemble
Original file line number Diff line number Diff line change
Expand Up @@ -47,3 +47,5 @@ find ./envs/* -follow -type f -name '*.js.map' -delete
if find_js; then
find ./envs/*/lib/python*/site-packages/bokeh/server/static -follow -type f -name '*.js' ! -name '*.min.js' -delete
fi

touch .assembled
2 changes: 1 addition & 1 deletion s2i/bin/run
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@

project_command=${CMD:-default}

exec anaconda-project run ${project_command} --anaconda-project-port 8086 --anaconda-project-address 0.0.0.0 --anaconda-project-no-browser --anaconda-project-use-xheaders
exec anaconda-project run ${project_command} --anaconda-project-port 8086 --anaconda-project-address 0.0.0.0 --anaconda-project-no-browser --anaconda-project-use-xheaders "$@"
80 changes: 80 additions & 0 deletions s2i/bin/run-lambda
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
#!/bin/bash

set -uo pipefail

RUN_COMMAND=${@}

# Constants
RUNTIME_PATH="2018-06-01/runtime"
mkdir -p /tmp/.aws
touch /tmp/.aws/config
# export HOME="/tmp"
# export AWS_CONFIG_FILE="/tmp/.aws/config"

# Send initialization error to Lambda API
sendInitError () {
ERROR_MESSAGE=$1
ERROR_TYPE=$2
ERROR="{\"errorMessage\": \"$ERROR_MESSAGE\", \"errorType\": \"$ERROR_TYPE\"}"
curl -sS -X POST -d "$ERROR" "http://${AWS_LAMBDA_RUNTIME_API}/${RUNTIME_PATH}/init/error" > /dev/null
}

# Send runtime error to Lambda API
sendRuntimeError () {
REQUEST_ID=$1
ERROR_MESSAGE=$2
ERROR_TYPE=$3
STACK_TRACE=$4
ERROR="{\"errorMessage\": \"$ERROR_MESSAGE\", \"errorType\": \"$ERROR_TYPE\", \"stackTrace\": \"$STACK_TRACE\"}"
curl -sS -X POST -d "$ERROR" "http://${AWS_LAMBDA_RUNTIME_API}/${RUNTIME_PATH}/invocation/${REQUEST_ID}/error" > /dev/null
}

# Send successful response to Lambda API
sendResponse () {
REQUEST_ID=$1
REQUEST_RESPONSE_FILE=$2
cat $REQUEST_RESPONSE_FILE | curl -sS -X POST -d @- "http://${AWS_LAMBDA_RUNTIME_API}/${RUNTIME_PATH}/invocation/${REQUEST_ID}/response" > /dev/null
}

# # Make sure handler function exists
# type "$(echo $_HANDLER | cut -d. -f2)" > /dev/null 2>&1
# if [[ ! $? -eq "0" ]]; then
# sendInitError "Failed to load handler '$(echo $_HANDLER | cut -d. -f2)' from module '$(echo $_HANDLER | cut -d. -f1)'. Function '$(echo $_HANDLER | cut -d. -f2)' does not exist." "InvalidHandlerException"
# exit 1
# fi

# Processing
while true
do
HEADERS="/tmp/headers-$(date +'%s')"
STDOUT="/tmp/stdout-$(date +'%s')"
STDERR="/tmp/stderr-$(date +'%s')"
touch $HEADERS
touch $STDOUT
touch $STDERR
EVENT_DATA=$(curl -sS -LD "$HEADERS" -X GET "http://${AWS_LAMBDA_RUNTIME_API}/${RUNTIME_PATH}/invocation/next")
REQUEST_ID=$(grep -Fi Lambda-Runtime-Aws-Request-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)
# Export some additional context
export AWS_LAMBDA_REQUEST_ID=$REQUEST_ID
export AWS_LAMBDA_DEADLINE_MS=$(grep -Fi Lambda-Runtime-Deadline-Ms "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)
export AWS_LAMBDA_FUNCTION_ARN=$(grep -Fi Lambda-Runtime-Invoked-Function-Arn "$HEADERS" | cut -d" " -f2)
export AWS_LAMBDA_TRACE_ID=$(grep -Fi Lambda-Runtime-Trace-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)
# Execute the command
$RUN_COMMAND "$EVENT_DATA" 1> $STDOUT 2> $STDERR
EXIT_CODE=$?
cat $STDOUT
cat $STDERR
# Respond to Lambda API
if [[ $EXIT_CODE -eq "0" ]]; then
sendResponse "$REQUEST_ID" "$STDOUT"
else
sendRuntimeError "$REQUEST_ID" "Exited with code $EXIT_CODE" "RuntimeErrorException" "$(cat $STDERR)"
fi
# Clean up
rm -f -- "$HEADERS"
rm -f -- "$STDOUT"
rm -f -- "$STDERR"
unset HEADERS
unset STDOUT
unset STDERR
done
Loading

0 comments on commit db772b4

Please sign in to comment.