Skip to content

Commit

Permalink
Merge pull request #337 from grycap/fdl
Browse files Browse the repository at this point in the history 
Function Definition Language
  • Loading branch information
@srisco
srisco authored Jan 7, 2020
2 parents bb5f56f + 959841a commit 99a8cb2
Show file tree
Hide file tree
Showing 84 changed files with 2,903 additions and 2,341 deletions.
254 changes: 150 additions & 104 deletions docs/source/advanced_usage.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,22 +6,43 @@ Define a shell-script for each invocation of the Lambda function

Instead of packaging the script to be used inside the container image and having to modify the image each time you want to modify the script, you can specify a shell-script when initializing the Lambda function to trigger its execution inside the container on each invocation of the Lambda function. For example::

cat >> init-script.yaml << EOF
cat >> cow.sh << EOF
#!/bin/bash
/usr/games/cowsay "Executing init script !!"
EOF

cat >> cow.yaml << EOF
functions:
scar-cowsay:
image: grycap/cowsay
init_script: src/test/test-cowsay.sh
aws:
- lambda:
name: scar-cowsay
init_script: cow.sh
container:
image: grycap/cowsay
EOF

scar init -f init-script.yaml
scar init -f cow.yaml

or using CLI parameters::

scar init -s src/test/test-cowsay.sh -n scar-cowsay -i grycap/cowsay
scar init -s cow.sh -n scar-cowsay -i grycap/cowsay

Now whenever this Lambda function is executed, the script will be run in the container::

scar run -f init-script.yaml
scar run -f cow.yaml

Request Id: fb925bfa-bc65-47d5-beed-077f0de471e2
Log Group Name: /aws/lambda/scar-cowsay
Log Stream Name: 2019/12/19/[$LATEST]0eb088e8a18d4599a572b7bf9f0ed321
__________________________
< Executing init script !! >
--------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||


As explained in next section, this can be overridden by speciying a different shell-script when running the Lambda function.

Expand All @@ -31,24 +52,69 @@ Executing an user-defined shell-script

You can execute the Lambda function and specify a shell-script locally available in your machine to be executed within the container::

cat >> run-script.yaml << EOF
cat >> runcow.sh << EOF
#!/bin/bash
/usr/games/cowsay "Executing run script !!"
EOF

cat >> cow.yaml << EOF
functions:
scar-cowsay:
image: grycap/cowsay
run_script: src/test/test-cowsay.sh
aws:
- lambda:
name: scar-cowsay
run_script: runcow.sh
container:
image: grycap/cowsay
EOF

scar run -f run-script.yaml
scar init -f cow.yaml

or using CLI parameters::
Now if you execute the function without passing more parameters, the entrypoint of the container is executed::

scar run -n scar-cowsay

Request Id: 97492a12-ca84-4539-be80-45696501ee4a
Log Group Name: /aws/lambda/scar-cowsay
Log Stream Name: 2019/12/19/[$LATEST]d5cc7a9db9b44e529873130f6d005fe1
____________________________________
/ No matter where I go, the place is \
\ always called "here". /
------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||

But, when you use the configuration file with the ``run_script`` property::
scar run -f cow.yaml

scar run -s src/test/test-cowsay.sh -n scar-cowsay
or use CLI parameters::

or a combination of both (to avoid editing the .yaml file)::
scar run -n scar-cowsay -s runcow.sh

scar run -f run-script.yaml -s /tmp/test-cowsay.sh
or a combination of both (to avoid editing the initial .yaml file)::

Have in mind that the script used in combination with the run command is no saved anywhere. It is uploaded and executed inside the container, but the container image is not updated. The shell-script needs to be specified and can be changed in each different execution of the Lambda function.
scar run -f cow.yaml -s runcow.sh

the passed script is executed::

Request Id: db3ff40e-ab51-4f90-95ad-7473751fb9c7
Log Group Name: /aws/lambda/scar-cowsay
Log Stream Name: 2019/12/19/[$LATEST]d5cc7a9db9b44e529873130f6d005fe1
_________________________
< Executing run script !! >
-------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||

Have in mind that the script used in combination with the run command is no saved anywhere.
It is uploaded and executed inside the container, but the container image is not updated.
The shell-script needs to be specified and can be changed in each different execution of the Lambda function.


Passing environment variables
Expand All @@ -57,131 +123,111 @@ Passing environment variables
You can specify environment variables to the init command which will be in turn passed to the executed Docker container and made available to your shell-script.
Using a configuration file::

cat >> env-var.yaml << EOF
functions:
scar-cowsay:
image: grycap/cowsay
init_script: src/test/test-global-vars.sh
environment:
TEST1: 45
TEST2: 69
cat >> cow.sh << EOF
#!/bin/bash
env | /usr/games/cowsay
EOF

scar init -f env-var.yaml

or using CLI parameters::

scar init -e TEST1=45 -e TEST2=69 -s src/test/test-global-vars.sh -n scar-cowsay

You can also update the environment variables by changing the configuration file and then using the update command::

cat >> env-var.yaml << EOF
cat >> cow-env.yaml << EOF
functions:
scar-cowsay:
image: grycap/cowsay
init_script: src/test/test-global-vars.sh
environment:
TEST1: 145
TEST2: i69
TEST3: 42
aws:
- lambda:
name: scar-cowsay
run_script: runcow.sh
container:
image: grycap/cowsay
environment:
Variables:
TESTKEY1: val1
TESTKEY2: val2
EOF

scar update -f env-var.yaml
scar init -f cow-env.yaml

or::

scar update -e EST1: 145 -e TEST2: i69 -e TEST2: 42 -n scar-cowsay

In addition, the following environment variables are automatically made available to the underlying Docker container:

* AWS_ACCESS_KEY_ID
* AWS_SECRET_ACCESS_KEY
* AWS_SESSION_TOKEN
* AWS_SECURITY_TOKEN

This allows a script running in the Docker container to access other AWS services. As an example, see how the AWS CLI runs on AWS Lambda in the `examples/aws-cli <https://github.com/grycap/scar/tree/master/examples/aws-cli>`_ folder.


Executing cli commands
----------------------

To run commands inside the docker image you can specify the command to be executed at the end of the command line::
or using CLI parameters::

scar run -f basic-cow.yaml ls
scar init -n scar-cowsay -i grycap/cowsay -e TEST1=45 -e TEST2=69 -s cow.sh


Passing arguments
^^^^^^^^^^^^^^^^^
Executing custom commands and arguments
---------------------------------------

You can also supply arguments which will be passed to the command executed in the Docker container::
To run commands inside the docker image you can specify the command to be executed at the end of the command line.
This command overrides any ``init`` or ``run`` script defined::

scar run -f basic-cow.yaml /usr/bin/perl /usr/games/cowsay Hello World
scar run -f cow.yaml df -h

Note that since cowsay is a Perl script you will have to prepend it with the location of the Perl interpreter (in the Docker container).
Request Id: 39e6fc0d-6831-48d4-aa03-8614307cf8b7
Log Group Name: /aws/lambda/scar-cowsay
Log Stream Name: 2019/12/19/[$LATEST]9764af5bf6854244a1c9469d8cb84484
Filesystem Size Used Avail Use% Mounted on
/dev/root 526M 206M 309M 41% /
/dev/vdb 1.5G 21M 1.4G 2% /dev


Obtaining a JSON Output
-----------------------

For easier scripting, a JSON output can be obtained by including the `-j` or the `-v` (even more verbose output) flags::

scar run -f basic-cow.yaml -j
scar run -f cow.yaml -j

Upload docker images using an S3 bucket
---------------------------------------

If you want to save some space inside the lambda function you can deploy a lambda function using an S3 bucket by issuing the following command::

cat >> s3-bucket.yaml << EOF
functions:
scar-cowsay:
image: grycap/cowsay
s3:
deployment_bucket: scar-cowsay
EOF
{ "LambdaOutput":
{
"StatusCode": 200,
"Payload": " _________________________________________\n/ \"I always avoid prophesying beforehand \\\n| because it is much better |\n| |\n| to prophesy after the event has already |\n| taken place. \" - Winston |\n| |\n\\ Churchill /\n -----------------------------------------\n \\ ^__^\n \\ (oo)\\_______\n (__)\\ )\\/\\\n ||----w |\n || ||\n",
"LogGroupName": "/aws/lambda/scar-cowsay",
"LogStreamName": "2019/12/19/[$LATEST]a4ba02914fd14ab4825d6c6635a1dfd6",
"RequestId": "fcc4e24c-1fe3-4ca9-9f00-b15ec18c1676"
}
}

scar init -f s3-bucket.yaml

or using the CLI::

scar init -db scar-cowsay -n scar-cowsay -i grycap/cowsay

The maximum deployment package size allowed by AWS is an unzipped file of 250MB. With this restriction in mind, SCAR downloads the docker image to a temporal folder and creates the udocker file structure needed.
* If the image information and the container filesystem fit in the 250MB SCAR will upload everything and the lambda function will not need to download or create a container structure thus improving the execution time of the function. This option gives the user the full 500MB of ``/tmp/`` storage.
* If the container filesystem doesn't fit in the deployment package SCAR will only upload the image information, that is, the layers. Also the lambda function execution time is improved because it doesn't need to dowload the container. In this case udocker needs to create the container filesystem so the first function invocation can be delayed for a few of seconds. This option usually duplicates the available space in the ``/tmp/`` folder with respect to the SCAR standard initialization.

Upload docker image files using an S3 bucket
--------------------------------------------

SCAR also allows to upload a saved docker image::
SCAR allows to upload a saved docker image.
We created the image file with the command ``docker save grycap/cowsay > cowsay.tar.gz``::

cat >> s3-bucket.yaml << EOF
cat >> cow.yaml << EOF
functions:
scar-cowsay:
image_file: slim_cow.tar.gz
s3:
deployment_bucket: scar-cowsay
aws:
- lambda:
name: scar-cowsay
container:
image_file: cowsay.tar.gz
deployment:
bucket: scar-test
EOF

scar init -f s3-bucket.yaml
scar init -f cow.yaml

and for the CLI fans::
or for the CLI fans::

scar init -db scar-cowsay -n scar-cowsay -if slim_cow.tar.gz
scar init -db scar-cowsay -n scar-cowsay -if cowsay.tar.gz

The behavior of SCAR is the same as in the case above (when uploading an image from docker hub). The image file is unpacked in a temporal folder and the udocker layers and container filesystem are created. Depending on the size of the layers and the filesystem, SCAR will try to upload everything or only the image layers.
Have in mind that the maximum deployment package size allowed by AWS is an unzipped file of 250MB.
The image file is unpacked in a temporal folder and the udocker layers are created.
Depending on the size of the layers, SCAR will try to upload them or will show the user an error.

Upload 'slim' docker image files in the payload
-----------------------------------------------

Finally, if the image is small enough, SCAR allows to upload it in the function payload. Due to the SCAR libraries weighting ~10MB, the maximum size of the image uploaded using this method should not be bigger than ~40MB::
Finally, if the image is small enough, SCAR allows to upload it in the function payload wich is ~50MB::

docker save grycap/minicow > minicow.tar.gz

cat >> slim-image.yaml << EOF
cat >> minicow.yaml << EOF
functions:
scar-cowsay:
image_file: slimcow.tar.gz
aws:
- lambda:
name: scar-cowsay
container:
image_file: minicow.tar.gz
EOF

scar init -f slim-image.yaml
scar init -f minicow.yaml

To help with the creation of slim images, you can use `minicon <https://github.com/grycap/minicon>`_. Minicon is a general tool to analyze applications and executions of these applications to obtain a filesystem that contains all the dependencies that have been detected. By using minicon the size of the cowsay image was reduced from 170MB to 11MB.
To help with the creation of slim images, you can use `minicon <https://github.com/grycap/minicon>`_.
Minicon is a general tool to analyze applications and executions of these applications to obtain a filesystem that contains all the dependencies that have been detected.
By using minicon the size of the cowsay image was reduced from 170MB to 11MB.
Loading

0 comments on commit 99a8cb2

Please sign in to comment.