Software

  • Python 3.8
  • Serverless Framework
  • Docker

Information from this post was inspired by this setup:

Also, there is a full configuration towards the end if you just want something that works.

First, make it work

Our goals are to have a final image that allows us to use the serverless framework to deploy to AWS lambda for python applications. Because Lambda really only supports up to python3.8, we want to be sure that’s the core focus of the image. Even though serverless primarily runs on node/npm, it’s easier to start with python as the base. This leads us to the python:3.8-alpine image.

Decide on your node/serverless framework versions. Unfortunately, using this combination restricts you to node 12.22.1. Once lambda supports python3.9, I’d recommend switching to alpine:3.14 + NODEJS_VERSION=14.17.1-r0.

Create and start building out your Dockerfile. 

FROM python:3.8-alpine

ENV NODEJS_VERSION=12.22.1-r0
ENV SERVERLESS_FRAMEWORK_VERSION=2.48.0
ENV GLIBC_VER=2.31-r0

It’s also necessary when working with npm modules to set your WORKDIR, so add:

WORKDIR /usr/app

Start by building out the necessary components of your image:

# Dependencies
RUN apk --no-cache add binutils curl nodejs=${NODEJS_VERSION} npm python3 \
    && curl -sL https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub -o /etc/apk/keys/sgerrand.rsa.pub \
    && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-${GLIBC_VER}.apk \
    && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-bin-${GLIBC_VER}.apk \
    && apk add --no-cache glibc-${GLIBC_VER}.apk glibc-bin-${GLIBC_VER}.apk

# AWS CLI
RUN curl -sL https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip
RUN unzip awscliv2.zip
RUN aws/install

# Serverless
RUN npm install -g serverless@${SERVERLESS_FRAMEWORK_VERSION}
RUN npm install --save-dev serverless-iam-roles-per-function
RUN npm install --save-dev serverless-python-requirements
RUN aws --version
RUN sls --version

From here, you should be able to build the image and have no errors.

docker build -t test_serverless_deploy .

Next, make it right

Ideally, we wouldn’t leave around items that aren’t necessary.

RUN rm -rf \
    awscliv2.zip \
    aws \
    /usr/local/aws-cli/v2/*/dist/aws_completer \
    /usr/local/aws-cli/v2/*/dist/awscli/data/ac.index \
    /usr/local/aws-cli/v2/*/dist/awscli/examples

RUN apk --no-cache del binutils curl
RUN rm glibc-${GLIBC_VER}.apk
RUN rm glibc-bin-${GLIBC_VER}.apk
RUN rm -rf /var/cache/apk/*

Finally, make it quick

The goals here are to make it smaller and have less layers. Here, we combine all the run commands into one to reduce the overally size of the image.

RUN apk --no-cache add binutils curl nodejs=${NODEJS_VERSION} npm gcc musl-dev linux-headers python3 \
    && curl -sL https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub -o /etc/apk/keys/sgerrand.rsa.pub \
    && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-${GLIBC_VER}.apk \
    && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-bin-${GLIBC_VER}.apk \
    && apk add --no-cache \
        glibc-${GLIBC_VER}.apk \
        glibc-bin-${GLIBC_VER}.apk \
    && curl -sL https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip \
    && unzip awscliv2.zip \
    && aws/install \
    && rm -rf \
        awscliv2.zip \
        aws \
        /usr/local/aws-cli/v2/*/dist/aws_completer \
        /usr/local/aws-cli/v2/*/dist/awscli/data/ac.index \
        /usr/local/aws-cli/v2/*/dist/awscli/examples \
    && apk --no-cache del \
        binutils \
        curl \
    && rm glibc-${GLIBC_VER}.apk \
    && rm glibc-bin-${GLIBC_VER}.apk \
    && rm -rf /var/cache/apk/* \
    && npm install -g serverless@${SERVERLESS_FRAMEWORK_VERSION} \
    && npm install --save-dev serverless-iam-roles-per-function \
    && npm install --save-dev serverless-python-requirements \
    && aws --version \
    && sls --version

Test

Update and use the following script to redeploy and test your container.

# Add to test.sh, chmod +x test.sh, run with ./test.sh
docker container stop test_serverless_deploy
docker container prune -f
docker run -dit \
    --name test_serverless_deploy \
    --env AWS_DEFAULT_REGION=us-east-1 \
    --mount type=bind,source="$(pwd)",target=/app \
    --mount type=bind,source="/Users/<your user dir>/.aws",target=/root/.aws \
    test_serverless_deploy
docker exec -it test_serverless_deploy sh

After executing the script, you’ll be at a shell prompt inside the container. Your host’s AWS configuration should be passed in for you to use. From here, you should be able to install any npm dependencies from your current working directory using npm install and deploy with serverless with sls deploy.

Full configuration

FROM python:3.8-alpine

ENV NODEJS_VERSION=12.22.1-r0
ENV SERVERLESS_FRAMEWORK_VERSION=2.48.0
ENV GLIBC_VER=2.31-r0

# Necessary for npm install of modules to work
WORKDIR /usr/app

RUN apk --no-cache add binutils curl nodejs=${NODEJS_VERSION} npm gcc musl-dev linux-headers python3 \
    && curl -sL https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub -o /etc/apk/keys/sgerrand.rsa.pub \
    && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-${GLIBC_VER}.apk \
    && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-bin-${GLIBC_VER}.apk \
    && apk add --no-cache \
        glibc-${GLIBC_VER}.apk \
        glibc-bin-${GLIBC_VER}.apk \
    && curl -sL https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip \
    && unzip awscliv2.zip \
    && aws/install \
    && rm -rf \
        awscliv2.zip \
        aws \
        /usr/local/aws-cli/v2/*/dist/aws_completer \
        /usr/local/aws-cli/v2/*/dist/awscli/data/ac.index \
        /usr/local/aws-cli/v2/*/dist/awscli/examples \
    && apk --no-cache del \
        binutils \
        curl \
    && rm glibc-${GLIBC_VER}.apk \
    && rm glibc-bin-${GLIBC_VER}.apk \
    && rm -rf /var/cache/apk/* \
    && npm install -g serverless@${SERVERLESS_FRAMEWORK_VERSION} \
    && npm install --save-dev serverless-iam-roles-per-function \
    && npm install --save-dev serverless-python-requirements \
    && aws --version \
    && sls --version

Use it in a pipeline

If you’re using this image in a CI/CD pipeline such as bitbucket pipelines, here are some of the steps you should make use of:

- mkdir -p ~/.aws/
- aws configure --profile <your profile> set aws_access_key_id $YOUR_ACCESS_KEY
- aws configure --profile <your profile> set aws_secret_access_key $YOUR_SECRET_KEY
- aws configure --profile <your profile> set region $YOUR_REGION
- npm install # will create node_packages directory
- sls deploy --stage <your stage> --region us-east-1

Troubleshooting

Proxy

Maybe you have a company proxy, you might need to include your certificate inside the image. If you’re on a mac, you can do something similar to:

Run this on your host 

security find-certificate -a -p -c "<your cert name>" /Library/Keychains/System.keychain > <your cert name>.crt

Next, add this to your dockerfile before running curl/wget. Be sure to replace with whatever certificate is for your proxy.

RUN apk update && apk add ca-certificates \
    && rm -rf /var/cache/apk/* \
    && mkdir /usr/local/share/ca-certificates/extra
COPY <your cert name>.crt /usr/local/share/ca-certificates/extra
RUN update-ca-certificates

Serverless isn’t working

Be sure you have a package.json in your working directory, and/or have installed the necessary npm dependencies from your serverless.yml. If you don’t have a node_modules directory inside your working directory, run npm install.