lambgeo / docker-lambda Public

GDAL based docker image made for AWS Lambda

AWS lambda (Amazonlinux) like docker images with GDAL.

Docker Images

Based on lambci/lambda-base-2:build (amazonlinux2) for newer runtimes (e.g python 3.8)

GDAL 3.3.0 (June 2021)
- lambgeo/lambda-gdal:3.3-al2
GDAL 3.2.3 (June 2021)
- lambgeo/lambda-gdal:3.2-al2
For python 3.8
- lambgeo/lambda-gdal:3.3-python3.8
- lambgeo/lambda-gdal:3.2-python3.8

Creating Lambda packages

Dockerfile

FROM lambgeo/lambda-gdal:3.2-al2 as gdal

# We use lambci docker image for the runtime
FROM lambci/lambda:build-python3.8

ENV PACKAGE_PREFIX=/var/task

# Bring C libs from lambgeo/lambda-gdal image
COPY --from=gdal /opt/lib/ ${PACKAGE_PREFIX}/lib/
COPY --from=gdal /opt/include/ ${PACKAGE_PREFIX}/include/
COPY --from=gdal /opt/share/ ${PACKAGE_PREFIX}/share/
COPY --from=gdal /opt/bin/ ${PACKAGE_PREFIX}/bin/
ENV \
  GDAL_DATA=${PACKAGE_PREFIX}/share/gdal \
  PROJ_LIB=${PACKAGE_PREFIX}/share/proj \
  GDAL_CONFIG=${PACKAGE_PREFIX}/bin/gdal-config \
  GEOS_CONFIG=${PACKAGE_PREFIX}/bin/geos-config \
  PATH=${PACKAGE_PREFIX}/bin:$PATH

# Set some useful env
ENV \
  LANG=en_US.UTF-8 \
  LC_ALL=en_US.UTF-8 \
  CFLAGS="--std=c99"

# Copy any local files to the package
COPY handler.py ${PACKAGE_PREFIX}/handler.py

# This is needed for rasterio
RUN pip3 install cython numpy --no-binary numpy

# Install some requirements to `/var/task` (using `-t` otpion)
RUN pip install numpy rasterio mercantile --no-binary :all: -t ${PACKAGE_PREFIX}/

# Reduce size of the C libs
RUN cd $PACKAGE_PREFIX && find lib -name \*.so\* -exec strip {} \;

# Create package.zip
RUN cd $PACKAGE_PREFIX && zip -r9q /tmp/package.zip *

Or if you are working with python, you can use lambgeo pre-build docker images:

FROM lambgeo/lambda-gdal:3.2-python3.8

# Copy any local files to the package
COPY handler.py ${PACKAGE_PREFIX}/handler.py

# Install some requirements to `/var/task` (using `-t` otpion)
RUN pip install numpy rasterio mercantile --no-binary :all: -t ${PACKAGE_PREFIX}/

# Reduce size of the C libs
RUN cd $PREFIX && find lib -name \*.so\* -exec strip {} \;

# Create package.zip
RUN cd $PACKAGE_PREFIX && zip -r9q /tmp/package.zip *
RUN cd $PREFIX && zip -r9q --symlinks /tmp/package.zip lib/*.so* share
RUN cd $PREFIX && zip -r9q --symlinks /tmp/package.zip bin/gdal* bin/ogr* bin/geos* bin/nearblack

Build and create package.zip

docker build --tag package:latest .
docker run --name lambda -w /var/task --volume $(shell pwd)/:/local -itd package:latest bash
docker cp lambda:/tmp/package.zip package.zip
docker stop lambda
docker rm lambda

Package content should be like:

package.zip
  |
  |___ bin/      # GDAL binaries
  |___ lib/      # Shared libraries (GDAL, PROJ, GEOS...)
  |___ share/    # GDAL/PROJ data directories
  |___ rasterio/
  ....
  |___ handler.py
  |___ other python module

Deploy and Set Environment variables

For Rasterio or other libraries to be aware of GDAL/PROJ C libraries, you need to set up those 2 envs:

GDAL_DATA: /var/task/share/gdal
PROJ_LIB: /var/task/share/proj

Other variable

Starting with gdal3.1 (PROJ 7.1), you can set PROJ_NETWORK=ON to use remote grids: https://proj.org/usage/network.html

AWS Lambda Layers

gdal	amazonlinux version	size (Mb)	unzipped size (Mb)	arn
3.3	2	27.7	67.3	arn:aws:lambda:{REGION}:524387336408:layer:gdal33-al2:{VERSION}
3.2	2	26.7	64.6	arn:aws:lambda:{REGION}:524387336408:layer:gdal32-al2:{VERSION}

archived

gdal	amazonlinux version	size (Mb)	unzipped size (Mb)	arn
3.1	2	25.8	61	arn:aws:lambda:{REGION}:524387336408:layer:gdal31-al2:{VERSION}
2.4	2	19.5	63.6	arn:aws:lambda:{REGION}:524387336408:layer:gdal24-al2:{VERSION}

see /layer.json for the list of arns

cat layer.json| jq '.[] | select(.region == "us-west-2")'
{
  "region": "us-west-2",
  "layers": [
    {
      "name": "gdal24",
      "arn": "arn:aws:lambda:us-west-2:524387336408:layer:gdal24:4",
      "version": 4
    },
    {
      "name": "gdal31",
      "arn": "arn:aws:lambda:us-west-2:524387336408:layer:gdal31:4",
      "version": 4
    },
    {
      "name": "gdal32",
      "arn": "arn:aws:lambda:us-west-2:524387336408:layer:gdal32:4",
      "version": 4
    },
    {
      "name": "gdal24-al2",
      "arn": "arn:aws:lambda:us-west-2:524387336408:layer:gdal24-al2:3",
      "version": 3
    },
    {
      "name": "gdal31-al2",
      "arn": "arn:aws:lambda:us-west-2:524387336408:layer:gdal31-al2:3",
      "version": 3
    },
    {
      "name": "gdal32-al2",
      "arn": "arn:aws:lambda:us-west-2:524387336408:layer:gdal32-al2:5",
      "version": 5
    },
    {
      "name": "gdal33-al2",
      "arn": "arn:aws:lambda:us-west-2:524387336408:layer:gdal33-al2:1",
      "version": 1
    }
  ]
}

Layer content:

layer.zip
  |
  |___ bin/      # Binaries
  |___ lib/      # Shared libraries (GDAL, PROJ, GEOS...)
  |___ share/    # GDAL/PROJ data directories

The layer content will be unzip in /opt directory in AWS Lambda. For the python libs to be able to use the C libraries you have to make sure to set 2 important environment variables:

GDAL_DATA: /opt/share/gdal
PROJ_LIB: /opt/share/proj

How To

There are 2 ways to use the layers:

1. Simple (No dependencies)

If you don't need to add more runtime dependencies, you can just create a lambda package (zip file) with you lambda handler.

zip -r9q package.zip handler.py

Content:

package.zip
  |___ handler.py   # aws lambda python handler

AWS Lambda Config:

arn: arn:aws:lambda:us-east-1:524387336408:layer:gdal32:1 (example)
env:
- GDAL_DATA: /opt/share/gdal
- PROJ_LIB: /opt/share/proj
lambda handler: handler.handler

2. Advanced (need other python dependencies)

If your lambda handler needs more dependencies you'll have to use the exact same environment. To ease this you can find the docker images for each lambda on docker hub.

Create a docker file

FROM lambgeo/lambda-gdal:3.2-al2

# We use lambci docker image for the runtime
FROM lambci/lambda:build-python3.8

# Bring C libs from lambgeo/lambda-gdal image
COPY --from=gdal /opt/lib/ /opt/lib/
COPY --from=gdal /opt/include/ /opt/include/
COPY --from=gdal /opt/share/ /opt/share/
COPY --from=gdal /opt/bin/ /opt/bin/
ENV \
  GDAL_DATA=/opt/share/gdal \
  PROJ_LIB=/opt/share/proj \
  GDAL_CONFIG=/opt/bin/gdal-config \
  GEOS_CONFIG=/opt/bin/geos-config \
  PATH=/opt/bin:$PATH

# Set some useful env
ENV \
  LANG=en_US.UTF-8 \
  LC_ALL=en_US.UTF-8 \
  CFLAGS="--std=c99"

ENV PYTHONUSERBASE=/var/task

# Install dependencies
COPY handler.py $PYTHONUSERBASE/handler.py

# Here we use the `--user` option to make sure to not replicate modules.
RUN pip install rio-tiler --user

# Move some files around
RUN mv ${PYTHONUSERBASE}/lib/python3.8/site-packages/* ${PYTHONUSERBASE}/
RUN rm -rf ${PYTHONUSERBASE}/lib

echo "Create archive"
RUN cd $PYTHONUSERBASE && zip -r9q /tmp/package.zip *

create package

docker build --tag package:latest .
docker run --name lambda -w /var/task -itd package:latest bash
docker cp lambda:/tmp/package.zip package.zip
docker stop lambda
docker rm lambda

Content:

package.zip
  |___ handler.py   # aws lambda python handler
  |___ module1/     # dependencies
  |___ module2/
  |___ module3/
  |___ ...

Refactor

We recently refactored the repo, to see old documentation please refer to https://github.com/lambgeo/docker-lambda/tree/ef66339724b1b7e1a375df912dfd58a9c59ac109

lambgeo / docker-lambda Public

README.md

GDAL based docker image made for AWS Lambda

Docker Images

Archives

Creating Lambda packages

Other variable

AWS Lambda Layers

archived

How To

1. Simple (No dependencies)

2. Advanced (need other python dependencies)

Refactor

About

Releases

Packages

Contributors 2

Languages

lambgeo / docker-lambda Public

Latest commit

Git stats

Files

README.md

GDAL based docker image made for AWS Lambda

Docker Images

Archives

Creating Lambda packages

Other variable

AWS Lambda Layers

archived

How To

1. Simple (No dependencies)

2. Advanced (need other python dependencies)

Refactor

About

Topics

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Contributors 2

Languages

Packages