Storing Secure Config in Amazon S3 for AWS ECS, Lambda, and More

We run containerized services in Amazon ECS (EC2 Container Service) and wanted
a way to set environment variables containing sensitive configuration in
running containers without defining those environment variables in ECS task
definitions. We do this by storing encrypted (at rest)
key/value JSON objects in S3 and reading them into the environment in either a
docker entrypoint script or in the application’s bootstrap process. We wrote
Snagsby
(https://github.com/roverdotcom/snagsby)
to make this process easier. We’ve found it a convenient way to read
configuration into lambda functions as well.

JSON Objects in S3 for Snagsby

Snagsby makes it easy to read a key/value JSON object from S3, output in a
format that can be evaluated by a shell for the purpose of setting environment
variables. This idea is covered in this AWS blog post:

https://aws.amazon.com/blogs/security/how-to-manage-secrets-for-amazon-ec2-container-service-based-applications-by-using-amazon-s3-and-docker/

The referenced AWS blog post describes ways to restrict an S3 bucket to only
allow kms encrypted objects, as well as requiring secure transport and even
restricting access using VPC S3 endpoints. We typically encrypt our JSON
objects using unique KMS keys that we then give the appropriate application the
ability to decrypt when using snagsby to read configuration. Individual KMS
keys allow for nice auditing.

An example python boto3 script to upload KMS encrypted snagsby compatible JSON
to S3:

import json

import boto3


s3 = boto3.resource('s3')

s3.Bucket('my-config-bucket').put_object(
    Body=json.dumps({
        'API_KEY': '12345',
        'CREDENTIAL': 'ABC',
    }),
    Key='myapp/production/config.json',
    ServerSideEncryption='aws:kms',
    SSEKMSKeyId='7777abcd',
)

Snagsby only requires s3:GetObject to read a key, so the following IAM policy
would be all an app needed to read the above configuration using snagsby.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::myapp/production/config.json"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": [
                "arn:aws:kms:us-west-2:777777777777:key/7777abcd"
            ]
        }
    ]
}

Using Snagsby

Snagsby is written in golang and binaries are available on github. After uploading a JSON object to S3 you would evaluate the output of running the snagsby command in a shell. Snagsby reads a list of sources from either the SNAGSBY_SOURCE environment variable, or from command line arguments. It will output export statements which can be evaluated by a shell.

Say you upload this JSON to s3://my-config/myapp/production/config.json:

{
    "API_KEY": "123",
    "CREDENTIALS": "7777"
}

Running snagsby would output this to stdout:

snagsby s3://my-config/myapp/production/config.json

export API_KEY="123"
export CREDENTIALS="7777"

Snagsby also reads its list of sources from the SNAGSBY_SOURCE environment
variables and will merge together multiple sources in order.

# s3://my-config/one.json
{
    "SETTING": "FIRST",
    "ONE": "1"
}

# s3://my-config/two.json
{
    "SETTING": "SECOND",
    "TWO": "2"
}

Running snagsby against both sources would merge the output in order

SNAGSBY_SOURCE="s3://my-config/one.json, s3://my-config/two.json" snagsby

export SETTING="SECOND"
export ONE="1"
export TWO="2"

How We Use Snagsby In AWS ECS (EC2 Container Service)

We use ECS task iam roles to allow our application to s3:GetObject and
kms:Decrypt the relevant key containing the configuration needed by the app.
We then set the SNAGSBY_SOURCE environment variable in our ecs task
definition. This allows us to use an entrypoint script in our containers that
evaluates snagsby before executing the docker CMD.

Example docker entrypoing script:

#!/bin/bash

set -e

# Snagsby will by default read sources found in the SNAGSBY_SOURCE env var. If
# the env var is not set or is empty snagsby exits 0 and outputs nothing
eval $(snagsby)


# Execute what is passed to the entrypoint scrip, which will be the
# Dockerfile CMD
exec "$@"

This entrypoint is safe for the container even if SNAGSBY_SOURCE isn’t set as
snagsby will just output nothing to stdout.

Using Snagsby in Python (snagsby-py)

We’ve also released a python library for reading snagsby JSON. It’s called
snagsby-py. This can be used to
read snagsby configuration directly in python apps, including python lambda
functions.

Example setting.py file in a python app

import os

import snagsby

# By default snagsby.load() reads from the SNAGSBY_SOURCE env var and injects
# the contents in os.environ. This is safe to call if no SNAGSBY_SOURCE is set.
snagsby.load()

print(os.environ['SETTING'])

Summary

Snagsby has worked well for us so far. It’s a fairly straightforward solution,
simple and somewhat limited by design, and didn’t require adding any additional
infrastructure.