Amazon Web Services users have been eager to find a simpler method for deploying serverless applications, built using Lambda functions, API gateways, and AWS DynamoDB. As a result, AWS released a new model called the AWS Serverless Application Model (SAM) which makes it easier for customers to deploy their serverless applications using AWS CloudFormation. With this announcement, there are now two serverless frameworks for building serverless architectures — deploying serverless applications using AWS CloudFormation and using the AWS SAM. However, the new AWS SAM uses CloudFormation natively to deploy, which is a definite plus for AWS users.
With AWS CodePipeline’s CloudFormation deployment action, users can now easily build an end to end workflow for deploying serverless applications using AWS services AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, AWS SAM, and AWS CloudFormation. We’ve modeled out this workflow, which we’d like to share with you.
Our criteria for the implementation was:
- The deployment infrastructure itself must be serverless
- We must support a pipeline with multiple stages complete with testing and both automated and manual promotions
- We must support the easy creation and deployment of multiple microservices by independent teams
AWS Serverless Application Model extends CloudFormation to provide a simplified way of defining the Amazon API Gateway APIs, AWS Lambda functions, S3 buckets, and Amazon DynamoDB tables needed by the serverless application. As a result, as a foundation to support our criteria above, and to start using AWS Lambda, there were two pieces we had to get into place. First, we needed a Code Delivery pipeline and second, we needed a CloudFormation template to create our Lambda functions, API Gateway connectors, AWS IAM roles, associated S3 bucket, and the associated DynamoDB table. For the second, we used AWS SAM.
To create a serverless application deployment pipeline, we used the following technologies:
- AWS SAM to define the serverless application and its resources
- AWS CodeCommit as the source repository
- AWS IAM to define permissions and roles
- AWS CodeBuild to package and test the source code and SAM templates
- AWS Lambda to deploy the Node modules as functions
- AWS API Gateway/Swagger to define and implement API endpoints
- AWS CloudFormation to deploy the application as well as the AWS CodePipeline
- AWS S3 to store the build and test artifacts
- AWS CodePipeline to orchestrate the application deployment
- AWS DynamoDB as a sample DB resource to simulate API operations
For our serverless application assessment, we crafted a simple microservices-based application with two microservices. Without getting into the specifics, we assume that each service consists of multiple Lambda functions to service different endpoints. I.e., assume two services Microservice1 and Microservice2 with two and three Lambda functions each:
- Microservice 1
- Lambda function 1
- Lambda function 2
- Microservice 2
- Lambda function 1
- Lambda function 2
- Lambda function 3
How the code is structured in the code repository is critical not only to building an efficient CI/CD pipeline but also to establishing organizational norms that encourage collaboration while reducing interference. We have developed a code structure where the code for each microservice lives in its own code repository. This gives each team the flexibility to completely own their own microservices without worrying about coordinating with other teams. Each code repository contains:
- A template.yml file describing the components of the architecture using AWS SAM. There include Lambda functions, their permissions, API gateway, DynamoDB tables, and S3 buckets needed by the service. A template-stage.yml also exists which defines the same for the staging environment.
- A buildspec.yml which instructs AWS CodeBuild on how to package the SAM CloudFormation templates.
- A testspec.yml that defines how to test the microservice in the staging environment using CodeBuild.
- test.js a simple JS script that CodeBuild runs (via testspec.yml) to run a test against the deployed service.
- swagger.yml which defines the API of the service.
At a high-level, The CodePipeline is configured to trigger each time a code update is pushed to the SCM (in our example, we are using AWS CodeCommit). When it detects a code update, CodePipeline will trigger the packaging and updating of the AWS SAM template using CodeBuild. CloudFormation will deploy the Lambda Based Application into a staging environment. After that, CodeBuild will run tests to check the functionality of the new code deployed in the staging stage. If everything works, CodePipeline again uses AWS SAM to update the production Lambda functions and new the new code is deployed.
While this workflow is conceptually simple, we ran into some engineering challenges posed by limitations in CodePipeline. The actual pipeline looked as follows:
- Get code from SCM.
- Run CodeBuild to package the dev CloudFormation Template and rename* the testspec.yml to buildspec.yml.
- Create changesets for development.
- Execute changesets.
- Run CodeBuild to test the deployed function on development and rename* the buildspec.yml.temp back to buildspec.yml.
- Run CodeBuild to package the production CFT and rename* the testspec.yml to buildspec.yml.
- Create changesets for production.
- Execute changesets for production.
- Run CodeBuild to test the deployed function on production.
*The files had to be renamed in the pipeline to overcome the limitation that AWS CodeBuild expects the build specification to be called buildspec.yml, however, we wanted to use CodeBuild for running the tests as well.
Before SAM, this exercise was more challenging because defining serverless applications required more complex CloudFormation templates or the use of AWS API Gateway. However, as you can see here, with these new resource types that allow us to create CloudFormation templates that define serverless resources with ease, we are easily able to package a serverless app and deploy it with CloudFormation. Indeed, the process of creating a deployment pipeline for our serverless application was straight-forward with SAM and easily replicable. If you’d like to get started using AWS SAM to build serverless applications, the documentation page and AWS serverless application model on GitHub are two great places to get started.
*This was originally written by Flux7 Inc., which has become Flux7, an NTT DATA Services Company as of December 30, 2019
Post Date: 2017-09-14