Share AWS infrastructure attributes from Terraform to Serverless

Dear cloud providers, what are Serverless and Terraform good for?

Terraform is an “infrastructure as a code” tool used to manage the lifecycle of cloud components. You write code that defines the cloud infrastructure, and then Terraform maps it to actual resources like databases, virtual networks, event buses, and so on.

The Serverless Framework is a command-line tool that can handle cloud infrastructure just like Terraform but rather serves as a place where you create code. After the code’s deployment, serverless interacts with cloud components to execute business logic. It’s not without downsides, though.

Separating dependencies

Imagine having multiple applications where all of them need to interact with cloud components to do their job. What is more, each of them sometimes may need to change some part of the infrastructure.

Let’s say you have cloud component definitions linked to application A. Now, whenever application B, C, or D needs to change the cloud infrastructure, you’d have to make changes to application A.

It is really inconvenient and completely against the single responsibility principle.

The solution here is to create a separate entity that will keep only the cloud infrastructure definition. It is way better to have it written down in a tool for this very purpose — like Terraform.

Which tool is better for handling state changes?

Since Terraform works as “infrastructure as a code”, it can create an execution plan before we confirm changes. When you’re unsure what will happen after you deploy the infrastructure definition, you can ask Terraform to simulate changes.

Unfortunately, the Serverless Framework doesn’t support this mechanism out of the box. For you to have it, it would require a workaround. Let’s go for a clean solution — let me show you how to combine both tools. Of all the cloud platforms, I’m using AWS as I’m trained in it.

Why fusing Terraform and Serverless is a problem

Let’s say you have an application written on the Serverless Framework where a single lambda has to be executed every time a record is added to DynamoDB. It could look something like this.

Configuration for the Serverless Framework application

Dummy handler

I used CloudFormation’s Fn::GetAtt function to tell the lambda which DynamoDB stream it should subscribe to. The function returns a value of an attribute StreamArn from the resource ExampleTable.

Run serverless deploy in a system console and go to AWS console -> Lambda -> Functions -> blog-post-dev-hello. See that our lambda was created successfully and subscribes to the DynamoDB stream.

The Serverless Framework was able to resolve Fn::GetAtt: [ExampleTable, StreamArn] without any problems because we wrote the cloud components and the function in the same application.

Now, let’s move our DynamoDB definition to the Terraform application. First, we will delete our current application to prevent later conflicts by executing serverless remove. Then we create a new Terraform application with one file.

But what about our Serverless Framework application?

As you can see, we lost the logical name of the ExampleTable and we cannot use Fn::GetAtt at all. What can we do?

The dirty way: use hardcoded values

The simplest solution is to hardcode DynamoDB stream ARN. So! Deploy your infrastructure. In your Terraform application, execute terraform init and terraform apply. Now you can go to the AWS console -> DynamoDB -> Tables -> example_table -> Exports and streams where you will find our DynamoDB stream ARN.

Let’s copy this value and paste it into serverless.yml.

After that, we can deploy the Serverless Framework application with serverless deploy to see the lambda once again subscribes to the DynamoDB stream.

But hold up! We had to execute a lot of additional steps to get the same solution that Serverless Framework offers. Is there a better way to do this?

The standard way: work with the Systems Manager Parameter Store

A simpler solution for auto-injecting DynamoDB stream ARN — just like the Serverless Framework does it  — is to make use of AWS’ Systems Manager Parameter Store. This service is a simple container with a purpose to store configuration data of your applications. Let’s make use of it!

Add the following code to the Terraform application.

I defined there a new Parameter Store named /blog-post/example-table-stream-arn and the value of the DynamoDB stream ARN as aws_dynamodb_table.this.stream_arn. Let’s look at how I built that value.

aws_dynamodb_table is a type of resource, this is a name of resource,  and stream_arn is one of many attributes that this resource exposes. The whole list of these attributes can be found in Terraform’s documentation.

Ok, now that you know what was added, run a deploy with terraform apply. Check if the resource was created by going to the AWS console -> Systems Manager -> Parameter Store -> /blog-post/example-table-stream-arn.

Now, you have to tell the Serverless Framework application how to fetch this data. Serverless provides a series of special variables that are resolved to various values after deployment. One of those variables references the System Manager Parameter Store: ${ssm:NAME_OF_PARAMETER_STORE}

Add the syntax to the Serverless Framework application.

Deploy the application with serverless deploy and see that the lambda function successfully subscribed to DynamoDB stream once again.

Nice! You now know how to solve the problem using the System Manager Parameter Store. But, surprise, there’s another approach you can use! Let’s see a different scenario.

The exotic way: connecting through AWS CloudFormation Outputs

Imagine you have to declare the infrastructure using a native CloudFormation template. The template is exactly the same one used to declare the infrastructure in the Serverless Framework.

Because of the CloudFormation template’s limitations, you lose access to resource attributes. You now can’t use resource.resource_name.resource_attribute to fetch the stream ARN. Shoo the problem away by using CloudFormation Outputs.

The Outputs is an optional section in CloudFormation templates where you can declare values to be used as cross-stack references. This means you can select which attributes of our infrastructure we want to be easily fetchable.

Adding DynamoDB stream ARN to the Outputs section looks like this.

Now, remove the current Terraform application to prevent a race condition between creating a new database and removing the old one. Run terraform destroy. Then, you can safely deploy our beauty with terraform apply.

Now make the reference to the CloudFormation Outputs variable in the Serverless Framework application. cf:STACK_NAME.OUTPUT_KEY is the syntax for this. In our case, this should look  like this.

Deploy it with serverless deploy, and voilà, the lambda function once again subscribes to the DynamoDB stream.

Remember the 3 ways

You might find other approaches to connecting your serverless applications to Terraform. Note that the bridge between them can be also made with other public cloud providers like Azure or GCP. 

During the development of a spaceship-big project based on AWS, I relied on three ways for connecting serverless and terraform that you learned from the tutorial:

• Hardcoding the DynamoDB stream ARN
• Fetching data from AWS’ Systems Manager Parameter Store
• Connecting through CloudFormation Outputs to the Serverless Framework

I hope they will serve you as good as they served me. Make good use of them and good luck : )