How to develop & test AWS Step Functions locally? With Step Functions Local!

Every company has access to some sort of data. It could be transaction data, the Net Promoter Score (NPS), customer preferences, or recommended results. All of those are later processed and often visualized with some analytical diagrams. You could say that every company, sooner or later, is going to use the data it has. Typically, the objective isn’t to gather the data (because they already have it), but ‌to process it, normalize it, and debug it. And that’s where Step Functions shine. However, there is a catch…

The challenges of Step Functions

Step Functions is a fully serverless AWS service used to build workflows. Under the hood, it is a simple state machine that describes what should be executed during a specific step and what method should be used to accomplish that.

Such workflow could be defined in multiple ways:

• using AWS Step Functions Workflow Studio, a visual no-code editor available as a AWS service,
• or with yaml / json configuration using a special language called Amazon States Language.

The main goal of any workflow is to provide a simple, visual, and easy-to-debug option for critical business operations. At TSH, we use Step Functions as data pipelines for our data processing. You could say it is a sort of Extract, Transform, Load (ETL) system that allows us to fill our data warehouse with information from multiple different sources (data lake, spreadsheets, external services, etc.).

In actual development, you use YAML configuration in connection with the Serverless framework and a special Step Functions plugin, making it possible to connect our framework with an AWS service. Yet, there is still a challenge when it comes to local development.

Generally‌, serverless services allow us to deploy things faster and more easily at the cost of much harder local development, especially when you compare that to typical containerized development. Since Step Functions is a serverless service, quite dependent on other AWS services, it also shares development challenges.

So is it possible to develop Step Functions locally? Yes, it is! 

Local development with Step Functions – tools

In order to have a better Development Experience, we had to improve our feedback loop. This is what made us investigate local development possibilities. The idea was simple. “Let’s create an environment that allows us to test things locally without the need to deploy anything”. It doesn’t mean you want a full offline experience. You rather want to test and design your workflows before you do the cloud integrations tests on real AWS services.

The stack to be built is based on three main tools:

• Serverless Framework – the most popular framework for serverless applications with huge community backing and a lot of plugins that support local development
• Step Functions Local – the official AWS Step Functions engine available to be run locally as a Java application or Docker container.
• AWS Toolkit – an IDE extension improving the overall Development Experience.

So how does the stack work? Let’s divide it into 2 stages:

• Step Functions design experience,
• Step Functions execution experience.

I assume you are familiar with how Step Functions are developed using the Serverless Framework. Usually, you use the serverless Step Functions plugin to add support for that feature to our framework.

AWS Toolkit

You’ll have to create a new workflow configuration YAML file. It could look like the one below:

Then, connect it to the framework itself (serverless.yaml) so that necessary resources are created during deployment.

It is fine for simple workflows, but for more complex ones you want to actually visualize those state machines before you implement. This is where you use AWS Toolkit.

There are three main features of the AWS Toolkit for Step Functions design:

• state machine real-time graph rendering,
• useful snippets with solving the common use-cases,
• ASL validation.

You ask what is “ASL”? Amazon States Language is a special language you use to define how the step machine should work.

In this example, it is everything you put in the definition part of the yaml workflow.

So basically you want this…

to be transformed into this:

At the same time, you want to catch all design errors as soon as possible by using the validation feature:

Make use of snippets to develop steps easier:

In order to use all of those, you need to separate the ASL definition from the step function code. To do so, divide the workflow.yml you had previously…

… into two separate files, workflow configuration:

and workflow definition (ASL):

By doing this simple change, you should gain access to all debugging / designing / features of the AWS Toolkit for Step Functions.

Step Functions Local

You’ve covered the design phase and now you need to execute the workflow locally. To that end, you need to run the local version of the Step Functions engine. In order to do so, use the Step Functions Local serverless plugin.

What it does is that it takes all of the Step Functions you defined in your serverless.yaml, runs the AWS Step Function Local Java engine, and creates a state machine for each of the workflows you have.

The plugin is very simple and requires very little configuration.

It all comes down to the custom section of serverless.yaml.

The plugin supports a new section called stepFunctionsLocal. 

You have to define a fake local accountID and region. With that, the most crucial part remaining is TaskResourceMapping.

This part is tricky and requires you to understand how the AWS Step Functions Local works in full.

An AWS app doesn’t run the code you create. Instead, it uses multiple different SDK’s to execute a service. For example, when you define a task that should run some lambda and the state machine reaches that step, it will call the Lambda SDK invoke method (basically a HTTP call). Then, it will wait for a result in order to move to the next step.

The invoke method HTTP call has to be handled by serverless framework and special serverless step functions local plugin. Basically it requires you to define what function is going to be executed for a specific ARN (unique identifier of a AWS resource). You can see it under TaskResourceMapping:

This also means that a step name must be unique in a whole application.

Since you’re going to invoke lambdas, you also need to support that in your framework. In order to do so, use the Serverless Offline plugin and configure it to run with lambda port listening set to 4000. This is the default port Lambda SDK sends requests to when using the invoke method.

serverless offline start -s dev --httpPort 1337 --lambdaPort 4000

The first time you run this command, it will download a AWS Step Functions local engine and some supporting libraries. It might take a while since it contains all of the AWS SDKs to support over 200 possible invoke operations for these services. The download takes place only once. In order to improve that, you can also use Docker containers.

Then, step machines based on our workflows will be created.

And after that, you’re finally ready to play with it by running:

aws stepfunctions --endpoint http://localhost:8083 start-execution --state-machine-arn arn:aws:states:us-east-1:101010101010:stateMachine:ExampleWorkflow

Or if you use our starter:

npm run start-workflow --workflow=ExampleWorkflow

First, you’ll get an acknowledgement message form the Step Function engine:

At the same time, the serverless framework will get the information about state machine execution.

Step Functions Local – worthwhile or not?

As you can see, developing and testing Step Functions locally. However, there a couple things you need to remember about:

• Not all of AWS Services are easily mockable.
• Queues (SQS), storages (RDS, DynamoDB, S3, ElasticSearch and others) and message broking services could be mocked locally, but AWS-specific services could not.This would require a hybrid approach for testing and in some cases connecting to the real AWS services.
• For most of the workflows where you build a data pipeline (a kind of an ETL) or a simple workflow, local development is surely possible so run Step Functions Local as much as you want!

If you want to try Step Functions Local, feel free to check our Serverless starter with local Step Functions pre-configured.