Introduction
AWS Step Functions is a serverless orchestration service. Orchestration is the coordination of several separate tasks into a single, reliable process. A workflow is the ordered sequence of those tasks. This guide explains what Step Functions is, the problem it was created to solve, how a workflow is defined and executed, and the situations in which it is appropriate.
What is it?
Step Functions coordinates multiple AWS services and functions into a workflow that is defined as a state machine. A state machine is a description of a workflow as a set of states and the transitions between them. A state is a single step in the workflow, for example a unit of work to perform or a decision to make. A transition is the movement from one state to the next. The workflow is written in the Amazon States Language, a format based on JavaScript Object Notation, known as JSON, which is a text format for structured data.
Why does it exist?
A business process is often composed of many steps that must be run in order, with decisions between them, retries when a step fails, and a record of what has happened. If this coordination is written by hand inside application code, the result is frequently complex and difficult to maintain. Step Functions was created so that the coordination is described declaratively, separate from the logic of each step. Retries, error handling, and the passing of data between steps are then managed by the service.
How it works
An execution begins with an input and moves from state to state until it reaches an end state. Several state types are available. A Task state performs work, most often by invoking a Lambda function or by calling an AWS service directly. A Choice state selects the next state based on the data. A Parallel state runs several branches at the same time. A Map state runs the same steps for each item in a list. A Wait state pauses for a period, and the Succeed and Fail states end the execution.
Each state can define retry and catch rules so that failures are handled without additional code. Data is passed from one state to the next, and portions of that data can be selected using JSONPath, a syntax for addressing parts of a JSON document. Two workflow types are offered. Standard workflows are durable and auditable, and Express workflows are optimized for high volume and short duration.
Architecture diagram
Advantages
- Declarative workflows. The sequence of steps is described separately from the logic of each step.
- Built-in error handling. Retries and catches are configured rather than coded.
- Managed state. Data is carried between steps by the service.
- Direct integration. Many AWS services can be called without connecting code.
- Auditability. Standard workflows record the full history of each execution.
Disadvantages
- Cost per transition. Standard workflows are charged for each state transition, which can add up in large workflows.
- Learning curve. The Amazon States Language must be learned.
- Added latency. Coordination introduces a small delay between steps.
- Unnecessary for simple tasks. A single step does not benefit from a workflow.
Common use cases
- Order and payment processing, in which several steps must complete in order.
- Data processing pipelines, in which data is extracted, transformed, and loaded.
- Machine learning pipelines, in which training and deployment steps are coordinated.
- Human approval workflows, in which an execution waits for a decision before continuing.
Best practices
- The work of each step should be kept inside a Lambda function or a direct service call, and the state machine should be reserved for coordination.
- Retry and catch rules should be defined so that transient failures are recovered automatically.
- A Map state should be used to process the items of a list in parallel.
- Express workflows should be chosen for high-volume, short-lived work, and Standard workflows for long-running or auditable processes.
Common mistakes
- A workflow is used for a single, trivial step, where a direct call would be simpler.
- The cost of state transitions is not considered in a large Standard workflow.
- Business logic is placed inside the state machine definition rather than inside the tasks.
- Error handling is omitted, so a single failed step ends the entire execution.
Related AWS services
- AWS Lambda performs the work of most Task states.
- Amazon EventBridge can start a workflow in response to an event.
- Amazon SQS and Amazon SNS can send or receive messages within a workflow.
- Amazon DynamoDB can be read from or written to directly from a Task state.
Frequently Asked Questions
- What is a state machine?
- A state machine is a definition of a workflow as a set of states and the transitions between them. In Step Functions, it describes the sequence of steps that an execution follows, including the work performed, the decisions made, and the handling of errors.
- What is the Amazon States Language?
- The Amazon States Language is a JSON-based format in which a Step Functions state machine is defined. It describes the states, their order, and their error handling.
- What is the difference between Standard and Express workflows?
- Standard workflows are durable, may run for up to one year, and record the full execution history. Express workflows are optimized for high volume and short duration and are billed differently. The choice depends on duration, volume, and auditing needs.
- How is AWS Step Functions priced?
- Standard workflows are charged per state transition. Express workflows are charged by the number of executions, their duration, and the memory used.
- Can Step Functions call services other than AWS Lambda?
- Yes. In addition to invoking Lambda functions, Step Functions can call many AWS services directly from a Task state.
This article is the summary. The book is the full, continuously updated reference: state machine design, the Amazon States Language, error handling, and production orchestration patterns.
View the book