Introduction
A number of workflow structures recur across AWS Step Functions projects. These recurring structures are known as design patterns. A design pattern is a reusable solution to a problem that occurs repeatedly. This guide describes the patterns that are most commonly applied and explains when each is appropriate. Readers who are new to the service should first read the guide on what AWS Step Functions is.
What is a design pattern?
In the context of Step Functions, a pattern is a proven arrangement of states that addresses a common coordination need. Rather than each workflow being designed from the beginning, an established pattern is selected and adapted. This makes workflows easier to understand, and it ensures that failure and scale are handled in a known way.
Why patterns are used
The problems that arise when steps are coordinated, such as running work in parallel, making decisions, recovering from failure, and undoing partial progress, occur again and again. Established patterns provide solutions that have already been tested. Their use reduces the risk of errors and shortens the time that is required to build a reliable workflow.
The common patterns
Sequential pipeline
A series of steps is performed in order, and the output of one step becomes the input of the next. This is the simplest pattern and suits a process whose steps have a fixed sequence.
Parallel fan-out
A Parallel state runs a fixed set of branches at the same time and waits for all of them to complete. This suits independent tasks that can be performed together, such as calling several services at once.
Dynamic parallelism with Map
A Map state runs the same steps for every item in a list. The amount of parallel work varies with the size of the input, which suits processing each record in a batch.
Branching with Choice
A Choice state selects the next state based on the data, which allows different paths to be taken according to a condition.
Retry and catch
A state defines retry rules that repeat a failed step, often with an increasing delay known as backoff, and catch rules that direct the workflow to an alternative path when an error is not recovered.
Wait for callback
A workflow pauses at a task and issues a task token, which is a unique identifier. The workflow resumes only when an external system returns that token. This suits human approval or waiting on a long external process.
Saga
The saga pattern coordinates steps that each change data in a separate service, and it defines a compensating action that reverses a completed step if a later step fails. This keeps the system consistent without a distributed transaction.
Pattern diagram
Advantages
- Reliability. Failure is handled in a proven, consistent way.
- Clarity. A recognized pattern is easier for others to understand.
- Scale. Parallel patterns process large inputs efficiently.
- Consistency. The saga pattern keeps data consistent across services.
Disadvantages
- Overhead for simple work. A pattern can be more than a trivial process requires.
- Design effort. Compensating actions and error paths must be designed carefully.
- Cost. Parallel and long-running patterns increase the number of state transitions.
Common use cases
- Order processing, in which a saga undoes a reservation if payment fails.
- Batch processing, in which a Map state handles each record.
- Approval workflows, in which a wait-for-callback task pauses for a human decision.
- Aggregation, in which a Parallel state gathers data from several sources at once.
Best practices
- The simplest pattern that solves the problem should be chosen.
- Retry and catch rules should be defined so that transient failures are recovered automatically.
- A compensating action should be provided for every step that changes external data when a saga is used.
- A Map state should be used in place of a fixed set of parallel branches when the number of items varies.
Common mistakes
- A complex pattern is applied to a process that a simple sequence would handle.
- Compensating actions are omitted, so a partial failure leaves the system inconsistent.
- Error handling is not defined, so a single failed step ends the entire execution.
- The cost of many state transitions in a large parallel workflow is not considered.
Related AWS services
- AWS Lambda performs the work within most patterns.
- Amazon DynamoDB and other services hold the data that the saga pattern keeps consistent.
- Amazon SQS can receive a task token and return it to resume a paused workflow.
Frequently Asked Questions
- What is a design pattern?
- A design pattern is a reusable solution to a problem that occurs repeatedly. In Step Functions, a pattern is a proven way of arranging states to handle a common coordination need.
- How is parallel work performed in Step Functions?
- A Parallel state runs a fixed set of branches at the same time. A Map state runs the same steps for every item in a list, which allows the amount of parallel work to vary with the input.
- What is the wait-for-callback pattern?
- A workflow pauses at a task and issues a task token, and it resumes only when an external system returns that token. This suits human approval or waiting on a long external process.
- What is the saga pattern?
- The saga pattern coordinates steps that each change data in a separate service and defines a compensating action to undo each step if a later step fails, which keeps the system consistent without a distributed transaction.
- How are failures handled within a workflow?
- Each state can define retry rules that repeat a failed step, often with an increasing delay known as backoff, and catch rules that direct the workflow to an alternative path when an error is not recovered.
This article is the summary. The book is the full, continuously updated reference: workflow patterns, callback and saga designs, error handling, and production orchestration.
View the book