Sqs DG
Sqs DG
Developer Guide
Amazon Simple Queue Service Developer Guide
Amazon's trademarks and trade dress may not be used in connection with any product or service that is not
Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or
discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may
or may not be affiliated with, connected to, or sponsored by Amazon.
Amazon Simple Queue Service Developer Guide
Table of Contents
What is Amazon SQS? ........................................................................................................................ 1
Benefits of using Amazon SQS .................................................................................................... 1
Differences between Amazon SQS, Amazon MQ, and Amazon SNS ................................................... 1
Queue types .............................................................................................................................. 2
Common tasks for getting started with Amazon SQS ..................................................................... 3
Pricing for Amazon SQS .............................................................................................................. 3
Setting up ......................................................................................................................................... 4
Step 1: Create an AWS account .................................................................................................... 4
Step 2: Create an IAM user .......................................................................................................... 4
Step 3: Get your access key ID and secret access key ...................................................................... 5
Step 4: Get ready to use the example code ................................................................................... 6
Next steps ................................................................................................................................. 6
Getting started .................................................................................................................................. 7
Prerequisites .............................................................................................................................. 7
Step 1: Create a queue ............................................................................................................... 7
Step 2: Send a message .............................................................................................................. 7
Step 3: Receive and delete your message ...................................................................................... 8
Step 4: Delete your queue ........................................................................................................... 9
Next steps ................................................................................................................................. 9
Configuring Amazon SQS .................................................................................................................. 10
Understanding the Amazon SQS console ..................................................................................... 10
Creating a queue ...................................................................................................................... 11
Editing a queue ........................................................................................................................ 12
Configuring queue parameters ................................................................................................... 13
Configuring access policy .......................................................................................................... 14
Configuring SSE-SQS for a queue ............................................................................................... 14
Configuring SSE-KMS for a queue .............................................................................................. 15
Configuring a dead-letter queue ................................................................................................ 16
Configuring a dead-letter queue redrive ...................................................................................... 16
Configuring tags for a queue ..................................................................................................... 17
Subscribing a queue to a topic ................................................................................................... 17
Configuring a Lambda trigger .................................................................................................... 18
Prerequisites .................................................................................................................... 18
Managing queues ............................................................................................................................. 20
Sending messages .................................................................................................................... 20
Message attributes ................................................................................................................... 21
Receiving and deleting a message .............................................................................................. 22
Purging a queue ....................................................................................................................... 22
Deleting a queue ...................................................................................................................... 23
Confirming that a queue is empty .............................................................................................. 23
How Amazon SQS works ................................................................................................................... 25
Basic architecture ..................................................................................................................... 25
Distributed queues ........................................................................................................... 25
Message lifecycle .............................................................................................................. 26
Standard queues ...................................................................................................................... 27
Message ordering ............................................................................................................. 28
At-least-once delivery ....................................................................................................... 28
FIFO queues ............................................................................................................................ 28
Message ordering ............................................................................................................. 29
Key terms ........................................................................................................................ 29
FIFO delivery logic ............................................................................................................ 30
Exactly-once processing .................................................................................................... 31
Moving from a standard queue to a FIFO queue ................................................................... 31
High throughput for FIFO queues ....................................................................................... 32
iii
Amazon Simple Queue Service Developer Guide
Compatibility ................................................................................................................... 37
Queue and message identifiers .................................................................................................. 37
Identifiers for Standard and FIFO queues ............................................................................ 37
Additional identifiers for FIFO queues ................................................................................. 38
Message metadata .................................................................................................................... 39
Message attributes ........................................................................................................... 39
Message system attributes ................................................................................................ 41
Resources required to process messages ...................................................................................... 42
List queue pagination ............................................................................................................... 42
Cost allocation tags .................................................................................................................. 42
Short and long polling .............................................................................................................. 43
Consuming messages using short polling ............................................................................ 43
Consuming messages using long polling ............................................................................. 44
Differences between long and short polling ......................................................................... 44
Dead-letter queues ................................................................................................................... 45
How do dead-letter queues work? ...................................................................................... 45
What are the benefits of dead-letter queues? ...................................................................... 46
How do different queue types handle message failure? ......................................................... 46
When should I use a dead-letter queue? ............................................................................. 47
Moving messages out of a dead-letter queue ....................................................................... 47
Troubleshooting dead-letter queues ................................................................................... 48
Visibility timeout ...................................................................................................................... 49
Inflight messages ............................................................................................................. 50
Setting the visibility timeout ............................................................................................. 50
Changing the visibility timeout for a message ...................................................................... 51
Terminating the visibility timeout for a message .................................................................. 51
Delay queues ........................................................................................................................... 51
Temporary queues .................................................................................................................... 52
Virtual queues ................................................................................................................. 53
Request-response messaging pattern (virtual queues) ........................................................... 53
Example scenario: Processing a login request ....................................................................... 54
Cleaning up queues .......................................................................................................... 55
Message timers ........................................................................................................................ 56
Best practices .................................................................................................................................. 57
Recommendations for standard and FIFO queues ......................................................................... 57
Working with messages ..................................................................................................... 57
Reducing costs ................................................................................................................. 59
Moving from a Standard queue to a FIFO queue .................................................................. 60
Additional recommendations for FIFO queues .............................................................................. 60
Using the message deduplication ID ................................................................................... 60
Using the message group ID .............................................................................................. 62
Using the receive request attempt ID .................................................................................. 63
Java SDK examples ........................................................................................................................... 64
Using server-side encryption ...................................................................................................... 64
Adding SSE to an existing queue ........................................................................................ 64
Disabling SSE for a queue ................................................................................................. 65
Creating a queue with SSE ................................................................................................ 65
Retrieving SSE attributes ................................................................................................... 65
Configuring tags ...................................................................................................................... 66
Listing tags ...................................................................................................................... 66
Adding or updating tags ................................................................................................... 66
Removing tags ................................................................................................................. 67
Sending message attributes ....................................................................................................... 67
Defining attributes ........................................................................................................... 67
Sending a message with attributes ..................................................................................... 68
Managing large messages .......................................................................................................... 69
Prerequisites .................................................................................................................... 69
iv
Amazon Simple Queue Service Developer Guide
v
Amazon Simple Queue Service Developer Guide
vi
Amazon Simple Queue Service Developer Guide
Benefits of using Amazon SQS
Amazon SQS supports both standard (p. 27) and FIFO queues (p. 28). For more information, see
Queue types (p. 2).
Topics
• Benefits of using Amazon SQS (p. 1)
• Differences between Amazon SQS, Amazon MQ, and Amazon SNS (p. 1)
• Queue types (p. 2)
• Common tasks for getting started with Amazon SQS (p. 3)
• Pricing for Amazon SQS (p. 3)
Server-side encryption (SSE) (p. 109) lets you transmit sensitive data by protecting the contents of
messages in queues using keys managed in AWS Key Management Service (AWS KMS).
• Durability – For the safety of your messages, Amazon SQS stores them on multiple servers. Standard
queues support at-least-once message delivery (p. 28), and FIFO queues support exactly-once
message processing (p. 31).
• Availability – Amazon SQS uses redundant infrastructure (p. 25) to provide highly-concurrent
access to messages and high availability for producing and consuming messages.
• Scalability – Amazon SQS can process each buffered request (p. 169) independently, scaling
transparently to handle any load increases or spikes without any provisioning instructions.
• Reliability – Amazon SQS locks your messages during processing, so that multiple producers can send
and multiple consumers can receive messages at the same time.
• Customization – Your queues don't have to be exactly alike—for example, you can set a default delay
on a queue (p. 51). You can store the contents of messages larger than 256 KB using Amazon
Simple Storage Service (Amazon S3) (p. 69) or Amazon DynamoDB, with Amazon SQS holding a
pointer to the Amazon S3 object, or you can split a large message into smaller messages.
1
Amazon Simple Queue Service Developer Guide
Queue types
Amazon MQ is a managed message broker service that provides compatibility with many popular
message brokers. We recommend Amazon MQ for migrating applications from existing message brokers
that rely on compatibility with APIs such as JMS or protocols such as AMQP, MQTT, OpenWire, and
STOMP.
Queue types
The following table describes the capabilities of standard queues and FIFO queues.
Unlimited Throughput – Standard queues High Throughput – If you use batching (p. 168),
support a nearly unlimited number of API calls FIFO queues support up to 3,000 messages per
per second, per API action (SendMessage, second, per API method (SendMessageBatch,
ReceiveMessage, or DeleteMessage). ReceiveMessage, or DeleteMessageBatch).
The 3000 messages per second represent 300
At-Least-Once Delivery – A message is delivered API calls, each with a batch of 10 messages.
at least once, but occasionally more than one copy To request a quota increase, submit a support
of a message is delivered. request. Without batching, FIFO queues
support up to 300 API calls per second, per API
Best-Effort Ordering – Occasionally, messages are method (SendMessage, ReceiveMessage, or
delivered in an order different from which they DeleteMessage).
were sent.
Exactly-Once Processing – A message is delivered
once and remains available until a consumer
processes and deletes it. Duplicates aren't
introduced into the queue.
Send data between applications when the Send data between applications when the order
throughput is important, for example: of events is important, for example:
• Decouple live user requests from intensive • Make sure that user-entered commands are run
background work: let users upload media while in the right order.
resizing or encoding it. • Display the correct product price by sending
• Allocate tasks to multiple worker nodes: price modifications in the right order.
process a high number of credit card validation • Prevent a student from enrolling in a course
requests. before registering for an account.
• Batch messages for future processing: schedule
multiple entries to be added to a database.
2
Amazon Simple Queue Service Developer Guide
Common tasks for getting started with Amazon SQS
3
Amazon Simple Queue Service Developer Guide
Step 1: Create an AWS account
Before you can use Amazon SQS for the first time, you must complete the following steps.
To avoid using your AWS account root user for Amazon SQS actions, it is a best practice to create an IAM
user for each person who needs administrative access to Amazon SQS.
1. Open https://portal.aws.amazon.com/billing/signup.
2. Follow the online instructions.
Part of the sign-up procedure involves receiving a phone call and entering a verification code on the
phone keypad.
1. Sign in to the IAM console as the account owner by choosing Root user and entering your AWS
account email address. On the next page, enter your password.
Note
We strongly recommend that you adhere to the best practice of using the Administrator
IAM user that follows and securely lock away the root user credentials. Sign in as the root
user only to perform a few account and service management tasks.
2. In the navigation pane, choose Users and then choose Add user.
3. For User name, enter Administrator.
4. Select the check box next to AWS Management Console access. Then select Custom password, and
then enter your new password in the text box.
5. (Optional) By default, AWS requires the new user to create a new password when first signing in. You
can clear the check box next to User must create a new password at next sign-in to allow the new
user to reset their password after they sign in.
4
Amazon Simple Queue Service Developer Guide
Step 3: Get your access key ID and secret access key
You can use this same process to create more groups and users and to give your users access to your AWS
account resources. To learn about using policies that restrict user permissions to specific AWS resources,
see Access management and Example policies.
Access keys consist of an access key ID and secret access key, which are used to sign programmatic
requests that you make to AWS. If you don't have access keys, you can create them from the AWS
Management Console. As a best practice, do not use the AWS account root user access keys for any task
where it's not required. Instead, create a new administrator IAM user with access keys for yourself.
The only time that you can view or download the secret access key is when you create the keys. You
cannot recover them later. However, you can create new access keys at any time. You must also have
permissions to perform the required IAM actions. For more information, see Permissions required to
access IAM resources in the IAM User Guide.
1. Sign in to the AWS Management Console and open the IAM console at https://
console.aws.amazon.com/iam/.
2. In the navigation pane, choose Users.
3. Choose the name of the user whose access keys you want to create, and then choose the Security
credentials tab.
4. In the Access keys section, choose Create access key.
5
Amazon Simple Queue Service Developer Guide
Step 4: Get ready to use the example code
5. To view the new access key pair, choose Show. You will not have access to the secret access key again
after this dialog box closes. Your credentials will look something like this:
Keep the keys confidential in order to protect your AWS account and never email them. Do not share
them outside your organization, even if an inquiry appears to come from AWS or Amazon.com. No
one who legitimately represents Amazon will ever ask you for your secret key.
7. After you download the .csv file, choose Close. When you create an access key, the key pair is active
by default, and you can use the pair right away.
Related topics
You can develop AWS applications in other programming languages, such as Go, JavaScript, Python and
Ruby. For more information, see Tools for developing and managing applications on AWS.
Note
You can explore Amazon SQS without writing code with tools such as the AWS Command Line
Interface (AWS CLI) or Windows PowerShell. You can find AWS CLI examples in the Amazon SQS
section of the AWS CLI Command Reference. You can find Windows PowerShell examples in the
Amazon Simple Queue Service section of the AWS Tools for PowerShell Cmdlet Reference.
Next steps
You are now ready for Getting started (p. 7) with managing Amazon SQS queues and messages using
the AWS Management Console.
6
Amazon Simple Queue Service Developer Guide
Prerequisites
Topics
• Prerequisites (p. 7)
• Step 1: Create a queue (p. 7)
• Step 2: Send a message (p. 7)
• Step 3: Receive and delete your message (p. 8)
• Step 4: Delete your queue (p. 9)
• Next steps (p. 9)
Prerequisites
Before you begin, complete the steps in Setting up Amazon SQS (p. 4).
You can't change the queue type after you create a queue.
5. Enter a Name for your queue. The name of a FIFO queue must end with the .fifo suffix.
6. To create your queue with the default parameters, scroll to the bottom and choose Create Queue.
Amazon SQS creates the queue and displays the queue's Details page.
Amazon SQS propagates information about the new queue across the system. Because Amazon
SQS is a distributed system, you may experience a slight delay before the queue is displayed on the
Queues page.
1. From the left navigation pane, choose Queues. From the queue list, select the queue that you
created.
2. From Actions, choose Send and receive messages.
7
Amazon Simple Queue Service Developer Guide
Step 3: Receive and delete your message
Your message is sent and the console displays a success message. Choose View details to display
information about the sent message.
Amazon SQS begins to poll servers to find messages in the queue. The progress bar on the right side
of the Receive messages section displays the polling duration.
The Messages section displays a list of the received messages. For each message, the list displays the
message ID, sent date, size, and receive count.
4. To delete messages, select the messages that you want to delete and then choose Delete.
5. In the Delete Messages dialog box, choose Delete.
8
Amazon Simple Queue Service Developer Guide
Step 4: Delete your queue
Next steps
Now that you've created a queue and learned how to send, receive, and delete messages and how to
delete a queue, you might want to try the following:
9
Amazon Simple Queue Service Developer Guide
Understanding the Amazon SQS console
Topics
• Understanding the Amazon SQS console (p. 10)
• Creating an Amazon SQS queue (console) (p. 11)
• Editing an Amazon SQS queue (console) (p. 12)
• Configuring queue parameters (console) (p. 13)
• Configuring access policy (console) (p. 14)
• Configuring server-side encryption (SSE) for a queue using SQS-owned encryption keys
(console) (p. 14)
• Configuring server-side encryption (SSE) for a queue (console) (p. 15)
• Configuring a dead-letter queue (console) (p. 16)
• Configuring a dead-letter queue redrive (console) (p. 16)
• Configuring cost allocation tags for an Amazon SQS queue (console) (p. 17)
• Subscribing an Amazon SQS queue to an Amazon SNS topic (console) (p. 17)
• Configuring a queue to trigger an AWS Lambda function (console) (p. 18)
The entry for each queue shows the queue type and other information about the queue. The Type
column helps you distinguish standard queues from First-In-First Out (FIFO) queues at a glance.
From the Queues page, there are two ways to perform actions on a queue. You can choose the option
next to the queue name and then choose the action you want to perform on the queue.
10
Amazon Simple Queue Service Developer Guide
Creating a queue
You can also choose the queue name, which open the Details page for the queue. The Details page
includes the same actions as the Queues page. In addition, you can choose one of the tabs below the
Details section to view additional configuration details and actions.
a. For Visibility timeout , enter the duration and units. The range is from 0 seconds to 12 hours.
The default value is 30 seconds.
b. For Message retention period, enter the duration and units. The range is from 1 minute to 14
days. The default value is 4 days.
11
Amazon Simple Queue Service Developer Guide
Editing a queue
c. For Delivery delay, enter the duration and units. The range is from 0 seconds to 15 minutes.
The default value is 0 seconds.
d. For Maximum message size, enter a value. The range is from 1 KB to 256 KB. The default value
is 256 KB.
e. For Receive message wait time, enter a value. The range is from 0 to 20 seconds. The default
value is 0 seconds, which sets short polling (p. 43). Any non-zero value sets long polling.
f. For a FIFO queue, choose Enable content-based deduplication to enable content-based
deduplication. The default setting is disabled.
g. (Optional) For a FIFO queue, to enable higher throughput for sending and receiving messages in
the queue, choose Enable high throughput FIFO.
Choosing this option changes the related options (Deduplication scope and FIFO throughput
limit) to the required settings for enabling high throughput for FIFO queues. If you change any
of the settings required for using high throughput FIFO, normal throughput is in effect for the
queue, and deduplication occurs as specified. For more information, see High throughput for
FIFO queues (p. 32) and Quotas related to messages (p. 103).
h. For Redrive allow policy, choose Allow all (the default), By queue or Deny all. When choosing
By queue, specify a list of up to 10 source queues by the Amazon Resource Name (ARN).
6. (Optional) Define an Access policy. The access policy (p. 139) defines the accounts, users, and
roles that can access the queue. The access policy also defines the actions (such as SendMessage,
ReceiveMessage, or DeleteMessage) that the users can access. The default policy allows only the
queue owner to send and receive messages.
• Choose Basic to configure who can send messages to the queue and who can receive messages
from the queue. The console creates the policy based on your choices and displays the resulting
access policy in the read-only JSON panel.
• Choose Advanced to modify the JSON access policy directly. This allows you to specify a custom
set of actions that each principal (account, user, or role) can perform.
7. (Optional) To configure encryption (p. 15) for the queue, expand Encryption.
8. (Optional) To configure a dead-letter queue (p. 16) to receive undeliverable messages, expand
Dead-letter queue.
9. (Optional) To add tags (p. 17) to the queue, expand Tags.
10. Choose Create queue. Amazon SQS creates the queue and displays the queue's Details page.
Amazon SQS propagates information about the new queue across the system. Because Amazon SQS is
a distributed system, you might experience a slight delay before the console displays the queue on the
Queues page.
After creating a queue, you can send messages (p. 20) to it, and receive and delete
messages (p. 22). You can also edit (p. 12) any of the queue configuration settings except the
queue type.
12
Amazon Simple Queue Service Developer Guide
Configuring queue parameters
• Visibility timeout – The length of time that a message received from a queue (by one consumer) won't
be visible to the other message consumers. For more information, see Visibility timeout (p. 49).
Note
Using the console to configure the visibility timeout configures the timeout value for all of the
messages in the queue. To configure the timeout for single or multiple messages, you must
use one of the AWS SDKs.
• Message retention period – The amount of time that Amazon SQS retains messages that remain in
the queue. By default, the queue retains messages for four days. You can configure a queue to retain
messages for up to 14 days. For more information, see Message retention period.
• Delivery delay – The amount of time that Amazon SQS will delay before delivering a message that is
added to the queue. For more information, see Delivery delay (p. 51).
• Maximum message size – The maximum message size for this queue. For more information, see
Maximum message size (p. 69).
• Receive message wait time – The maximum amount of time that Amazon SQS waits for messages to
become available after the queue gets a receive request. For more information, see Amazon SQS short
and long polling (p. 43).
• Enable content-based deduplication – Amazon SQS can automatically create deduplication IDs
based on the body of the message. For more information, see Amazon SQS FIFO (First-In-First-Out)
queues (p. 28).
• Enable high throughput FIFO – Use to enable high throughput for messages in the queue. Choosing
this option changes the related options (Deduplication scope and FIFO throughput limit) to the
required settings for enabling high throughput for FIFO queues. For more information, see High
throughput for FIFO queues (p. 32) and Quotas related to messages (p. 103).
• Redrive allow policy: defines which source queues can use this queue as the dead-letter queue. For
more information, see Amazon SQS dead-letter queues (p. 45).
13
Amazon Simple Queue Service Developer Guide
Configuring access policy
6. For Delivery delay, enter the duration and units. The range is 0 seconds to 15 minutes. The default
value is 0 seconds.
7. For Maximum message size, enter a value. The range is 1 KB to 256 KB. The default value is 256 KB.
8. For a standard queue, enter a value for Receive message wait time. The range is 0 to 20 seconds.
The default value is 0 seconds, which sets short polling (p. 43). Any non-zero value sets long
polling.
9. For a FIFO queue, choose Enable content-based deduplication to enable content-based
deduplication. The default setting is disabled.
10. (Optional) For a FIFO queue, to enable higher throughput for sending and receiving messages in the
queue, choose Enable high throughput FIFO.
Choosing this option changes the related options (Deduplication scope and FIFO throughput
limit) to the required settings for enabling high throughput for FIFO queues. If you change any
of the settings required for using high throughput FIFO, normal throughput is in effect for the
queue, and deduplication occurs as specified. For more information, see High throughput for FIFO
queues (p. 32) and Quotas related to messages (p. 103).
11. For Redrive allow policy, choose Allow all (the default), By queue or Deny all. When choosing By
queue, specify a list of up to 10 source queues by the Amazon Resource Name (ARN).
12. When you finish configuring the queue parameters, choose Save.
The access policy defines the accounts, users, and roles that can access the queue. The access policy also
defines the actions (such as SendMessage, ReceiveMessage, or DeleteMessage) that the users can
access. The default policy allows only the queue owner to send and receive messages.
SSE-SQS protects data at rest using 256-bit Advanced Encryption Standard (AES-256) encryption. SSE
encrypts messages as soon as Amazon SQS receives them. Amazon SQS stores messages in encrypted
form and decrypts them only when sending them to an authorized consumer.
14
Amazon Simple Queue Service Developer Guide
Configuring SSE-KMS for a queue
The KMS key that you assign to your queue must have a key policy that includes permissions for all
principals that are authorized to use the queue. For information, see Key Management (p. 111).
If you aren't the owner of the KMS key, or if you log in with an account that doesn't have
kms:ListAliases and kms:DescribeKey permissions, you won't be able to view information about
the KMS key on the Amazon SQS console. Ask the owner of the KMS key to grant you these permissions.
For more information, see Key Management (p. 111).
When you create (p. 11) or edit (p. 12) a queue, you can configure SSE-KMS.
The console displays the Description, the Account, and the KMS key ARN of the KMS key.
7. Specify the KMS key ID for the queue. For more information, see Key terms (p. 110).
15
Amazon Simple Queue Service Developer Guide
Configuring a dead-letter queue
Amazon SQS does not create the dead-letter queue automatically. You must first create the queue before
using it as a dead-letter queue..
The dead-letter queue of a FIFO queue must also be a FIFO queue. Similarly, the dead-letter queue of a
standard queue must also be a standard queue.
When you create (p. 11) or edit (p. 12) a queue, you can configure a dead-letter queue.
After you save the queue, the console displays the Details page for your queue. On the Details page,
the Dead-letter queue tab displays the Maximum Receives and Dead Letter Queue ARN in the
Dead-letter queue.
• System optimized - Redrive dead-letter queue messages at the maximum number of messages
per second.
16
Amazon Simple Queue Service Developer Guide
Configuring tags for a queue
• Custom max velocity - Redrive dead-letter queue messages with a custom maximum rate of
messages per second. The maximum allowed rate is 500 messages per second.
7. When you finish configuring the dead-letter queue redrive, choose Redrive messages.
Note
A dead-letter queue redrive task can run a maximum of 36 hours. Amazon SQS supports a
maximum of 100 active redrive tasks per account.
The redrive task resets the retention period. A new messageID and enqueueTime are
assigned to redriven messages.
8. If you want to cancel the message redrive task, on the Details page for your queue, choose Cancel
DLQ redrive. When canceling an in progress message redrive, any messages that have already been
successfully moved to their move destination queue will remain in the destination queue.
On the Details page for a queue, the Tagging tab displays the tags for the queue.
When you create (p. 11) or edit (p. 12) a queue, you can configure tags for it.
a. To add a tag, choose Add new tag, enter a Key and Value, and then choose Add new tag.
b. To update a tag, change its Key and Value.
c. To remove a tag, choose Remove next to its key-value pair.
6. When you finish configuring the tags, choose Save.
When you subscribe an Amazon SQS queue to an SNS topic, Amazon SNS uses HTTPS to forward
messages to Amazon SQS. For information about using Amazon SNS with encrypted Amazon SQS
queues, see Configure KMS permissions for AWS services (p. 112).
17
Amazon Simple Queue Service Developer Guide
Configuring a Lambda trigger
If the SNS topic isn't listed in the menu, choose Enter Amazon SNS topic ARN and then enter the
topic's Amazon Resource Name (ARN).
6. Choose Save.
7. To verify the result of the subscription, publish to the topic and then view the message that the
topic sends to the queue. For more information, see Publishing a message to an Amazon SNS topic
in the Amazon Simple Notification Service Developer Guide.
If your Amazon SQS queue and SNS topic are in different AWS accounts, the topic owner must first
confirm the subscription. For more information, see Confirm the subscription in the Amazon Simple
Notification Service Developer Guide.
For information on subscribing to a cross-region SNS topic, see Sending Amazon SNS messages to an
Amazon SQS queue or AWS Lambda function in a different Region in the Amazon Simple Notification
Service Developer Guide
A Lambda function can process items from multiple queues (using one Lambda event source for each
queue). You can use the same queue with multiple Lambda functions.
If you associate an encrypted queue with a Lambda function but Lambda doesn't poll for messages, add
the kms:Decrypt permission to your Lambda execution role.
• Your queue and the Lambda function must be in the same AWS Region.
• An encrypted queue (p. 109) that uses the default key (AWS managed KMS key for Amazon SQS)
cannot invoke a Lambda function in a different AWS account.
For information about implementing the Lambda function, see Using AWS Lambda with Amazon SQS in
the AWS Lambda Developer Guide.
Prerequisites
To configure Lambda function triggers, you must meet the following requirements:
• If you use an IAM user, your Amazon SQS role must include the following permissions:
18
Amazon Simple Queue Service Developer Guide
Prerequisites
• lambda:CreateEventSourceMapping
• lambda:ListEventSourceMappings
• lambda:ListFunctions
• The Lambda execution role must include the following permissions:
• sqs:DeleteMessage
• sqs:GetQueueAttributes
• sqs:ReceiveMessage
• If you associate an encrypted queue with a Lambda function, add the kms:Decrypt permission to the
Lambda execution role.
For more information, see Overview of managing access in Amazon SQS (p. 119).
If the list doesn't include the Lambda trigger that you need, choose Configure Lambda function
trigger. Enter the Amazon Resource Name (ARN) of the Lambda function or choose an existing
resource. Then choose Save.
6. Choose Save. The console saves the configuration and displays the Details page for the queue.
On the Details page, the Lambda triggers tab displays the Lambda function and its status. It takes
approximately 1 minute for the Lambda function to become associated with your queue.
7. To verify the results of the configuration, send a message to your queue (p. 20) and then view the
triggered Lambda function in the Lambda console.
19
Amazon Simple Queue Service Developer Guide
Sending messages
Topics
• Sending messages to a queue (console) (p. 20)
• Sending a message with attributes (console) (p. 21)
• Receiving and deleting messages (console) (p. 22)
• Purging messages from an Amazon SQS queue (console) (p. 22)
• Deleting an Amazon SQS queue (p. 23)
• Confirming that a queue is empty (p. 23)
20
Amazon Simple Queue Service Developer Guide
Message attributes
When your message is sent, the console displays a success message. Choose View details to display
information about the sent message.
7. You can modify the attribute values any time before sending the message.
8. To delete an attribute, choose Remove. To delete the first attribute, close Message attributes.
9. When you finish adding attributes to the message, choose Send message. Your message is sent and
the console displays a success message. To view information about the message attributes of the
sent message, choose View details. Choose Done to close the Message details dialog box.
21
Amazon Simple Queue Service Developer Guide
Receiving and deleting a message
Amazon SQS doesn't automatically delete a message after retrieving it for you, in case you don't
successfully receive the message (for example, if the consumers fail or you lose connectivity). To delete
a message, you must send a separate request which acknowledges that you've successfully received and
processed the message. Note that you must receive a message before you can delete it.
Note
After receiving messages from the Amazon SQS console, the console immediately sets the
messages back to visible, so that the messages can be received again.
Amazon SQS begins to poll for messages in the queue. The progress bar on the right side of the
Receive messages section displays the duration of polling.
The Messages section displays a list of the received messages. For each message, the list displays the
message ID, Sent date, Size, and Receive count.
6. To delete messages, choose the messages that you want to delete and choose Delete.
7. In the Delete Messages dialog box, choose Delete.
22
Amazon Simple Queue Service Developer Guide
Deleting a queue
Important
When you purge a queue, you can't retrieve any of the deleted messages.
All messages are purged from the queue. The console displays a confirmation banner.
You can delete a queue even when it isn't empty. To delete the messages in a queue but not the queue
itself, purge the queue (p. 22).
23
Amazon Simple Queue Service Developer Guide
Confirming that a queue is empty
When all of them show 0 values for several minutes, the queue is empty.
• ApproximateNumberOfMessagesDelayed
• ApproximateNumberOfMessagesNotVisible
• ApproximateNumberOfMessages
When all of them are 0 for several minutes, the queue is empty.
If you rely on Amazon CloudWatch metrics, make sure that you see multiple consecutive zero data
points before considering that queue empty. For more information on CloudWatch metrics, see Available
CloudWatch metrics for Amazon SQS (p. 153).
24
Amazon Simple Queue Service Developer Guide
Basic architecture
Topics
• Basic Amazon SQS architecture (p. 25)
• Amazon SQS Standard queues (p. 27)
• Amazon SQS FIFO (First-In-First-Out) queues (p. 28)
• Amazon SQS queue and message identifiers (p. 37)
• Message metadata (p. 39)
• Resources required to process Amazon SQS messages (p. 42)
• List queue pagination (p. 42)
• Amazon SQS cost allocation tags (p. 42)
• Amazon SQS short and long polling (p. 43)
• Amazon SQS dead-letter queues (p. 45)
• Amazon SQS visibility timeout (p. 49)
• Amazon SQS delay queues (p. 51)
• Amazon SQS temporary queues (p. 52)
• Amazon SQS message timers (p. 56)
Distributed queues
There are three main parts in a distributed messaging system: the components of your distributed
system, your queue (distributed on Amazon SQS servers), and the messages in the queue.
In the following scenario, your system has several producers (components that send messages to the
queue) and consumers (components that receive messages from the queue). The queue (which holds
messages A through E) redundantly stores the messages across multiple Amazon SQS servers.
25
Amazon Simple Queue Service Developer Guide
Message lifecycle
Message lifecycle
The following scenario describes the lifecycle of an Amazon SQS message in a queue, from creation to
deletion.
26
Amazon Simple Queue Service Developer Guide
Standard queues
A producer (component 1) sends message A to a queue, and the message is distributed across the
Amazon SQS servers redundantly.
When a consumer (component 2) is ready to process messages, it consumes messages from the
queue, and message A is returned. While message A is being processed, it remains in the queue and isn't
returned to subsequent receive requests for the duration of the visibility timeout (p. 49).
The consumer (component 2) deletes message A from the queue to prevent the message from being
received and processed again when the visibility timeout expires.
Note
Amazon SQS automatically deletes messages that have been in a queue for more than the
maximum message retention period. The default message retention period is 4 days. However,
you can set the message retention period to a value from 60 seconds to 1,209,600 seconds (14
days) using the SetQueueAttributes action.
27
Amazon Simple Queue Service Developer Guide
Message ordering
Standard queues support at-least-once message delivery. However, occasionally (because of the highly
distributed architecture that allows nearly unlimited throughput), more than one copy of a message
might be delivered out of order. Standard queues provide best-effort ordering which ensures that
messages are generally delivered in the same order as they're sent.
For information about how to create and configure queues using the Amazon SQS console, see
Creating an Amazon SQS queue (console) (p. 11). For Java examples, see Amazon SQS Java SDK
examples (p. 64).
You can use standard message queues in many scenarios, as long as your application can process
messages that arrive more than once and out of order, for example:
• Decouple live user requests from intensive background work – Let users upload media while resizing
or encoding it.
• Allocate tasks to multiple worker nodes – Process a high number of credit card validation requests.
• Batch messages for future processing – Schedule multiple entries to be added to a database.
For quotas related to standard queues, see Quotas related to queues (p. 102).
For best practices of working with standard queues, see Recommendations for Amazon SQS standard
and FIFO queues (p. 57).
Message ordering
A standard queue makes a best effort to preserve the order of messages, but more than one copy of a
message might be delivered out of order. If your system requires that order be preserved, we recommend
using a FIFO (First-In-First-Out) queue (p. 28) or adding sequencing information in each message so
you can reorder the messages when they're received.
At-least-once delivery
Amazon SQS stores copies of your messages on multiple servers for redundancy and high availability. On
rare occasions, one of the servers that stores a copy of a message might be unavailable when you receive
or delete a message.
If this occurs, the copy of the message isn't deleted on that unavailable server, and you might get that
message copy again when you receive messages. Design your applications to be idempotent (they should
not be affected adversely when processing the same message more than once).
FIFO (First-In-First-Out) queues are designed to enhance messaging between applications when the
order of operations and events is critical, or where duplicates can't be tolerated. Examples of situations
where you might use FIFO queues include the following:
• To make sure that user-entered commands are run in the right order.
• To display the correct product price by sending price modifications in the right order.
• To prevent a student from enrolling in a course before registering for an account.
FIFO queues also provide exactly-once processing but have a limited number of transactions per second
(TPS). For information on throughput quotas, see Quotas related to messages (p. 103).
28
Amazon Simple Queue Service Developer Guide
Message ordering
Note
The name of a FIFO queue must end with the .fifo suffix. The suffix counts towards the 80-
character queue name quota. To determine whether a queue is FIFO (p. 28), you can check
whether the queue name ends with the suffix.
Amazon SQS FIFO queues are available in all Regions where Amazon SQS is available.
For information about how to create and configure queues using the Amazon SQS console, see
Creating an Amazon SQS queue (console) (p. 11). For Java examples, see Amazon SQS Java SDK
examples (p. 64).
For best practices of working with FIFO queues, see Additional recommendations for Amazon SQS FIFO
queues (p. 60) and Recommendations for Amazon SQS standard and FIFO queues (p. 57).
For information about compatibility of clients and services with FIFO queues, see
Compatibility (p. 37).
Topics
• Message ordering (p. 29)
• Key terms (p. 29)
• FIFO delivery logic (p. 30)
• Exactly-once processing (p. 31)
• Moving from a standard queue to a FIFO queue (p. 31)
• High throughput for FIFO queues (p. 32)
• Compatibility (p. 37)
Message ordering
The FIFO queue improves upon and complements the standard queue (p. 27). The most
important features of this queue type are FIFO (First-In-First-Out) delivery (p. 30) and exactly-once
processing (p. 31):
• The order in which messages are sent and received is strictly preserved and a message is delivered
once and remains available until a consumer processes and deletes it.
• Duplicates aren't introduced into the queue.
In addition, FIFO queues support message groups that allow multiple ordered message groups within a
single queue. There is no quota to the number of message groups within a FIFO queue.
Key terms
The following key terms can help you better understand the functionality of FIFO queues. For more
information, see the Amazon Simple Queue Service API Reference.
Message deduplication ID
The token used for deduplication of sent messages. If a message with a particular message
deduplication ID is sent successfully, any messages sent with the same message deduplication ID are
accepted successfully but aren't delivered during the 5-minute deduplication interval.
Note
Message deduplication applies to an entire queue, not to individual message groups.
Amazon SQS continues to keep track of the message deduplication ID even after the
message is received and deleted.
29
Amazon Simple Queue Service Developer Guide
FIFO delivery logic
Message group ID
The tag that specifies that a message belongs to a specific message group. Messages that belong to
the same message group are always processed one by one, in a strict order relative to the message
group (however, messages that belong to different message groups might be processed out of
order).
Receive request attempt ID
The large, non-consecutive number that Amazon SQS assigns to each message.
Sending messages
If multiple messages are sent in succession to a FIFO queue, each with a distinct message
deduplication ID, Amazon SQS stores the messages and acknowledges the transmission. Then, each
message can be received and processed in the exact order in which the messages were transmitted.
In FIFO queues, messages are ordered based on message group ID. If multiple hosts (or different
threads on the same host) send messages with the same message group ID to a FIFO queue, Amazon
SQS stores the messages in the order in which they arrive for processing. To make sure that Amazon
SQS preserves the order in which messages are sent and received, each producer should use a unique
message group ID to send all its messages.
FIFO queue logic applies only per message group ID. Each message group ID represents a distinct
ordered message group within an Amazon SQS queue. For each message group ID, all messages are
sent and received in strict order. However, messages with different message group ID values might
be sent and received out of order. You must associate a message group ID with a message. If you
don't provide a message group ID, the action fails. If you require a single group of ordered messages,
provide the same message group ID for messages sent to the FIFO queue.
Receiving messages
You can't request to receive messages with a specific message group ID.
When receiving messages from a FIFO queue with multiple message group IDs, Amazon SQS first
attempts to return as many messages with the same message group ID as possible. This allows other
consumers to process messages with a different message group ID. When you receive a message
with a message group ID, no more messages for the same message group ID are returned unless you
delete the message or it becomes visible.
Note
It is possible to receive up to 10 messages in a single call using the MaxNumberOfMessages
request parameter of the ReceiveMessage action. These messages retain their FIFO
order and can have the same message group ID. Thus, if there are fewer than 10 messages
available with the same message group ID, you might receive messages from another
message group ID, in the same batch of 10 messages, but still in FIFO order.
Retrying multiple times
30
Amazon Simple Queue Service Developer Guide
Exactly-once processing
one acknowledgement before the deduplication interval expires, multiple retries neither affect the
ordering of messages nor introduce duplicates.
• If the consumer detects a failed ReceiveMessage action, it can retry as many times as necessary,
using the same receive request attempt ID. Assuming that the consumer receives at least one
acknowledgement before the visibility timeout expires, multiple retries don't affect the ordering
of messages.
• When you receive a message with a message group ID, no more messages for the same message
group ID are returned unless you delete the message or it becomes visible.
Exactly-once processing
Unlike standard queues, FIFO queues don't introduce duplicate messages. FIFO queues help you avoid
sending duplicates to a queue. If you retry the SendMessage action within the 5-minute deduplication
interval, Amazon SQS doesn't introduce any duplicates into the queue.
• Enable content-based deduplication. This instructs Amazon SQS to use a SHA-256 hash to generate
the message deduplication ID using the body of the message—but not the attributes of the message.
For more information, see the documentation on the CreateQueue, GetQueueAttributes, and
SetQueueAttributes actions in the Amazon Simple Queue Service API Reference.
• Explicitly provide the message deduplication ID (or view the sequence number) for the message.
For more information, see the documentation on the SendMessage, SendMessageBatch, and
ReceiveMessage actions in the Amazon Simple Queue Service API Reference.
To make sure that your application correctly works with a FIFO queue, use the following checklist:
• If you use batching (p. 168), FIFO queues support up to 3,000 messages per second, per API method
(SendMessageBatch, ReceiveMessage, or DeleteMessageBatch). The 3000 messages per second
represent 300 API calls, each with a batch of 10 messages. To request a quota increase, submit a
support request. Without batching, FIFO queues support up to 300 API calls per second, per API
method (SendMessage, ReceiveMessage, or DeleteMessage).
• FIFO queues don't support per-message delays, only per-queue delays. If your application sets the
same value of the DelaySeconds parameter on each message, you must modify your application to
remove the per-message delay and set DelaySeconds on the entire queue instead.
• Every message sent to a FIFO queue requires a message group ID. If you don't need multiple ordered
message groups, specify the same message group ID for all your messages.
• Before sending messages to a FIFO queue, confirm the following:
• If your application can send messages with identical message bodies, you can modify your
application to provide a unique message deduplication ID for each sent message.
• If your application sends messages with unique message bodies, you can enable content-based
deduplication.
31
Amazon Simple Queue Service Developer Guide
High throughput for FIFO queues
• You don't have to make any code changes to your consumer. However, if it takes a long time to process
messages and your visibility timeout is set to a high value, consider adding a receive request attempt
ID to each ReceiveMessage action. This allows you to retry receive attempts in case of networking
failures and prevents queues from pausing due to failed receive attempts.
For more information, see the Amazon Simple Queue Service API Reference.
You can enable high throughput for any new or existing FIFO queue. The feature includes three new
options when you create and edit FIFO queues:
• Enable high throughput FIFO – Makes higher throughput available for messages in the current FIFO
queue.
• Deduplication scope – Specifies whether deduplication occurs at the queue or message group level.
• FIFO throughput limit – Specifies whether the throughput quota on messages in the FIFO queue is set
at the queue or message group level.
Enabling high throughput for FIFO queues sets the related options as follows:
• Deduplication scope is set to Message group, the required setting for using high throughput for
FIFO queues.
• FIFO throughput limit is set to Per message group ID, the required setting for using high
throughput for FIFO queues.
If you change any of the settings required for using high throughput for FIFO queues, normal
throughput is in effect for the queue, and deduplication occurs as specified.
3. Continue specifying all options for the queue. When you finish, choose Create queue or Save.
After creating or editing the FIFO queue, you can send messages (p. 20) to it and receive and delete
messages (p. 22), all at a higher TPS.
For FIFO queues, Amazon SQS modifies the number of partitions in a queue in the following situations:
32
Amazon Simple Queue Service Developer Guide
High throughput for FIFO queues
• If the current request rate approaches or exceeds what the existing partitions can support, additional
partitions are allocated until the queue reaches the regional quota. For information on quotas, see
Quotas related to messages (p. 103).
• If the current partitions have low utilization, the number of partitions may be reduced.
Partition management occurs automatically in the background and is transparent to your applications.
Your queue and messages are available at all times.
The following diagram shows a queue that spans multiple partitions. The queue’s message group ID is
based on item number. Amazon SQS uses its hash function to determine where to store a new item; in
this case, it's based on the hash value of the string item0. Note that the items are stored in the same
order in which they are added to the queue. Each item's location is determined by the hash value of its
message group ID.
33
Amazon Simple Queue Service Developer Guide
High throughput for FIFO queues
34
Amazon Simple Queue Service Developer Guide
High throughput for FIFO queues
Note
Amazon SQS is optimized for uniform distribution of items across a FIFO queue's partitions,
regardless of the number of partitions. AWS recommends that you use message group IDs that
can have a large number of distinct values.
When using batch APIs, each message is routed based on the process described in Distributing data by
message group IDs (p. 33). Messages that are routed to the same partition are grouped and processed
in a single transaction.
To optimize partition utilization for the SendMessageBatch API, AWS recommends batching messages
with the same message group IDs when possible.
In the following example, a batch of messages with various message group IDs is sent. The batch is split
into three groups, each of which counts against the quota for the partition.
35
Amazon Simple Queue Service Developer Guide
High throughput for FIFO queues
36
Amazon Simple Queue Service Developer Guide
Compatibility
Note
Amazon SQS only guarantees that messages with the same message group ID are grouped
within a batch request. Depending on the output of the internal hash function and the number
of partitions, messages with different message group IDs might be grouped. Since the hash
function or number of partitions can change at any time, messages that are grouped at one
point may not be grouped later.
Compatibility
Clients
The Amazon SQS Buffered Asynchronous Client doesn't currently support FIFO queues.
Services
If your application uses multiple AWS services, or a mix of AWS and external services, it is important
to understand which service functionality doesn't support FIFO queues.
Some AWS or external services that send notifications to Amazon SQS might not be compatible with
FIFO queues, despite allowing you to set a FIFO queue as a target.
The following features of AWS services aren't currently compatible with FIFO queues:
• Amazon S3 Event Notifications
• Auto Scaling Lifecycle Hooks
• AWS IoT Rule Actions
• AWS Lambda Dead-Letter Queues
For information about compatibility of other services with FIFO queues, see your service
documentation.
Topics
• Identifiers for Amazon SQS Standard and FIFO queues (p. 37)
• Additional identifiers for Amazon SQS FIFO queues (p. 38)
37
Amazon Simple Queue Service Developer Guide
Additional identifiers for FIFO queues
The name of a FIFO queue must end with the .fifo suffix. The suffix counts towards the 80-character
queue name quota. To determine whether a queue is FIFO (p. 28), you can check whether the queue
name ends with the suffix.
The following is the queue URL for a queue named MyQueue owned by a user with the AWS account
number 123456789012.
https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue
You can retrieve the URL of a queue programmatically by listing your queues and parsing the string that
follows the account number. For more information, see ListQueues.
Message ID
Each message receives a system-assigned message ID that Amazon SQS returns to you in the
SendMessage response. This identifier is useful for identifying messages. (However, to delete a message
you need the message's receipt handle.) The maximum length of a message ID is 100 characters.
Receipt handle
Every time you receive a message from a queue, you receive a receipt handle for that message. This
handle is associated with the action of receiving the message, not with the message itself. To delete
the message or to change the message visibility, you must provide the receipt handle (not the message
ID). Thus, you must always receive a message before you can delete it (you can't put a message into the
queue and then recall it). The maximum length of a receipt handle is 1,024 characters.
Important
If you receive a message more than once, each time you receive it, you get a different receipt
handle. You must provide the most recently received receipt handle when you request to delete
the message (otherwise, the message might not be deleted).
MbZj6wDWli+JvwwJaBV+3dcjk2YW2vA3+STFFljTM8tJJg6HRG6PYSasuWXPJB+Cw
Lj1FjgXUv1uSj1gUPAWV66FU/WeR4mq2OKpEGYWbnLmpRCJVAyeMjeU5ZBdtcQ+QE
auMZc8ZRv37sIW2iJKq3M9MFx1YvV11A2x/KSbkJ0=
Message deduplication ID
The token used for deduplication of sent messages. If a message with a particular message deduplication
ID is sent successfully, any messages sent with the same message deduplication ID are accepted
successfully but aren't delivered during the 5-minute deduplication interval.
Message group ID
The tag that specifies that a message belongs to a specific message group. Messages that belong to the
same message group are always processed one by one, in a strict order relative to the message group
(however, messages that belong to different message groups might be processed out of order).
38
Amazon Simple Queue Service Developer Guide
Message metadata
Sequence number
The large, non-consecutive number that Amazon SQS assigns to each message.
Message metadata
You can use message attributes to attach custom metadata to Amazon SQS messages for your
applications. You can use message system attributes to store metadata for other AWS services, such as
AWS X-Ray.
Topics
• Amazon SQS message attributes (p. 39)
• Amazon SQS message system attributes (p. 41)
Topics
• Message attribute components (p. 39)
• Message attribute data types (p. 40)
• Calculating the MD5 message digest for message attributes (p. 40)
• Name – The message attribute name can contain the following characters: A-Z, a-z, 0-9, underscore
(_), hyphen (-), and period (.). The following restrictions apply:
• Can be up to 256 characters long
• Can't start with AWS. or Amazon. (or any casing variations)
• Is case-sensitive
• Must be unique among all attribute names for the message
• Must not start or end with a period
• Must not have periods in a sequence
39
Amazon Simple Queue Service Developer Guide
Message attributes
• Type – The message attribute data type. Supported types include String, Number, and Binary.
You can also add custom information for any data type. The data type has the same restrictions as
the message body (for more information, see SendMessage in the Amazon Simple Queue Service API
Reference). In addition, the following restrictions apply:
• Can be up to 256 characters long
• Is case-sensitive
• Value – The message attribute value. For String data types, the attribute values has the same
restrictions as the message body.
Amazon SQS supports the logical data types String, Number, and Binary with optional custom data
type labels with the format .custom-data-type
• String – String attributes can store Unicode text using any valid XML characters.
• Number – Number attributes can store positive or negative numerical values. A number can have up to
38 digits of precision, and it can be between 10^-128 and 10^+126.
Note
Amazon SQS removes leading and trailing zeroes.
• Binary – Binary attributes can store any binary data such as compressed data, encrypted data, or
images.
• Custom – To create a custom data type, append a custom-type label to any data type. For example:
• Number.byte, Number.short, Number.int, and Number.float can help distinguish between
number types.
• Binary.gif and Binary.png can help distinguish between file types.
Note
Amazon SQS doesn't interpret, validate, or use the appended data.
The custom-type label has the same restrictions as the message body.
If you use either the Query API or one of the AWS SDKs that doesn't support MD5 message digests for
Amazon SQS message attributes, you must use the following guidelines to perform the MD5 message
digest calculation.
Note
Always include custom data type suffixes in the MD5 message-digest calculation.
Overview
The following is an overview of the MD5 message digest calculation algorithm:
40
Amazon Simple Queue Service Developer Guide
Message system attributes
The following diagram shows the encoding of the MD5 message digest for a single message attribute:
a. For the String transport type, encode the attribute value: the length (4 bytes) and the UTF-8
bytes of the value.
b. For the Binary transport type, encode the attribute value: the length (4 bytes) and the raw
bytes of the value.
Message system attributes are structured exactly like message attributes, with the following exceptions:
• Currently, the only supported message system attribute is AWSTraceHeader. Its type must be String
and its value must be a correctly formatted AWS X-Ray trace header string.
• The size of a message system attribute doesn't count towards the total size of a message.
41
Amazon Simple Queue Service Developer Guide
Resources required to process messages
The following table lists the attribute name to use with the GetQueueAttributes action:
If you set MaxResults, the response includes a value for NextToken if there are additional results to
display. Use NextToken as a parameter in your next request to listQueues to receive the next page of
results. If there are no additional results to display, the NextToken value in the response is null.
42
Amazon Simple Queue Service Developer Guide
Short and long polling
You can use cost allocation tags to organize your AWS bill to reflect your own cost structure. To do this,
sign up to get your AWS account bill to include tag keys and values. For more information, see Setting
Up a Monthly Cost Allocation Report in the AWS Billing and Cost Management User Guide.
Each tag consists of a key-value pair that you define. For example, you can easily identify your production
and testing queues if you tag your queues as follows:
Note
When you use queue tags, keep the following guidelines in mind:
For a full list of tag restrictions, see Quotas related to queues (p. 102).
With short polling, the ReceiveMessage request queries only a subset of the servers (based on a
weighted random distribution) to find messages that are available to include in the response. Amazon
SQS sends the response right away, even if the query found no messages.
With long polling, the ReceiveMessage request queries all of the servers for messages. Amazon
SQS sends a response after it collects at least one available message, up to the maximum number of
messages specified in the request. Amazon SQS sends an empty response only if the polling wait time
expires.
The following sections explain the details of short polling and long polling.
Topics
• Consuming messages using short polling (p. 43)
• Consuming messages using long polling (p. 44)
• Differences between long and short polling (p. 44)
43
Amazon Simple Queue Service Developer Guide
Consuming messages using long polling
Thus, a particular ReceiveMessage request might not return all of your messages. However, if you
have fewer than 1,000 messages in your queue, a subsequent request will return your messages. If you
keep consuming from your queues, Amazon SQS samples all of its servers, and you receive all of your
messages.
The following diagram shows the short-polling behavior of messages returned from a standard queue
after one of your system components makes a receive request. Amazon SQS samples several of its
servers (in gray) and returns messages A, C, D, and B from these servers. Message E isn't returned for this
request, but is returned for a subsequent request.
• Reduce empty responses by allowing Amazon SQS to wait until a message is available in
a queue before sending a response. Unless the connection times out, the response to the
ReceiveMessage request contains at least one of the available messages, up to the maximum
number of messages specified in the ReceiveMessage action. In rare cases, you might receive empty
responses even when a queue still contains messages, especially if you specify a low value for the
ReceiveMessageWaitTimeSeconds parameter.
• Reduce false empty responses by querying all—rather than a subset of—Amazon SQS servers.
• Return messages as soon as they become available.
For information about how to confirm that a queue is empty, see Confirming that a queue is
empty (p. 23).
44
Amazon Simple Queue Service Developer Guide
Dead-letter queues
Topics
• How do dead-letter queues work? (p. 45)
• What are the benefits of dead-letter queues? (p. 46)
• How do different queue types handle message failure? (p. 46)
• When should I use a dead-letter queue? (p. 47)
• Moving messages out of a dead-letter queue (p. 47)
• Troubleshooting dead-letter queues (p. 48)
Occasionally, producers and consumers might fail to interpret aspects of the protocol that they use to
communicate, causing message corruption or loss. Also, the consumer's hardware errors might corrupt
message payload.
The redrive policy specifies the source queue, the dead-letter queue, and the conditions under which
Amazon SQS moves messages from the former to the latter if the consumer of the source queue fails
to process a message a specified number of times. The maxReceiveCount is the number of times a
consumer tries receiving a message from a queue without deleting it before being moved to the dead-
letter queue. Setting the maxReceiveCount to a low value such as 1 would result in any failure to
receive a message to cause the message to be moved to the dead-letter queue. Such failures include
network errors and client dependency errors. To ensure that your system is resilient against errors, set
the maxReceiveCount high enough to allow for sufficient retries.
The redrive allow policy specifies which source queues can access the dead-letter queue. This policy
applies to a potential dead-letter queue. You can choose whether to allow all source queues, allow
specific source queues, or deny all source queues. The default is to allow all source queues to use the
dead-letter queue. If you choose to allow specific queues (using the byQueue option), you can specify up
to 10 source queues using the source queue Amazon Resource Name (ARN). If you specify denyAll, the
queue cannot be used as a dead-letter queue.
45
Amazon Simple Queue Service Developer Guide
What are the benefits of dead-letter queues?
To specify a dead-letter queue, you can use the console or the AWS SDK for Java. You must do
this for each queue that sends messages to a dead-letter queue. Multiple queues of the same type
can target a single dead-letter queue. For more information, see Configuring a dead-letter queue
(console) (p. 16) and the RedrivePolicy and RedriveAllowPolicy attributes of the CreateQueue
or SetQueueAttributes action.
Important
The dead-letter queue of a FIFO queue must also be a FIFO queue. Similarly, the dead-letter
queue of a standard queue must also be a standard queue.
You must use the same AWS account to create the dead-letter queue and the other queues
that send messages to the dead-letter queue. Also, dead-letter queues must reside in the same
region as the other queues that use the dead-letter queue. For example, if you create a queue in
the US East (Ohio) region and you want to use a dead-letter queue with that queue, the second
queue must also be in the US East (Ohio) region.
The expiration of a message is always based on its original enqueue timestamp. When
a message is moved to a dead-letter queue, the enqueue timestamp is unchanged. The
ApproximateAgeOfOldestMessage metric indicates when the message moved to the dead-
letter queue, not when the message was originally sent. For example, assume that a message
spends 1 day in the original queue before it's moved to a dead-letter queue. If the dead-letter
queue's retention period is 4 days, the message is deleted from the dead-letter queue after 3
days and the ApproximateAgeOfOldestMessage is 3 days. Thus, it is a best practice to always
set the retention period of a dead-letter queue to be longer than the retention period of the
original queue.
In a system that processes thousands of messages, having a large number of messages that the
consumer repeatedly fails to acknowledge and delete might increase costs and place extra load on the
hardware. Instead of trying to process failing messages until they expire, it is better to move them to a
dead-letter queue after a few processing attempts.
Note
Standard queues allow a high number of inflight messages. If the majority of your messages
can't be consumed and aren't sent to a dead-letter queue, your rate of processing valid
messages can slow down. Thus, to maintain the efficiency of your queue, make sure that your
application correctly handles message processing.
46
Amazon Simple Queue Service Developer Guide
When should I use a dead-letter queue?
FIFO queues
FIFO queues (p. 28) provide exactly-once processing by consuming messages in sequence from a
message group. Thus, although the consumer can continue to retrieve ordered messages from another
message group, the first message group remains unavailable until the message blocking the queue is
processed successfully.
Note
FIFO queues allow a lower number of inflight messages. Thus, to keep your FIFO queue from
getting blocked by a message, make sure that your application correctly handles message
processing.
Do use dead-letter queues with standard queues. You should always take advantage of dead-
letter queues when your applications don't depend on ordering. Dead-letter queues can help you
troubleshoot incorrect message transmission operations.
Note
Even when you use dead-letter queues, you should continue to monitor your queues and retry
sending messages that fail for transient reasons.
Do use dead-letter queues to decrease the number of messages and to reduce the possibility of
exposing your system to poison-pill messages (messages that can be received but can't be processed).
Don't use a dead-letter queue with standard queues when you want to be able to keep retrying the
transmission of a message indefinitely. For example, don't use a dead-letter queue if your program must
wait for a dependent process to become active or available.
Don't use a dead-letter queue with a FIFO queue if you don't want to break the exact order of
messages or operations. For example, don't use a dead-letter queue with instructions in an Edit Decision
List (EDL) for a video editing suite, where changing the order of edits changes the context of subsequent
edits.
47
Amazon Simple Queue Service Developer Guide
Troubleshooting dead-letter queues
letter queues to the source queues, it makes the dead-letter queue redrive to source queues a low-cost
and straightforward option for users to move messages from dead-letter queues easily.
By default, dead-letter queue redrive moves messages from a dead-letter queue to a source queue.
However, you can also configure any other standard queue as the redrive destination. Additionally, you
can configure the redrive velocity to set the rate at which Amazon SQS moves messages. For instructions
about configuring a dead-letter queue redrive using the Amazon SQS console, see Configuring a dead-
letter queue redrive (console) (p. 16).
Note
Amazon SQS supports dead-letter queue redrive only for standard queues in the Amazon SQS
console.
Amazon SQS doesn't support filtering and modifying messages while redriving them from the
dead-letter queue.
A dead-letter queue redrive task can run a maximum of 36 hours. Amazon SQS supports a
maximum of 100 active redrive tasks per account.
• Increase the Maximum Receives setting for the corresponding queue's redrive policy.
• Avoid viewing the corresponding queue's messages in the console.
48
Amazon Simple Queue Service Developer Guide
Visibility timeout
Immediately after a message is received, it remains in the queue. To prevent other consumers from
processing the message again, Amazon SQS sets a visibility timeout, a period of time during which
Amazon SQS prevents other consumers from receiving and processing the message. The default
visibility timeout for a message is 30 seconds. The minimum is 0 seconds. The maximum is 12 hours. For
information about configuring visibility timeout for a queue using the console, see Configuring queue
parameters (console) (p. 13).
Note
For standard queues, the visibility timeout isn't a guarantee against receiving a message twice.
For more information, see At-least-once delivery (p. 28).
FIFO queues allow the producer or consumer to attempt multiple retries:
• If the producer detects a failed SendMessage action, it can retry sending as many times as
necessary, using the same message deduplication ID. Assuming that the producer receives at
least one acknowledgement before the deduplication interval expires, multiple retries neither
affect the ordering of messages nor introduce duplicates.
• If the consumer detects a failed ReceiveMessage action, it can retry as many times as
necessary, using the same receive request attempt ID. Assuming that the consumer receives at
least one acknowledgement before the visibility timeout expires, multiple retries don't affect
the ordering of messages.
• When you receive a message with a message group ID, no more messages for the same
message group ID are returned unless you delete the message or it becomes visible.
Topics
49
Amazon Simple Queue Service Developer Guide
Inflight messages
Inflight messages
An Amazon SQS message has three basic states:
A message is considered to be stored after it is sent to a queue by a producer, but not yet received from
the queue by a consumer (that is, between states 1 and 2). There is no quota to the number of stored
messages. A message is considered to be in flight after it is received from a queue by a consumer, but not
yet deleted from the queue (that is, between states 2 and 3). There is a quota to the number of inflight
messages.
Important
Quotas that apply to inflight messages are unrelated to the unlimited number of stored
messages.
For most standard queues (depending on queue traffic and message backlog), there can be a maximum
of approximately 120,000 inflight messages (received from a queue by a consumer, but not yet deleted
from the queue). If you reach this quota while using short polling (p. 43), Amazon SQS returns the
OverLimit error message. If you use long polling (p. 44), Amazon SQS returns no error messages.
To avoid reaching the quota, you should delete messages from the queue after they're processed. You
can also increase the number of queues you use to process your messages. To request a quota increase,
submit a support request.
For FIFO queues, there can be a maximum of 20,000 inflight messages (received from a queue by a
consumer, but not yet deleted from the queue). If you reach this quota, Amazon SQS returns no error
messages.
Important
When working with FIFO queues, DeleteMessage operations will fail if the request is received
outside of the visibility timeout window. If the visibility timeout is 0 seconds, the message must
be deleted within the same millisecond it was sent, or it is considered abandoned. This can
cause Amazon SQS to include duplicate messages in the same response to a ReceiveMessage
operation if the MaxNumberOfMessages parameter is greater than 1. For additional details see
How the Amazon SQS FIFO API Works.
Every Amazon SQS queue has the default visibility timeout setting of 30 seconds. You can change this
setting for the entire queue. Typically, you should set the visibility timeout to the maximum time that
it takes your application to process and delete a message from the queue. When receiving messages,
you can also set a special visibility timeout for the returned messages without changing the overall
50
Amazon Simple Queue Service Developer Guide
Changing the visibility timeout for a message
queue timeout. For more information, see the best practices in the Processing messages in a timely
manner (p. 57) section.
If you don't know how long it takes to process a message, create a heartbeat for your consumer process:
Specify the initial visibility timeout (for example, 2 minutes) and then—as long as your consumer still
works on the message—keep extending the visibility timeout by 2 minutes every minute.
Important
The maximum visibility timeout is 12 hours from the time that Amazon SQS receives the
ReceiveMessage request. Extending the visibility timeout does not reset the 12 hour
maximum.
Additionally, you may be unable to set the timeout on an individual message to the full
12 hours (e.g. 43,200 seconds) since the ReceiveMessage request initiates the timer. For
example, if you receive a message and immediately set the 12 hour maximum by sending a
ChangeMessageVisibility call with VisibilityTimeout equal to 43,200 seconds, it will
likely fail. However, using a value of 43,195 seconds will work unless there is a significant delay
between requesting the message via ReceiveMessage and updating the visibility timeout. If
your consumer needs longer than 12 hours, consider using Step Functions.
For example, if the default timeout for a queue is 60 seconds, 15 seconds have elapsed
since you received the message, and you send a ChangeMessageVisibility call with
VisibilityTimeout set to 10 seconds, the 10 seconds begin to count from the time that you make the
ChangeMessageVisibility call. Thus, any attempt to change the visibility timeout or to delete that
message 10 seconds after you initially change the visibility timeout (a total of 25 seconds) might result in
an error.
Note
The new timeout period takes effect from the time you call the ChangeMessageVisibility
action. In addition, the new timeout period applies only to the particular receipt of the message.
ChangeMessageVisibility doesn't affect the timeout of later receipts of the message or
later queues.
51
Amazon Simple Queue Service Developer Guide
Temporary queues
Note
For standard queues, the per-queue delay setting is not retroactive—changing the setting
doesn't affect the delay of messages already in the queue.
For FIFO queues, the per-queue delay setting is retroactive—changing the setting affects the
delay of messages already in the queue.
Delay queues are similar to visibility timeouts (p. 49) because both features make messages
unavailable to consumers for a specific period of time. The difference between the two is that, for delay
queues, a message is hidden when it is first added to queue, whereas for visibility timeouts a message
is hidden only after it is consumed from the queue. The following diagram illustrates the relationship
between delay queues and visibility timeouts.
To set delay seconds on individual messages, rather than on an entire queue, use message
timers (p. 56) to allow Amazon SQS to use the message timer's DelaySeconds value instead of the
delay queue's DelaySeconds value.
The client maps multiple temporary queues—application-managed queues created on demand for a
particular process—onto a single Amazon SQS queue automatically. This allows your application to make
fewer API calls and have a higher throughput when the traffic to each temporary queue is low. When
a temporary queue is no longer in use, the client cleans up the temporary queue automatically, even if
some processes that use the client aren't shut down cleanly.
Topics
52
Amazon Simple Queue Service Developer Guide
Virtual queues
Virtual queues
Virtual queues are local data structures that the Temporary Queue Client creates. Virtual queues let you
combine multiple low-traffic destinations into a single Amazon SQS queue. For best practices, see Avoid
reusing the same message group ID with virtual queues (p. 62).
Note
• Creating a virtual queue creates only temporary data structures for consumers to receive
messages in. Because a virtual queue makes no API calls to Amazon SQS, virtual queues incur
no cost.
• TPS quotas apply to all virtual queues across a single host queue. For more information, see
Quotas related to messages (p. 103).
The AmazonSQSVirtualQueuesClient wrapper class adds support for attributes related to virtual
queues. To create a virtual queue, you must call the CreateQueue API action using the HostQueueURL
attribute. This attribute specifies the existing queue that hosts the virtual queues.
https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue#MyVirtualQueueName
When a producer calls the SendMessage or SendMessageBatch API action on a virtual queue URL, the
Temporary Queue Client does the following:
While the producer sends messages, a background thread polls the host queue and sends received
messages to virtual queues according to the corresponding message attributes.
While the consumer calls the ReceiveMessage API action on a virtual queue URL, the Temporary Queue
Client blocks the call locally until the background thread sends a message into the virtual queue. (This
process is similar to message prefetching in the Buffered Asynchronous Client (p. 169): a single API
action can provide messages to up to 10 virtual queues.) Deleting a virtual queue removes any client-side
resources without calling Amazon SQS itself.
The AmazonSQSTemporaryQueuesClient class turns all queues it creates into temporary queues
automatically. It also creates host queues with the same queue attributes automatically, on demand.
These queues' names share a common, configurable prefix (by default, __RequesterClientQueues__)
that identifies them as temporary queues. This allows the client to act as a drop-in replacement
that optimizes existing code which creates and deletes queues. The client also includes the
AmazonSQSRequester and AmazonSQSResponder interfaces that allow two-way communication
between queues.
53
Amazon Simple Queue Service Developer Guide
Example scenario: Processing a login request
temporary queues without making any Amazon SQS API calls. For more information, see Implementing
request-response systems (p. 59).
54
Amazon Simple Queue Service Developer Guide
Cleaning up queues
return reply.getBody();
}
}
// Extract the URL of the temporary queue from the message attribute
// and send the response to the temporary queue.
sqsResponder.sendResponseMessage(MessageContent.fromMessage(message),
new MessageContent(response));
}
}
Cleaning up queues
To make sure that Amazon SQS reclaims any in-memory resources used by virtual queues, when your
application no longer needs the Temporary Queue Client, it should call the shutdown() method. You
can also use the shutdown() method of the AmazonSQSRequester interface.
The Temporary Queue Client also provides a way to eliminate orphaned host queues. For each queue
that receives an API call over a period of time (by default, five minutes), the client uses the TagQueue
API action to tag a queue that remains in use.
55
Amazon Simple Queue Service Developer Guide
Message timers
Note
Any API action taken on a queue marks it as non-idle, including a ReceiveMessage action that
returns no messages.
The background thread uses the ListQueues and ListTags API actions to check all queues with the
configured prefix, deleting any queues that haven't been tagged for at least five minutes. In this way,
if one client doesn't shut down cleanly, the other active clients clean up after it. In order to reduce the
duplication of work, all clients with the same prefix communicate through a shared, internal work queue
named after the prefix.
To set a delay period on an entire queue, rather than on individual messages, use delay queues (p. 51).
A message timer setting for an individual message overrides any DelaySeconds value on an Amazon
SQS delay queue.
56
Amazon Simple Queue Service Developer Guide
Recommendations for standard and FIFO queues
Topics
• Recommendations for Amazon SQS standard and FIFO queues (p. 57)
• Additional recommendations for Amazon SQS FIFO queues (p. 60)
Topics
• Working with Amazon SQS messages (p. 57)
• Reducing Amazon SQS costs (p. 59)
• Moving from an Amazon SQS Standard queue to a FIFO queue (p. 60)
Topics
• Processing messages in a timely manner (p. 57)
• Handling request errors (p. 58)
• Setting up long polling (p. 58)
• Capturing problematic messages (p. 59)
• Setting up dead-letter queue retention (p. 59)
• Avoiding inconsistent message processing (p. 59)
• Implementing request-response systems (p. 59)
To make sure that there is sufficient time to process messages, use one of the following strategies:
• If you know (or can reasonably estimate) how long it takes to process a message, extend the
message's visibility timeout to the maximum time it takes to process and delete the message. For more
information, see Configuring the Visibility Timeout (p. 50).
57
Amazon Simple Queue Service Developer Guide
Working with messages
• If you don't know how long it takes to process a message, create a heartbeat for your consumer
process: Specify the initial visibility timeout (for example, 2 minutes) and then—as long as your
consumer still works on the message—keep extending the visibility timeout by 2 minutes every
minute.
Important
The maximum visibility timeout is 12 hours from the time that Amazon SQS receives the
ReceiveMessage request. Extending the visibility timeout does not reset the 12 hour
maximum.
Additionally, you may be unable to set the timeout on an individual message to the full
12 hours (e.g. 43,200 seconds) since the ReceiveMessage request initiates the timer. For
example, if you receive a message and immediately set the 12 hour maximum by sending a
ChangeMessageVisibility call with VisibilityTimeout equal to 43,200 seconds, it
will likely fail. However, using a value of 43,195 seconds will work unless there is a significant
delay between requesting the message via ReceiveMessage and updating the visibility
timeout. If your consumer needs longer than 12 hours, consider using Step Functions.
• If you use an AWS SDK, you already have automatic retry and backoff logic at your disposal. For more
information, see Error Retries and Exponential Backoff in AWS in the Amazon Web Services General
Reference.
• If you don't use the AWS SDK features for retry and backoff, allow a pause (for example, 200 ms)
before retrying the ReceiveMessage action after receiving no messages, a timeout, or an error message
from Amazon SQS. For subsequent use of ReceiveMessage that gives the same results, allow a
longer pause (for example, 400 ms).
• In most cases, you can set the ReceiveMessage wait time to 20 seconds. If 20 seconds is too long
for your application, set a shorter ReceiveMessage wait time (1 second minimum). If you don't use
an AWS SDK to access Amazon SQS, or if you configure an AWS SDK to have a shorter wait time, you
might have to modify your Amazon SQS client to either allow longer requests or use a shorter wait
time for long polling.
• If you implement long polling for multiple queues, use one thread for each queue instead of a single
thread for all queues. Using a single thread for each queue allows your application to process the
messages in each of the queues as they become available, while using a single thread for polling
multiple queues might cause your application to become unable to process messages available in
other queues while the application waits (up to 20 seconds) for the queue which doesn't have any
available messages.
Important
To avoid HTTP errors, make sure that the HTTP response timeout for ReceiveMessage
requests is longer than the WaitTimeSeconds parameter. For more information, see
ReceiveMessage.
58
Amazon Simple Queue Service Developer Guide
Reducing costs
• The redrive policy redirects messages to a dead-letter queue after the source queue fails to process a
message a specified number of times.
• Using a dead-letter queue decreases the number of messages and reduces the possibility of exposing
you to poison pill messages (messages that are received but can't be processed).
• Including a poison pill message in a queue can distort the
ApproximateAgeOfOldestMessage (p. 153) CloudWatch metric by giving an incorrect age of the
poison pill message. Configuring a dead-letter queue helps avoid false alarms when using this metric.
• Don't create reply queues per message. Instead, create reply queues on startup, per producer, and use a
correlation ID message attribute to map replies to requests.
• Don't let your producers share reply queues. This can cause a producer to receive response messages
intended for another producer.
For more information about implementing the request-response pattern using the Temporary Queue
Client, see Request-response messaging pattern (virtual queues) (p. 53).
59
Amazon Simple Queue Service Developer Guide
Moving from a Standard queue to a FIFO queue
• To send, receive, and delete messages, and to change the message visibility timeout for multiple
messages with a single action, use the Amazon SQS batch API actions (p. 168).
• To combine client-side buffering with request batching, use long polling together with the buffered
asynchronous client (p. 169) included with the AWS SDK for Java.
Note
The Amazon SQS Buffered Asynchronous Client doesn't currently support FIFO queues.
For more information, see Moving from a standard queue to a FIFO queue (p. 31).
Topics
• Using the Amazon SQS message deduplication ID (p. 60)
• Using the Amazon SQS message group ID (p. 62)
• Using the Amazon SQS receive request attempt ID (p. 63)
60
Amazon Simple Queue Service Developer Guide
Using the message deduplication ID
deduplication ID are accepted successfully but aren't delivered during the 5-minute deduplication
interval.
Note
Message deduplication applies to an entire queue, not to individual message groups.
Amazon SQS continues to keep track of the message deduplication ID even after the message is
received and deleted.
• Messages sent with identical message bodies that Amazon SQS must treat as unique.
• Messages sent with identical content but different message attributes that Amazon SQS must treat as
unique.
• Messages sent with different content (for example, retry counts included in the message body) that
Amazon SQS must treat as duplicates.
• Enable content-based deduplication for the queue (each of your messages has a unique body). The
producer can omit the message deduplication ID.
• Although the consumer isn't required to provide a receive request attempt ID for each request, it's a
best practice because it allows fail-retry sequences to execute faster.
• You can retry send or receive requests because they don't interfere with the ordering of messages in
FIFO queues.
• The producer must be aware of the deduplication interval of the queue. Amazon SQS has a
deduplication interval of 5 minutes. Retrying SendMessage requests after the deduplication interval
expires can introduce duplicate messages into the queue. For example, a mobile device in a car sends
messages whose order is important. If the car loses cellular connectivity for a period of time before
receiving an acknowledgement, retrying the request after regaining cellular connectivity can create a
duplicate.
• The consumer must have a visibility timeout that minimizes the risk of being unable to process
messages before the visibility timeout expires. You can extend the visibility timeout while the
messages are being processed by calling the ChangeMessageVisibility action. However, if the
visibility timeout expires, another consumer can immediately begin to process the messages, causing a
message to be processed multiple times. To avoid this scenario, configure a dead-letter queue (p. 45).
61
Amazon Simple Queue Service Developer Guide
Using the message group ID
62
Amazon Simple Queue Service Developer Guide
Using the receive request attempt ID
During a long-lasting network outage that causes connectivity issues between your SDK and Amazon
SQS, it's a best practice to provide the receive request attempt ID and to retry with the same receive
request attempt ID if the SDK operation fails.
63
Amazon Simple Queue Service Developer Guide
Using server-side encryption
For examples of basic Amazon SQS queue operations, such as how to create a queue or send a message,
see Working with Amazon SQS Message Queues in the AWS SDK for Java 2.x Developer Guide.
The examples in this topic demonstrate additional Amazon SQS features, such as server-side encryption
(SSE), cost-allocation tags, and message attributes.
Topics
• Using server-side encryption (SSE) (p. 64)
• Configuring tags for a queue (p. 66)
• Sending message attributes (p. 67)
• Managing large Amazon SQS messages using Amazon S3 (p. 69)
The following code example sets the AWS KMS key as the AWS managed KMS key for Amazon SQS. The
example also sets the AWS KMS key reuse period (p. 110) to 140 seconds.
Before you run the example code, make sure that you have set your AWS credentials. For more
information, see Set up AWS Credentials and Region for Development in the AWS SDK for Java 2.x
Developer Guide.
sqsClient.getQueueUrl(GetQueueUrlRequest.builder().queueName(myQueueName).build());
String queueUrl = getQueueUrlResponse.queueUrl();
// Create a hashmap for the attributes. Add the key alias and reuse period to the hashmap.
HashMap<QueueAttributeName, String> attributes = new HashMap<QueueAttributeName, String>();
final String kmsMasterKeyAlias = "alias/aws/sqs"; // the alias of the AWS managed KMS key
for Amazon SQS.
64
Amazon Simple Queue Service Developer Guide
Disabling SSE for a queue
attributes.put(QueueAttributeName.KMS_MASTER_KEY_ID, kmsMasterKeyAlias);
attributes.put(QueueAttributeName.KMS_DATA_KEY_REUSE_PERIOD_SECONDS, "140");
sqsClient.setQueueAttributes(set_attrs_request);
The following example creates a new queue with SSE enabled. The queue uses the AWS managed KMS
key for Amazon SQS. The example also sets the AWS KMS key reuse period (p. 110) to 160 seconds.
Before you run the example code, make sure that you have set your AWS credentials. For more
information, see Set up AWS Credentials and Region for Development in the AWS SDK for Java 2.x
Developer Guide.
// Create a hashmap for the attributes. Add the key alias and reuse period to the hashmap.
HashMap<QueueAttributeName, String> attributes = new HashMap<QueueAttributeName, String>();
final String kmsMasterKeyAlias = "alias/aws/sqs"; // the alias of the AWS managed KMS key
for Amazon SQS.
attributes.put(QueueAttributeName.KMS_MASTER_KEY_ID, kmsMasterKeyAlias);
attributes.put(QueueAttributeName.KMS_DATA_KEY_REUSE_PERIOD_SECONDS, "140");
To retrieve the KMS key ID or the data key reuse period for a particular queue,
run the GetQueueAttributes method and retrieve the KmsMasterKeyId and
KmsDataKeyReusePeriodSeconds values.
65
Amazon Simple Queue Service Developer Guide
Configuring tags
Before you run the example code, make sure that you have set your AWS credentials. For more
information, see Set up AWS Credentials and Region for Development in the AWS SDK for Java 2.x
Developer Guide.
Listing tags
To list the tags for a queue, use the ListQueueTags method.
ListQueueTagsRequest.builder().queueUrl(queueUrl).build();
66
Amazon Simple Queue Service Developer Guide
Removing tags
sqsClient.tagQueue(tagQueueRequest);
Removing tags
To remove one or more tags from the queue, use the UntagQueue method. The following example
removes the Accounting ID tag.
Before you run the example code, make sure that you have set your AWS credentials. For more
information, see Set up AWS Credentials and Region for Development in the AWS SDK for Java 2.x
Developer Guide.
Defining attributes
To define an attribute for a message, add the following code, which uses the MessageAttributeValue
data type. For more information, see Message attribute components (p. 39) and Message attribute data
types (p. 40).
The AWS SDK for Java automatically calculates the message body and message attribute checksums and
compares them with the data that Amazon SQS returns. For more information, see the AWS SDK for Java
2.x Developer Guide and Calculating the MD5 message digest for message attributes (p. 40) for other
programming languages.
String
This example defines a String attribute named Name with the value Jane.
Number
This example defines a Number attribute named AccurateWeight with the value
230.000000000000000001.
67
Amazon Simple Queue Service Developer Guide
Sending a message with attributes
Binary
This example defines a Binary attribute named ByteArray with the value of an uninitialized 10-
byte array.
String (custom)
This example defines the custom attribute String.EmployeeId named EmployeeId with the
value ABC123456.
Number (custom)
This example defines the custom attribute Number.AccountId named AccountId with the value
000123456.
Note
Because the base data type is Number, the ReceiveMessage method returns 123456.
Binary (custom)
This example defines the custom attribute Binary.JPEG named ApplicationIcon with the value
of an uninitialized 10-byte array.
68
Amazon Simple Queue Service Developer Guide
Managing large messages
Important
If you send a message to a First-In-First-Out (FIFO) queue, make sure that the sendMessage
method executes after you provide the message group ID.
If you use the SendMessageBatch method instead of SendMessage, you must specify
message attributes for each message in the batch.
You can use the Amazon SQS Extended Client Library for Java to do the following:
• Specify whether messages are always stored in Amazon S3 or only when the size of a message exceeds
256 KB
• Send a message that references a single message object stored in an S3 bucket
• Retrieve the message object from an S3 bucket
• Delete the message object from an S3 bucket
You can use the Amazon SQS Extended Client Library for Java to manage Amazon SQS messages
using Amazon S3 only with the AWS SDK for Java. You can't do this with the AWS CLI, the Amazon SQS
console, the Amazon SQS HTTP API, or any of the other AWS SDKs.
The SDK for Java and Amazon SQS Extended Client Library for Java require the J2SE Development Kit
8.0 or later.
Prerequisites
The following example uses the AWS Java SDK. To install and set up the SDK, see Set up the AWS SDK
for Java in the AWS SDK for Java Developer Guide.
Before you run the example code, configure your AWS credentials. For more information, see Set up AWS
Credentials and Region for Development in the AWS SDK for Java Developer Guide.
/*
* Copyright 2010-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
69
Amazon Simple Queue Service Developer Guide
Example: Using Amazon S3 to
manage large Amazon SQS messages
import com.amazon.sqs.javamessaging.AmazonSQSExtendedClient;
import com.amazon.sqs.javamessaging.ExtendedClientConfiguration;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.*;
import com.amazonaws.services.sqs.AmazonSQS;
import com.amazonaws.services.sqs.AmazonSQSClientBuilder;
import com.amazonaws.services.sqs.model.*;
import org.joda.time.DateTime;
import org.joda.time.format.DateTimeFormat;
import java.util.Arrays;
import java.util.List;
import java.util.UUID;
/*
* Create a new instance of the builder with all defaults (credentials
* and region) set automatically. For more information, see
* Creating Service Clients in the AWS SDK for Java Developer Guide.
*/
final AmazonS3 s3 = AmazonS3ClientBuilder.defaultClient();
/*
* Set the Amazon S3 bucket name, and then set a lifecycle rule on the
* bucket to permanently delete objects 14 days after each object's
* creation date.
*/
final BucketLifecycleConfiguration.Rule expirationRule =
new BucketLifecycleConfiguration.Rule();
expirationRule.withExpirationInDays(14).withStatus("Enabled");
final BucketLifecycleConfiguration lifecycleConfig =
new BucketLifecycleConfiguration().withRules(expirationRule);
// Create the bucket and allow message objects to be stored in the bucket.
s3.createBucket(S3_BUCKET_NAME);
s3.setBucketLifecycleConfiguration(S3_BUCKET_NAME, lifecycleConfig);
System.out.println("Bucket created and configured.");
/*
* Set the Amazon SQS extended client configuration with large payload
* support enabled.
*/
final ExtendedClientConfiguration extendedClientConfig =
new ExtendedClientConfiguration()
.withLargePayloadSupportEnabled(s3, S3_BUCKET_NAME);
70
Amazon Simple Queue Service Developer Guide
Example: Using Amazon S3 to
manage large Amazon SQS messages
new AmazonSQSExtendedClient(AmazonSQSClientBuilder
.defaultClient(), extendedClientConfig);
/*
* Create a long string of characters for the message object which will
* be stored in the bucket.
*/
int stringLength = 300000;
char[] chars = new char[stringLength];
Arrays.fill(chars, 'x');
final String myLongString = new String(chars);
sqsExtended.deleteQueue(new DeleteQueueRequest(myQueueUrl));
System.out.println("Deleted the queue.");
deleteBucketAndAllContents(s3);
System.out.println("Deleted the bucket.");
}
while (true) {
for (S3ObjectSummary objectSummary : objectListing
.getObjectSummaries()) {
client.deleteObject(S3_BUCKET_NAME, objectSummary.getKey());
}
if (objectListing.isTruncated()) {
objectListing = client.listNextBatchOfObjects(objectListing);
71
Amazon Simple Queue Service Developer Guide
Example: Using Amazon S3 to
manage large Amazon SQS messages
} else {
break;
}
}
client.deleteBucket(S3_BUCKET_NAME);
}
}
72
Amazon Simple Queue Service Developer Guide
Prerequisites
The library supports sending and receiving messages to a queue (the JMS point-to-point model)
according to the JMS 1.1 specification. The library supports sending text, byte, or object messages
synchronously to Amazon SQS queues. The library also supports receiving objects synchronously or
asynchronously.
For information about features of the Amazon SQS Java Messaging Library that support the JMS 1.1
specification, see Supported JMS 1.1 implementations (p. 94) and the Amazon SQS FAQs.
Topics
• Prerequisites (p. 73)
• Getting started with the Amazon SQS Java Messaging Library (p. 74)
• Using the Amazon SQS Java Message Service (JMS) Client with other Amazon SQS clients (p. 79)
• Working Java example for using JMS with Amazon SQS Standard queues (p. 80)
• Supported JMS 1.1 implementations (p. 94)
Prerequisites
Before you begin, you must have the following prerequisites:
There are two ways to include the SDK for Java in your project:
• Download and install the SDK for Java.
• Use Maven to get the Amazon SQS Java Messaging Library.
Note
The SDK for Java is included as a dependency.
The SDK for Java and Amazon SQS Extended Client Library for Java require the J2SE
Development Kit 8.0 or later.
For information about downloading the SDK for Java, see SDK for Java.
• Amazon SQS Java Messaging Library
If you don't use Maven, you must add the amazon-sqs-java-messaging-lib.jar package to
the Java class path. For information about downloading the library, see Amazon SQS Java Messaging
Library.
Note
The Amazon SQS Java Messaging Library includes support for Maven and the Spring
Framework.
73
Amazon Simple Queue Service Developer Guide
Getting started with the Java Messaging Library
For code samples that use Maven, the Spring Framework, and the Amazon SQS Java
Messaging Library, see Working Java example for using JMS with Amazon SQS Standard
queues (p. 80).
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>amazon-sqs-java-messaging-lib</artifactId>
<version>1.0.4</version>
<type>jar</type>
</dependency>
Create a queue using the AWS Management Console for Amazon SQS, the CreateQueue API, or the
wrapped Amazon SQS client included in the Amazon SQS Java Messaging Library.
• For information about creating a queue with Amazon SQS using either the AWS Management
Console or the CreateQueue API, see Creating a Queue (p. 11).
• For information about using the Amazon SQS Java Messaging Library, see Getting started with the
Amazon SQS Java Messaging Library (p. 74).
// Create a new connection factory with all defaults (credentials and region) set
automatically
SQSConnectionFactory connectionFactory = new SQSConnectionFactory(
new ProviderConfiguration(),
AmazonSQSClientBuilder.defaultClient()
);
The SQSConnection class extends javax.jms.Connection. Together with the JMS standard
connection methods, SQSConnection offers additional methods, such as getAmazonSQSClient
and getWrappedAmazonSQSClient. Both methods let you perform administrative
operations not included in the JMS specification, such as creating new queues. However, the
getWrappedAmazonSQSClient method also provides a wrapped version of the Amazon SQS client
used by the current connection. The wrapper transforms every exception from the client into an
JMSException, allowing it to be more easily used by existing code that expects JMSException
occurrences.
2. You can use the client objects returned from getAmazonSQSClient and
getWrappedAmazonSQSClient to perform administrative operations not included in the JMS
specification (for example, you can create an Amazon SQS queue).
74
Amazon Simple Queue Service Developer Guide
Creating an Amazon SQS queue
If you have existing code that expects JMS exceptions, then you should use
getWrappedAmazonSQSClient:
• If you use getWrappedAmazonSQSClient, the returned client object transforms all exceptions
into JMS exceptions.
• If you use getAmazonSQSClient, the exceptions are all Amazon SQS exceptions.
If a queue doesn't exist, the client creates it. If the queue does exist, the function doesn't
return anything. For more information, see the "Create the queue if needed" section in the
TextMessageSender.java (p. 82) example.
// Create an Amazon SQS FIFO queue named MyQueue.fifo, if it doesn't already exist
if (!client.queueExists("MyQueue.fifo")) {
Map<String, String> attributes = new HashMap<String, String>();
attributes.put("FifoQueue", "true");
attributes.put("ContentBasedDeduplication", "true");
client.createQueue(new
CreateQueueRequest().withQueueName("MyQueue.fifo").withAttributes(attributes));
}
Note
The name of a FIFO queue must end with the .fifo suffix.
For more information about the ContentBasedDeduplication attribute, see Exactly-once
processing (p. 31).
2. To send a text message to the queue, create a JMS queue identity and a message producer.
// Create a queue identity and specify the queue name to the session
75
Amazon Simple Queue Service Developer Guide
Receiving messages synchronously
• To send a message to a standard queue, you don't need to set any additional parameters.
• To send a message to a FIFO queue, you must set the message group ID. You can also set a
message deduplication ID. For more information, see Key terms (p. 29).
You can call the start method on the connection at any time. However, the consumer doesn't begin
to receive messages until you call it.
2. Call the receive method on the consumer with a timeout set to 1 second, and then print the
contents of the received message.
• After receiving a message from a standard queue, you can access the contents of the message.
76
Amazon Simple Queue Service Developer Guide
Receiving messages asynchronously
• After receiving a message from a FIFO queue, you can access the contents of the message and
other, FIFO-specific message attributes, such as the message group ID, message deduplication ID,
and sequence number. For more information, see Key terms (p. 29).
Note
You can use the Spring Framework to initialize these objects.
For additional information, see SpringExampleConfiguration.xml,
SpringExample.java, and the other helper classes in ExampleConfiguration.java and
ExampleCommon.java in the Working Java example for using JMS with Amazon SQS Standard
queues (p. 80) section.
For complete examples of sending and receiving objects, see TextMessageSender.java (p. 82) and
SyncMessageReceiver.java (p. 83).
The following example shows how to receive the messages asynchronously through a listener.
@Override
public void onMessage(Message message) {
try {
// Cast the received message as TextMessage and print the text to screen.
System.out.println("Received: " + ((TextMessage) message).getText());
} catch (JMSException e) {
e.printStackTrace();
}
}
77
Amazon Simple Queue Service Developer Guide
Using client acknowledge mode
The onMessage method of the MessageListener interface is called when you receive a message.
In this listener implementation, the text stored in the message is printed.
2. Instead of explicitly calling the receive method on the consumer, set the message listener of
the consumer to an instance of the MyListener implementation. The main thread waits for one
second.
// Wait for 1 second. The listener onMessage() method is invoked when a message is
received.
Thread.sleep(1000);
The rest of the steps are identical to the ones in the Getting started with the Amazon SQS Java
Messaging Library (p. 74) example. For a complete example of an asynchronous consumer, see
AsyncMessageReceiver.java in Working Java example for using JMS with Amazon SQS Standard
queues (p. 80).
1. To explicitly acknowledge the messages after they're processed, you must create the session with
CLIENT_ACKNOWLEDGE mode.
2. When the message is received, display it and then explicitly acknowledge it.
// Cast the received message as TextMessage and print the text to screen. Also
acknowledge the message.
if (receivedMessage != null) {
System.out.println("Received: " + ((TextMessage) receivedMessage).getText());
receivedMessage.acknowledge();
System.out.println("Acknowledged: " + message.getJMSMessageID());
}
Note
In this mode, when a message is acknowledged, all messages received before this message
are implicitly acknowledged as well. For example, if 10 messages are received, and only
78
Amazon Simple Queue Service Developer Guide
Using unordered acknowledge mode
the 10th message is acknowledged (in the order the messages are received), then all of the
previous nine messages are also acknowledged.
The rest of the steps are identical to the ones in the Getting started with the Amazon SQS Java
Messaging Library (p. 74) example. For a complete example of a synchronous consumer with client
acknowledge mode, see SyncMessageReceiverClientAcknowledge.java in Working Java example
for using JMS with Amazon SQS Standard queues (p. 80).
The Amazon SQS Java Messaging Library provides another acknowledgement mode. When using
UNORDERED_ACKNOWLEDGE mode, all received messages must be individually and explicitly
acknowledged by the client, regardless of their reception order. To do this, create a session with
UNORDERED_ACKNOWLEDGE mode.
The remaining steps are identical to the ones in the Using client acknowledge mode (p. 78) example.
For a complete example of a synchronous consumer with UNORDERED_ACKNOWLEDGE mode, see
SyncMessageReceiverUnorderedAcknowledge.java.
The following Java code example creates the JMS provider for the Extended Client Library:
79
Amazon Simple Queue Service Developer Guide
Working Java example for using JMS
with Amazon SQS Standard queues
// Set the Amazon S3 bucket name, and set a lifecycle rule on the bucket to
// permanently delete objects a certain number of days after each object's creation date.
// Next, create the bucket, and enable message objects to be stored in the bucket.
BucketLifecycleConfiguration.Rule expirationRule = new BucketLifecycleConfiguration.Rule();
expirationRule.withExpirationInDays(14).withStatus("Enabled");
BucketLifecycleConfiguration lifecycleConfig = new
BucketLifecycleConfiguration().withRules(expirationRule);
s3.createBucket(s3BucketName);
s3.setBucketLifecycleConfiguration(s3BucketName, lifecycleConfig);
System.out.println("Bucket created and configured.");
// Set the SQS extended client configuration with large payload support enabled.
ExtendedClientConfiguration extendedClientConfig = new ExtendedClientConfiguration()
.withLargePayloadSupportEnabled(s3, s3BucketName);
// Create the connection factory using the environment variable credential provider.
// Pass the configured Amazon SQS Extended Client to the JMS connection factory.
SQSConnectionFactory connectionFactory = new SQSConnectionFactory(
new ProviderConfiguration(),
sqsExtended
);
ExampleConfiguration.java
The following Java code example sets the default queue name, the region, and the credentials to be used
with the other Java examples.
/*
* Copyright 2010-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* https://aws.amazon.com/apache2.0
*
80
Amazon Simple Queue Service Developer Guide
ExampleConfiguration.java
/**
* Parse the command line and return the resulting config. If the config parsing fails
* print the error and the usage message and then call System.exit
*
* @param app the app to use when printing the usage string
* @param args the command line arguments
* @return the parsed config
*/
public static ExampleConfiguration parseConfig(String app, String args[]) {
try {
return new ExampleConfiguration(args);
} catch (IllegalArgumentException e) {
System.err.println( "ERROR: " + e.getMessage() );
System.err.println();
System.err.println( "Usage: " + app + " [--queue <queue>] [--region <region>]
[--credentials <credentials>] ");
System.err.println( " or" );
System.err.println( " " + app + " <spring.xml>" );
System.exit(-1);
return null;
}
}
81
Amazon Simple Queue Service Developer Guide
TextMessageSender.java
i++;
} else {
throw new IllegalArgumentException("Unrecognized option " + arg);
}
}
}
TextMessageSender.java
The following Java code example creates a text message producer.
/*
* Copyright 2010-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* https://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*
*/
82
Amazon Simple Queue Service Developer Guide
SyncMessageReceiver.java
ExampleCommon.setupLogging();
sendMessages(session, producer);
try {
String input;
while( true ) {
System.out.print( "Enter message to send (leave empty to exit): " );
input = inputReader.readLine();
if( input == null || input.equals("" ) ) break;
SyncMessageReceiver.java
The following Java code example creates a synchronous message consumer.
/*
* Copyright 2010-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
83
Amazon Simple Queue Service Developer Guide
SyncMessageReceiver.java
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* https://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*
*/
ExampleCommon.setupLogging();
connection.start();
receiveMessages(session, consumer);
84
Amazon Simple Queue Service Developer Guide
AsyncMessageReceiver.java
AsyncMessageReceiver.java
The following Java code example creates an asynchronous message consumer.
/*
* Copyright 2010-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* https://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*
*/
ExampleCommon.setupLogging();
callback.waitForOneMinuteOfSilence();
System.out.println( "Returning after one minute of silence" );
85
Amazon Simple Queue Service Developer Guide
SyncMessageReceiverClientAcknowledge.java
@Override
public void onMessage(Message message) {
try {
ExampleCommon.handleMessage(message);
message.acknowledge();
System.out.println( "Acknowledged message " + message.getJMSMessageID() );
timeOfLastMessage = System.nanoTime();
} catch (JMSException e) {
System.err.println( "Error processing message: " + e.getMessage() );
e.printStackTrace();
}
}
}
}
SyncMessageReceiverClientAcknowledge.java
The following Java code example creates a synchronous consumer with client acknowledge mode.
/*
* Copyright 2010-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* https://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*
*/
/**
* An example class to demonstrate the behavior of CLIENT_ACKNOWLEDGE mode for received
messages. This example
* complements the example given in {@link SyncMessageReceiverUnorderedAcknowledge} for
UNORDERED_ACKNOWLEDGE mode.
*
* First, a session, a message producer, and a message consumer are created. Then, two
messages are sent. Next, two messages
* are received but only the second one is acknowledged. After waiting for the visibility
time out period, an attempt to
* receive another message is made. It's shown that no message is returned for this attempt
since in CLIENT_ACKNOWLEDGE mode,
* as expected, all the messages prior to the acknowledged messages are also acknowledged.
86
Amazon Simple Queue Service Developer Guide
SyncMessageReceiverClientAcknowledge.java
*
* This ISN'T the behavior for UNORDERED_ACKNOWLEDGE mode. Please see {@link
SyncMessageReceiverUnorderedAcknowledge}
* for an example.
*/
public class SyncMessageReceiverClientAcknowledge {
// Visibility time-out for the queue. It must match to the one set for the queue for
this example to work.
private static final long TIME_OUT_SECONDS = 1;
// Wait for the visibility time out, so that unacknowledged messages reappear in
the queue
System.out.println("Waiting for visibility timeout...");
Thread.sleep(TimeUnit.SECONDS.toMillis(TIME_OUT_SECONDS));
// Attempt to receive another message and acknowledge it. This results in receiving
no messages since
// we have acknowledged the second message. Although we didn't explicitly
acknowledge the first message,
// in the CLIENT_ACKNOWLEDGE mode, all the messages received prior to the
explicitly acknowledged message
87
Amazon Simple Queue Service Developer Guide
SyncMessageReceiverUnorderedAcknowledge.java
/**
* Sends a message through the producer.
*
* @param producer Message producer
* @param session Session
* @param messageText Text for the message to be sent
* @throws JMSException
*/
private static void sendMessage(MessageProducer producer, Session session, String
messageText) throws JMSException {
// Create a text message and send it
producer.send(session.createTextMessage(messageText));
}
/**
* Receives a message through the consumer synchronously with the default timeout
(TIME_OUT_SECONDS).
* If a message is received, the message is printed. If no message is received, "Queue
is empty!" is
* printed.
*
* @param consumer Message consumer
* @param acknowledge If true and a message is received, the received message is
acknowledged.
* @throws JMSException
*/
private static void receiveMessage(MessageConsumer consumer, boolean acknowledge)
throws JMSException {
// Receive a message
Message message = consumer.receive(TimeUnit.SECONDS.toMillis(TIME_OUT_SECONDS));
if (message == null) {
System.out.println("Queue is empty!");
} else {
// Since this queue has only text messages, cast the message object and print
the text
System.out.println("Received: " + ((TextMessage) message).getText());
SyncMessageReceiverUnorderedAcknowledge.java
The following Java code example creates a synchronous consumer with unordered acknowledge mode.
/*
* Copyright 2010-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
88
Amazon Simple Queue Service Developer Guide
SyncMessageReceiverUnorderedAcknowledge.java
* https://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*
*/
/**
* An example class to demonstrate the behavior of UNORDERED_ACKNOWLEDGE mode for received
messages. This example
* complements the example given in {@link SyncMessageReceiverClientAcknowledge} for
CLIENT_ACKNOWLEDGE mode.
*
* First, a session, a message producer, and a message consumer are created. Then, two
messages are sent. Next, two messages
* are received but only the second one is acknowledged. After waiting for the visibility
time out period, an attempt to
* receive another message is made. It's shown that the first message received in the prior
attempt is returned again
* for the second attempt. In UNORDERED_ACKNOWLEDGE mode, all the messages must be
explicitly acknowledged no matter what
* the order they're received.
*
* This ISN'T the behavior for CLIENT_ACKNOWLEDGE mode. Please see {@link
SyncMessageReceiverClientAcknowledge}
* for an example.
*/
public class SyncMessageReceiverUnorderedAcknowledge {
// Visibility time-out for the queue. It must match to the one set for the queue for
this example to work.
private static final long TIME_OUT_SECONDS = 1;
89
Amazon Simple Queue Service Developer Guide
SyncMessageReceiverUnorderedAcknowledge.java
// Wait for the visibility time out, so that unacknowledged messages reappear in
the queue
System.out.println("Waiting for visibility timeout...");
Thread.sleep(TimeUnit.SECONDS.toMillis(TIME_OUT_SECONDS));
// Attempt to receive another message and acknowledge it. This results in receiving
the first message since
// we have acknowledged only the second message. In the UNORDERED_ACKNOWLEDGE mode,
all the messages must
// be explicitly acknowledged.
receiveMessage(consumer, true);
/**
* Sends a message through the producer.
*
* @param producer Message producer
* @param session Session
* @param messageText Text for the message to be sent
* @throws JMSException
*/
private static void sendMessage(MessageProducer producer, Session session, String
messageText) throws JMSException {
// Create a text message and send it
producer.send(session.createTextMessage(messageText));
}
/**
* Receives a message through the consumer synchronously with the default timeout
(TIME_OUT_SECONDS).
* If a message is received, the message is printed. If no message is received, "Queue
is empty!" is
* printed.
*
* @param consumer Message consumer
* @param acknowledge If true and a message is received, the received message is
acknowledged.
* @throws JMSException
*/
private static void receiveMessage(MessageConsumer consumer, boolean acknowledge)
throws JMSException {
// Receive a message
Message message = consumer.receive(TimeUnit.SECONDS.toMillis(TIME_OUT_SECONDS));
if (message == null) {
System.out.println("Queue is empty!");
} else {
90
Amazon Simple Queue Service Developer Guide
SpringExampleConfiguration.xml
// Since this queue has only text messages, cast the message object and print
the text
System.out.println("Received: " + ((TextMessage) message).getText());
SpringExampleConfiguration.xml
The following XML code example is a bean configuration file for SpringExample.java (p. 92).
<!--
Copyright 2010-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
https://aws.amazon.com/apache2.0
<bean id="CredentialsProviderBean"
class="com.amazonaws.auth.DefaultAWSCredentialsProviderChain"/>
<bean id="ProviderConfiguration"
class="com.amazon.sqs.javamessaging.ProviderConfiguration">
<property name="numberOfMessagesToPrefetch" value="5"/>
</bean>
91
Amazon Simple Queue Service Developer Guide
SpringExample.java
factory-method="createConnection"
init-method="start"
destroy-method="close" />
SpringExample.java
The following Java code example uses the bean configuration file to initialize your objects.
/*
* Copyright 2010-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* https://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*
*/
ExampleCommon.setupLogging();
FileSystemXmlApplicationContext context =
new FileSystemXmlApplicationContext( "file://" +
springFile.getAbsolutePath() );
Connection connection;
try {
connection = context.getBean(Connection.class);
} catch( NoSuchBeanDefinitionException e ) {
System.err.println( "Can't find the JMS connection to use: " +
e.getMessage() );
System.exit(3);
return;
}
String queueName;
try {
queueName = context.getBean("QueueName", String.class);
} catch( NoSuchBeanDefinitionException e ) {
92
Amazon Simple Queue Service Developer Guide
ExampleCommon.java
receiveMessages(session, consumer);
ExampleCommon.java
The following Java code example checks if an Amazon SQS queue exists and then creates one if it
doesn't. It also includes example logging code.
/*
* Copyright 2010-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* https://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*
*/
93
Amazon Simple Queue Service Developer Guide
Supported JMS 1.1 implementations
/**
* In most cases, you can do this with just a createQueue call, but GetQueueUrl
* (called by queueExists) is a faster operation for the common case where the
queue
* already exists. Also many users and roles have permission to call GetQueueUrl
* but don't have permission to call CreateQueue.
*/
if( !client.queueExists(queueName) ) {
client.createQueue( queueName );
}
}
94
Amazon Simple Queue Service Developer Guide
Supported message types
• Session
• MessageConsumer
• MessageProducer
Note
The UNORDERED_ACKNOWLEDGE mode isn't part of the JMS 1.1 specification. This mode helps
Amazon SQS allow a JMS client to explicitly acknowledge a message.
• JMSXGroupID (required for FIFO queues, not allowed for standard queues)
• JMS_SQS_DeduplicationId (optional for FIFO queues, not allowed for standard queues)
After you send messages, Amazon SQS sets the following headers and properties for each message:
• JMSMessageID
• JMS_SQS_SequenceNumber (only for FIFO queues)
• JMSDestination
• JMSMessageID
• JMSRedelivered
• JMSXDeliveryCount
• JMSXGroupID (only for FIFO queues)
• JMS_SQS_DeduplicationId (only for FIFO queues)
• JMS_SQS_SequenceNumber (only for FIFO queues)
95
Amazon Simple Queue Service Developer Guide
Creating an Amazon SQS queue (AWS CloudFormation)
Topics
• Creating an Amazon SQS queue (AWS CloudFormation) (p. 96)
• Tutorial: Sending a message to an Amazon SQS queue from Amazon Virtual Private Cloud (p. 97)
1. Copy the following JSON code to a file named MyQueue.json. To create a standard queue, omit
the FifoQueue and ContentBasedDeduplication properties. For more information on content-
based deduplication, see Exactly-once processing (p. 31).
Note
The name of a FIFO queue must end with the .fifo suffix.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"MyQueue": {
"Properties": {
"QueueName": "MyQueue.fifo",
"FifoQueue": true,
"ContentBasedDeduplication": true
},
"Type": "AWS::SQS::Queue"
}
},
"Outputs": {
"QueueName": {
"Description": "The name of the queue",
"Value": {
"Fn::GetAtt": [
"MyQueue",
"QueueName"
]
}
},
"QueueURL": {
"Description": "The URL of the queue",
"Value": {
"Ref": "MyQueue"
}
},
96
Amazon Simple Queue Service Developer Guide
Sending a message from a VPC
"QueueARN": {
"Description": "The ARN of the queue",
"Value": {
"Fn::GetAtt": [
"MyQueue",
"Arn"
]
}
}
}
}
2. Sign in to the AWS CloudFormation console, and then choose Create Stack.
3. On the Specify Template panel, choose Upload a template file, choose your MyQueue.json file,
and then choose Next.
4. On the Specify Details page, type MyQueue for Stack Name, and then choose Next.
5. On the Options page, choose Next.
6. On the Review page, choose Create.
AWS CloudFormation begins to create the MyQueue stack and displays the CREATE_IN_PROGRESS
status. When the process is complete, AWS CloudFormation displays the CREATE_COMPLETE status.
7. (Optional) To display the name, URL, and ARN of the queue, choose the name of the stack and then
on the next page expand the Outputs section.
• You can use Amazon Virtual Private Cloud only with HTTPS Amazon SQS endpoints.
• When you configure Amazon SQS to send messages from Amazon VPC, you must enable
private DNS and specify endpoints in the format sqs.us-east-2.amazonaws.com.
• Private DNS doesn't support legacy endpoints such as queue.amazonaws.com or us-
east-2.queue.amazonaws.com.
Topics
• Step 1: Create an Amazon EC2 key pair (p. 98)
• Step 2: Create AWS resources (p. 98)
• Step 3: Confirm that your EC2 instance isn't publicly accessible (p. 99)
• Step 4: Create an Amazon VPC endpoint for Amazon SQS (p. 99)
• Step 5: Send a message to your Amazon SQS queue (p. 100)
97
Amazon Simple Queue Service Developer Guide
Step 1: Create an Amazon EC2 key pair
• A VPC and the associated networking resources, including a subnet, a security group, an internet
gateway, and a route table
• An Amazon EC2 instance launched into the VPC subnet
• An Amazon SQS queue
AWS CloudFormation begins to create the stack and displays the CREATE_IN_PROGRESS status. When
the process is complete, AWS CloudFormation displays the CREATE_COMPLETE status.
98
Amazon Simple Queue Service Developer Guide
Step 3: Confirm that your EC2
instance isn't publicly accessible
ping amazon.com
99
Amazon Simple Queue Service Developer Guide
Step 5: Send a message to your Amazon SQS queue
The interface VPC endpoint is created and its ID is displayed, for example,
vpce-0ab1cdef2ghi3j456k.
9. Choose Close.
Amazon VPC begins to create the endpoint and displays the pending status. When the process is
complete, Amazon VPC displays the available status.
2. Try to publish a message to the queue again using the following command, for example:
The sending attempt succeeds and the MD5 digest of the message body and the message ID are
displayed, for example:
{
"MD5OfMessageBody": "a1bcd2ef3g45hi678j90klmn12p34qr5",
"MessageId": "12345a67-8901-2345-bc67-d890123e45fg"
}
For information about receiving and deleting the message from the queue created by your AWS
CloudFormation template (for example, VPCE-SQS-Tutorial-Stack-CFQueue-1ABCDEFGH2IJK), see
Receiving and deleting messages (console) (p. 22).
100
Amazon Simple Queue Service Developer Guide
Step 5: Send a message to your Amazon SQS queue
• Terminate Your Instance in the Amazon EC2 User Guide for Linux Instances
• Deleting Your VPC in the Amazon VPC User Guide
• Deleting a Stack on the AWS CloudFormation Console in the AWS CloudFormation User Guide
• Deleting Your Key Pair in the Amazon EC2 User Guide for Linux Instances
101
Amazon Simple Queue Service Developer Guide
Quotas related to queues
Topics
• Quotas related to queues (p. 102)
• Quotas related to messages (p. 103)
• Quotas related to policies (p. 106)
Quota Description
Delay queue The default (minimum) delay for a queue is 0 seconds. The
maximum is 15 minutes.
Long polling wait time The maximum long polling wait time is 20 seconds.
Messages per queue (backlog) The number of messages that an Amazon SQS queue can
store is unlimited.
Messages per queue (in flight) For most standard queues (depending on queue traffic
and message backlog), there can be a maximum of
approximately 120,000 inflight messages (received from a
queue by a consumer, but not yet deleted from the queue).
If you reach this quota while using short polling (p. 43),
Amazon SQS returns the OverLimit error message. If
you use long polling (p. 44), Amazon SQS returns no error
messages. To avoid reaching the quota, you should delete
messages from the queue after they're processed. You can
also increase the number of queues you use to process your
messages. To request a quota increase, submit a support
request.
102
Amazon Simple Queue Service Developer Guide
Quotas related to messages
Quota Description
The name of a FIFO queue must end with the .fifo suffix.
The suffix counts towards the 80-character queue name
quota. To determine whether a queue is FIFO (p. 28), you can
check whether the queue name ends with the suffix.
The tag Key or Value must not include the reserved prefix
aws: (you can't delete tag keys or values with this prefix).
Quota Description
Message content A message can include only XML, JSON, and unformatted
text. The following Unicode characters are allowed: #x9 |
#xA | #xD | #x20 to #xD7FF | #xE000 to #xFFFD | #x10000
to #x10FFFF
Any characters not included in this list are rejected. For more
information, see the W3C specification for characters.
103
Amazon Simple Queue Service Developer Guide
Quotas related to messages
Quota Description
MessageGroupId is required for FIFO queues. You can't use
it for Standard queues.
FIFO queues
104
Amazon Simple Queue Service Developer Guide
Quotas related to messages
Quota Description
Note
The above quotas are available in the following
AWS Regions:
• US East (Ohio)
• US East (N. Virginia)
• US West (Oregon)
• Europe (Ireland)
Message timer The default (minimum) delay for a message is 0 seconds. The
maximum is 15 minutes.
To send messages larger than 256 KB, you can use the
Amazon SQS Extended Client Library for Java. This library
allows you to send an Amazon SQS message that contains
a reference to a message payload in Amazon S3. The
maximum payload size is 2 GB.
Message visibility timeout The default visibility timeout for a message is 30 seconds.
The minimum is 0 seconds. The maximum is 12 hours.
105
Amazon Simple Queue Service Developer Guide
Quotas related to policies
Name Maximum
Bytes 8,192
Conditions 10
Principals 50
Statements 20
106
Amazon Simple Queue Service Developer Guide
Automating notifications using EventBridge
Topics
• Automating notifications from AWS services to Amazon SQS using Amazon EventBridge (p. 107)
• Troubleshooting Amazon Simple Queue Service queues using AWS X-Ray (p. 107)
EventBridge lets you set a variety of targets—such as Amazon SQS standard and FIFO queues—which
receive events in JSON format. For more information, see the Amazon EventBridge User Guide.
To send AWS X-Ray trace headers through Amazon SQS, you can do one of the following:
To collect data on errors and latency, you must instrument the AmazonSQS client using the AWS X-Ray
SDK.
You can use the AWS X-Ray console to view the map of connections between Amazon SQS and other
services that your application uses. You can also use the console to view metrics such as average latency
and failure rates. For more information, see Amazon SQS and AWS X-Ray in the AWS X-Ray Developer
Guide.
107
Amazon Simple Queue Service Developer Guide
Data protection
Topics
• Data protection (p. 108)
• Identity and access management in Amazon SQS (p. 117)
• Logging and monitoring in Amazon SQS (p. 145)
• Compliance validation for Amazon SQS (p. 157)
• Resilience in Amazon SQS (p. 158)
• Infrastructure security in Amazon SQS (p. 159)
• Amazon SQS security best practices (p. 159)
Data protection
The AWS shared responsibility model applies to data protection in Amazon Simple Queue Service.
As described in this model, AWS is responsible for protecting the global infrastructure that runs all
of the AWS Cloud. You are responsible for maintaining control over your content that is hosted on
this infrastructure. This content includes the security configuration and management tasks for the
AWS services that you use. For more information about data privacy, see the Data Privacy FAQ. For
information about data protection in Europe, see the AWS Shared Responsibility Model and GDPR blog
post on the AWS Security Blog.
For data protection purposes, we recommend that you protect AWS account credentials and set up
individual user accounts with AWS Identity and Access Management (IAM). That way each user is given
only the permissions necessary to fulfill their job duties. We also recommend that you secure your data
in the following ways:
We strongly recommend that you never put confidential or sensitive information, such as your
customers' email addresses, into tags or free-form fields such as a Name field. This includes when you
work with Amazon SQS or other AWS services using the console, API, AWS CLI, or AWS SDKs. Any data
that you enter into tags or free-form fields used for names may be used for billing or diagnostic logs.
If you provide a URL to an external server, we strongly recommend that you do not include credentials
information in the URL to validate your request to that server.
108
Amazon Simple Queue Service Developer Guide
Data encryption
The following sections provide information about data protection in Amazon SQS.
Topics
• Data encryption (p. 109)
• Internetwork traffic privacy (p. 116)
Data encryption
Data protection refers to protecting data while in-transit (as it travels to and from Amazon SQS) and at
rest (while it is stored on disks in Amazon SQS data centers). You can protect data in transit using Secure
Sockets Layer (SSL) or client-side encryption. You can protect data at rest by requesting Amazon SQS to
encrypt your messages before saving them to disk in its data centers and then decrypt them when the
messages are received.
Topics
• Encryption at rest (p. 109)
• Key management (p. 111)
Encryption at rest
Server-side encryption (SSE) lets you transmit sensitive data in encrypted queues. SSE protects the
contents of messages in queues using SQS-owned encryption keys (SSE-SQS) or keys managed in
the AWS Key Management Service (SSE-KMS). For information about managing SSE using the AWS
Management Console, see the following:
For information about managing SSE using the AWS SDK for Java (and the CreateQueue,
SetQueueAttributes, and GetQueueAttributes actions), see the following examples:
SSE encrypts messages as soon as Amazon SQS receives them. The messages are stored in encrypted
form and Amazon SQS decrypts messages only when they are sent to an authorized consumer.
Important
All requests to queues with SSE enabled must use HTTPS and Signature Version 4.
An encrypted queue (p. 109) that uses the default key (AWS managed KMS key for Amazon
SQS) cannot invoke a Lambda function in a different AWS account.
Some features of AWS services that can send notifications to Amazon SQS using the AWS
Security Token Service AssumeRole action are compatible with SSE but work only with standard
queues:
For information about compatibility of other services with encrypted queues, see Configure KMS
permissions for AWS services (p. 112) and your service documentation.
109
Amazon Simple Queue Service Developer Guide
Data encryption
AWS KMS combines secure, highly available hardware and software to provide a key management
system scaled for the cloud. When you use Amazon SQS with AWS KMS, the data keys (p. 110) that
encrypt your message data are also encrypted and stored with the data they protect.
• You can create and manage AWS KMS keys (p. 110) yourself.
• You can also use the AWS managed KMS key for Amazon SQS, which is unique for each account and
region.
• The AWS KMS security standards can help you meet encryption-related compliance requirements.
For more information, see What is AWS Key Management Service? in the AWS Key Management Service
Developer Guide.
Topics
• Encryption scope (p. 110)
• Key terms (p. 110)
Encryption scope
SSE encrypts the body of a message in an Amazon SQS queue.
Encrypting a message makes its contents unavailable to unauthorized or anonymous users. This doesn't
affect the normal functioning of Amazon SQS:
• A message is encrypted only if it is sent after the encryption of a queue is enabled. Amazon SQS
doesn't encrypt backlogged messages.
• Any encrypted message remains encrypted even if the encryption of its queue is disabled.
Moving a message to a dead-letter queue (p. 45) doesn't affect its encryption:
• When Amazon SQS moves a message from an encrypted source queue to an unencrypted dead-letter
queue, the message remains encrypted.
• When Amazon SQS moves a message from an unencrypted source queue to an encrypted dead-letter
queue, the message remains unencrypted.
Key terms
The following key terms can help you better understand the functionality of SSE. For detailed
descriptions, see the Amazon Simple Queue Service API Reference.
Data key
The data encryption key (DEK) responsible for encrypting the contents of Amazon SQS messages.
For more information, see Data Keys in the AWS Key Management Service Developer Guide in the AWS
Encryption SDK Developer Guide.
110
Amazon Simple Queue Service Developer Guide
Data encryption
The length of time, in seconds, for which Amazon SQS can reuse a data key to encrypt or decrypt
messages before calling AWS KMS again. An integer representing seconds, between 60 seconds (1
minute) and 86,400 seconds (24 hours). The default is 300 (5 minutes). For more information, see
Understanding the data key reuse period (p. 114).
Note
In the unlikely event of being unable to reach AWS KMS, Amazon SQS continues to use the
cached data key until a connection is reestablished.
KMS key ID
The alias, alias ARN, key ID, or key ARN of an AWS managed KMS key or a custom KMS key—in your
account or in another account. While the alias of the AWS managed KMS key for Amazon SQS is
always alias/aws/sqs, the alias of a custom KMS key can, for example, be alias/MyAlias. You
can use these KMS keys to protect the messages in Amazon SQS queues.
Note
Keep the following in mind:
• If you don't specify a custom KMS key, Amazon SQS uses the AWS managed KMS key for
Amazon SQS.
• The first time you use the AWS Management Console to specify the AWS managed KMS
key for Amazon SQS for a queue, AWS KMS creates the AWS managed KMS key for
Amazon SQS.
• Alternatively, the first time you use the SendMessage or SendMessageBatch action on
a queue with SSE enabled, AWS KMS creates the AWS managed KMS key for Amazon SQS.
You can create KMS keys, define the policies that control how KMS keys can be used, and audit
KMS key usage using the Customer managed keys section of the AWS KMS console or the
CreateKey AWS KMS action. For more information, see KMS keys and Creating Keys in the AWS Key
Management Service Developer Guide. For more examples of KMS key identifiers, see KeyId in the
AWS Key Management Service API Reference. For information about finding KMS key identifiers, see
Find the Key ID and ARN in the AWS Key Management Service Developer Guide.
Important
There are additional charges for using AWS KMS. For more information, see Estimating AWS
KMS costs (p. 115) and AWS Key Management Service Pricing.
Envelope Encryption
The security of your encrypted data depends in part on protecting the data key that can decrypt it.
Amazon SQS uses the KMS key to encrypt the data key and then the encrypted data key is stored
with the encrypted message. This practice of using a KMS key to encrypt data keys is known as
envelope encryption.
For more information, see Envelope Encryption in the AWS Encryption SDK Developer Guide.
Key management
Amazon SQS integrates with the AWS Key Management Service (KMS) to manage KMS keys for server-
side encryption (SSE). See Encryption at rest (p. 109) for SSE information and key management
definitions. Amazon SQS uses KMS keys to validate and secure the data keys that encrypt and decrypt
the messages. The following sections provide information about working with KMS keys and data keys in
the Amazon SQS service.
Topics
• Configuring AWS KMS permissions (p. 112)
111
Amazon Simple Queue Service Developer Guide
Data encryption
For a customer managed KMS key, you must configure the key policy to add permissions for each queue
producer and consumer. To do this, you name the producer and consumer as users in the KMS key policy.
For more information about AWS KMS permissions, see AWS KMS resources and operations or AWS KMS
API permissions reference in the AWS Key Management Service Developer Guide.
Alternatively, you can specify the required permissions in an IAM policy assigned to the principals that
produce and consume encrypted messages. For more information, see Using IAM Policies with AWS KMS
in the AWS Key Management Service Developer Guide.
Note
While you can configure global permissions to send to and receive from Amazon SQS, AWS KMS
requires explicitly naming the full ARN of KMS keys in specific regions in the Resource section
of an IAM policy.
Several AWS services act as event sources that can send events to Amazon SQS queues. To allow these
event sources to work with encrypted queues, you must create a customer managed KMS key and add
permissions in the key policy for the service to use the required AWS KMS API methods. Perform the
following steps to configure the permissions.
1. Create a customer managed KMS key. For more information, see Creating Keys in the AWS Key
Management Service Developer Guide.
2. To allow the AWS service event source to use the kms:GenerateDataKey and kms:Decrypt API
methods, add the following statement to the KMS key key policy.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Service": "service.amazonaws.com"
},
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt"
],
"Resource": "*"
}]
}
Replace "service" in the above example with the Service name of the event source. Event sources
include the following services.
112
Amazon Simple Queue Service Developer Guide
Data encryption
When the data key reuse period (p. 114) expires, the producer's next call to SendMessage or
SendMessageBatch also triggers calls to kms:GenerateDataKey and kms:Decrypt. The call to
kms:Decrypt is to verify the integrity of the new data key before using it. Therefore, the producer must
have the kms:GenerateDataKey and kms:Decrypt permissions for the KMS key.
Add the following statement to the IAM policy of the producer. Remember to use the correct ARN values
for the key resource and the queue resource.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt"
],
"Resource": "arn:aws:kms:us-
east-2:123456789012:key/1234abcd-12ab-34cd-56ef-1234567890ab"
}, {
"Effect": "Allow",
"Action": [
"sqs:SendMessage"
],
"Resource": "arn:aws:sqs:*:123456789012:MyQueue"
}]
}
When the data key reuse period expires, the consumer's next call to ReceiveMessage also triggers a
call to kms:Decrypt, to verify the integrity of the new data key before using it. Therefore, the consumer
must have the kms:Decrypt permission for any KMS key that is used to encrypt the messages in the
specified queue. If the queue acts as a dead-letter queue (p. 45), the consumer must also have the
kms:Decrypt permission for any KMS key that is used to encrypt the messages in the source queue. Add
the following statement to the IAM policy of the consumer. Remember to use the correct ARN values for
the key resource and the queue resource.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"kms:Decrypt"
],
"Resource": "arn:aws:kms:us-
east-2:123456789012:key/1234abcd-12ab-34cd-56ef-1234567890ab"
}, {
"Effect": "Allow",
"Action": [
113
Amazon Simple Queue Service Developer Guide
Data encryption
"sqs:ReceiveMessage"
],
"Resource": "arn:aws:sqs:*:123456789012:MyQueue"
}]
}
In this KMS key policy, a specific resource from service that is owned by account 111122223333 is
allowed to call KMS for Decrypt and GenerateDataKey actions, which occur during SSE usage of
Amazon SQS.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Service": "<replaceable>service</replaceable>.amazonaws.com"
},
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt"
],
"Resource": "*",
"Condition": {
"ArnEquals": {
"aws:SourceArn": [
"arn:aws:service::111122223333:resource"
]
}
}
}]
}
When using SSE enabled Amazon SQS queues, the following services support aws:SourceArn:
• Amazon SNS
• Amazon S3
• CloudWatch Events
• AWS Lambda
• CodeBuild
• Amazon Connect Customer Profiles
• AWS Auto Scaling
• Amazon Chime
• A shorter reuse period provides better security but results in more calls to AWS KMS, which might incur
charges beyond the Free Tier.
114
Amazon Simple Queue Service Developer Guide
Data encryption
• Although the data key is cached separately for encryption and for decryption, the reuse period applies
to both copies of the data key.
• When the data key reuse period ends, the next call to SendMessage or SendMessageBatch typically
triggers a call to the AWS KMS GenerateDataKey method to get a new data key. Also, the next calls
to SendMessage and ReceiveMessage will each trigger a call to AWS KMS Decrypt to verify the
integrity of the data key before using it.
• Principals (AWS accounts or IAM users) don't share data keys (messages sent by unique principals
always get unique data keys). Thus, the volume of calls to AWS KMS is a multiple of the number of
unique principals in use during the data key reuse period:
To calculate the number of API requests (R) per queue, use the following formula:
R = B / D * (2 * P + C)
P is the number of producing principals that send to the Amazon SQS queue.
C is the number of consuming principals that receive from the Amazon SQS queue.
Important
In general, producing principals incur double the cost of consuming principals. For more
information, see Understanding the data key reuse period (p. 114).
If the producer and consumer have different IAM users, the cost increases.
The following are example calculations. For exact pricing information, see AWS Key Management Service
Pricing.
Example 1: Calculating the number of AWS KMS API calls for 2 principals and 1 queue
Example 2: Calculating the number of AWS KMS API calls for multiple producers and consumers
and 2 queues
115
Amazon Simple Queue Service Developer Guide
Internetwork traffic privacy
Topics
• Amazon Virtual Private Cloud endpoints for Amazon SQS (p. 116)
• Creating an Amazon VPC endpoint policy for Amazon SQS (p. 117)
Amazon VPC lets you launch AWS resources in a custom virtual network. You can use a VPC to control
your network settings, such as the IP address range, subnets, route tables, and network gateways. For
more information about VPCs, see the Amazon VPC User Guide.
To connect your VPC to Amazon SQS, you must first define an interface VPC endpoint, which lets you
connect your VPC to other AWS services. The endpoint provides reliable, scalable connectivity to
Amazon SQS without requiring an internet gateway, network address translation (NAT) instance, or
VPN connection. For more information, see Tutorial: Sending a message to an Amazon SQS queue from
Amazon Virtual Private Cloud (p. 97) and Example 5: Deny access if it isn't from a VPC endpoint (p. 141)
in this guide and Interface VPC Endpoints (AWS PrivateLink) in the Amazon VPC User Guide.
Important
• You can use Amazon Virtual Private Cloud only with HTTPS Amazon SQS endpoints.
• When you configure Amazon SQS to send messages from Amazon VPC, you must enable
private DNS and specify endpoints in the format sqs.us-east-2.amazonaws.com.
• Private DNS doesn't support legacy endpoints such as queue.amazonaws.com or us-
east-2.queue.amazonaws.com.
116
Amazon Simple Queue Service Developer Guide
Identity and access management
For more information, see Controlling Access to Services with VPC Endpoints in the Amazon VPC User
Guide
The following example VPC endpoint policy specifies that the IAM user MyUser is allowed to send
messages to the Amazon SQS queue MyQueue.
{
"Statement": [{
"Action": ["sqs:SendMessage"],
"Effect": "Allow",
"Resource": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
"Principal": {
"AWS": "arn:aws:iam:123456789012:user/MyUser"
}
}]
}
Note
The IAM user can still use other Amazon SQS API actions from outside the VPC. For more
information, see Example 5: Deny access if it isn't from a VPC endpoint (p. 141).
Topics
• Authentication (p. 118)
• Access control (p. 118)
• Overview of managing access in Amazon SQS (p. 119)
• Using identity-based policies with Amazon SQS (p. 123)
• Using custom policies with the Amazon SQS Access Policy Language (p. 131)
• Using temporary security credentials with Amazon SQS (p. 142)
• Amazon SQS API permissions: Actions and resource reference (p. 143)
117
Amazon Simple Queue Service Developer Guide
Authentication
Authentication
You can access AWS as any of the following types of identities:
• AWS account root user – When you first create an AWS account, you begin with a single sign-in
identity that has complete access to all AWS services and resources in the account. This identity is
called the AWS account root user and is accessed by signing in with the email address and password
that you used to create the account. We strongly recommend that you do not use the root user for
your everyday tasks, even the administrative ones. Instead, adhere to the best practice of using the
root user only to create your first IAM user. Then securely lock away the root user credentials and use
them to perform only a few account and service management tasks.
• IAM user – An IAM user is an identity within your AWS account that has specific custom permissions
(for example, permissions to create a queue in Amazon SQS). You can use an IAM user name and
password to sign in to secure AWS webpages like the AWS Management Console, AWS Discussion
Forums, or the AWS Support Center.
In addition to a user name and password, you can also generate access keys for each user. You can
use these keys when you access AWS services programmatically, either through one of the several
SDKs or by using the AWS Command Line Interface (CLI). The SDK and CLI tools use the access keys
to cryptographically sign your request. If you don’t use AWS tools, you must sign the request yourself.
Amazon SQS supports Signature Version 4, a protocol for authenticating inbound API requests. For
more information about authenticating requests, see Signature Version 4 signing process in the AWS
General Reference.
• IAM role – An IAM role is an IAM identity that you can create in your account that has specific
permissions. An IAM role is similar to an IAM user in that it is an AWS identity with permissions policies
that determine what the identity can and cannot do in AWS. However, instead of being uniquely
associated with one person, a role is intended to be assumable by anyone who needs it. Also, a role
does not have standard long-term credentials such as a password or access keys associated with it.
Instead, when you assume a role, it provides you with temporary security credentials for your role
session. IAM roles with temporary credentials are useful in the following situations:
• Federated user access – Instead of creating an IAM user, you can use existing identities from AWS
Directory Service, your enterprise user directory, or a web identity provider. These are known as
federated users. AWS assigns a role to a federated user when access is requested through an identity
provider. For more information about federated users, see Federated users and roles in the IAM User
Guide.
• AWS service access – A service role is an IAM role that a service assumes to perform actions on your
behalf. An IAM administrator can create, modify, and delete a service role from within IAM. For more
information, see Creating a role to delegate permissions to an AWS service in the IAM User Guide.
• Applications running on Amazon EC2 – You can use an IAM role to manage temporary credentials
for applications that are running on an EC2 instance and making AWS CLI or AWS API requests. This
is preferable to storing access keys within the EC2 instance. To assign an AWS role to an EC2 instance
and make it available to all of its applications, you create an instance profile that is attached to
the instance. An instance profile contains the role and enables programs that are running on the
EC2 instance to get temporary credentials. For more information, see Using an IAM role to grant
permissions to applications running on Amazon EC2 instances in the IAM User Guide.
Access control
Amazon SQS has its own resource-based permissions system that uses policies written in the same
language used for AWS Identity and Access Management (IAM) policies. This means that you can achieve
similar things with Amazon SQS policies and IAM policies.
Note
It is important to understand that all AWS accounts can delegate their permissions to users
under their accounts. Cross-account access allows you to share access to your AWS resources
118
Amazon Simple Queue Service Developer Guide
Overview
without having to manage additional users. For information about using cross-account access,
see Enabling Cross-Account Access in the IAM User Guide.
See Limitations of Custom Policies (p. 138) for further details on cross-content permissions
and condition keys within Amazon SQS custom policies.
When granting permissions, you specify what users get permissions, the resource they get permissions
for, and the specific actions that you want to allow on the resource.
Topics
• Amazon Simple Queue Service resource and operations (p. 119)
• Understanding resource ownership (p. 120)
• Managing access to resources (p. 120)
• Specifying policy elements: Actions, effects, resources, and principals (p. 123)
• Specifying conditions in a policy (p. 123)
Queue arn:aws:sqs:region:account_id:queue_name
• An ARN for a queue named my_queue in the US East (Ohio) region, belonging to AWS Account
123456789012:
arn:aws:sqs:us-east-2:123456789012:my_queue
• An ARN for a queue named my_queue in each of the different regions that Amazon SQS supports:
arn:aws:sqs:*:123456789012:my_queue
• An ARN that uses * or ? as a wildcard for the queue name. In the following examples, the ARN
matches all queues prefixed with my_prefix_:
arn:aws:sqs:*:123456789012:my_prefix_*
119
Amazon Simple Queue Service Developer Guide
Overview
You can get the ARN value for an existing queue by calling the GetQueueAttributes action. The value
of the QueueArn attribute is the ARN of the queue. For more information about ARNs, see IAM ARNs in
the IAM User Guide.
Amazon SQS provides a set of actions that work with the queue resource. For more information, see
Amazon SQS API permissions: Actions and resource reference (p. 143).
• If you use the root account credentials of your AWS account to create an Amazon SQS queue, your
AWS account is the owner of the resource (in Amazon SQS, the resource is the Amazon SQS queue).
• If you create an IAM user in your AWS account and grant permissions to create a queue to the user, the
user can create the queue. However, your AWS account (to which the user belongs) owns the queue
resource.
• If you create an IAM role in your AWS account with permissions to create an Amazon SQS queue,
anyone who can assume the role can create a queue. Your AWS account (to which the role belongs)
owns the queue resource.
Policies attached to an IAM identity are referred to as identity-based policies (IAM policies) and policies
attached to a resource are referred to as resource-based policies.
• Attach a permission policy to a user or a group in your account – To grant user permissions to create
an Amazon SQS queue, attach a permissions policy to a user or group that the user belongs to.
• Attach a permission policy to a user in another AWS account – To grant user permissions to create an
Amazon SQS queue, attach an Amazon SQS permissions policy to a user in another AWS account.
120
Amazon Simple Queue Service Developer Guide
Overview
• ListQueueTags
• RemovePermission
• SetQueueAttributes
• TagQueue
• UntagQueue
• Attach a permission policy to a role (grant cross-account permissions) – To grant cross-account
permissions, attach an identity-based permissions policy to an IAM role. For example, the AWS account
A administrator can create a role to grant cross-account permissions to AWS account B (or an AWS
service) as follows:
• The account A administrator creates an IAM role and attaches a permissions policy—that grants
permissions on resources in account A—to the role.
• The account A administrator attaches a trust policy to the role that identifies account B as the
principal who can assume the role.
• The account B administrator delegates the permission to assume the role to any users in account B.
This allows users in account B to create or access queues in account A.
Note
If you want to grant the permission to assume the role to an AWS service, the principal in
the trust policy can also be an AWS service principal.
For more information about using IAM to delegate permissions, see Access Management in the IAM User
Guide.
While Amazon SQS works with IAM policies, it has its own policy infrastructure. You can use an Amazon
SQS policy with a queue to specify which AWS Accounts have access to the queue. You can specify the
type of access and conditions (for example, a condition that grants permissions to use SendMessage,
ReceiveMessage if the request is made before December 31, 2010). The specific actions you can grant
permissions for are a subset of the overall list of Amazon SQS actions. When you write an Amazon SQS
policy and specify * to "allow all Amazon SQS actions," it means that a user can perform all actions in
this subset.
The following diagram illustrates the concept of one of these basic Amazon SQS policies that covers
the subset of actions. The policy is for queue_xyz, and it gives AWS Account 1 and AWS Account 2
permissions to use any of the allowed actions with the specified queue.
Note
The resource in the policy is specified as 123456789012/queue_xyz, where 123456789012 is
the AWS Account ID of the account that owns the queue.
121
Amazon Simple Queue Service Developer Guide
Overview
With the introduction of IAM and the concepts of Users and Amazon Resource Names (ARNs), a few things
have changed about SQS policies. The following diagram and table describe the changes.
For information about giving permissions to users in different accounts, see Tutorial: Delegate Access
Across AWS Accounts Using IAM Roles in the IAM User Guide.
The subset of actions included in * has expanded. For a list of allowed actions, see Amazon SQS API
permissions: Actions and resource reference (p. 143).
You can specify the resource using the Amazon Resource Name (ARN), the standard means of
specifying resources in IAM policies. For information about the ARN format for Amazon SQS queues, see
Amazon Simple Queue Service resource and operations (p. 119).
For example, according to the Amazon SQS policy in the preceding diagram, anyone who possesses the
security credentials for AWS Account 1 or AWS Account 2 can access queue_xyz. In addition, Users Bob
and Susan in your own AWS Account (with ID 123456789012) can access the queue.
Before the introduction of IAM, Amazon SQS automatically gave the creator of a queue full control over
the queue (that is, access to all of the possible Amazon SQS actions on that queue). This is no longer
true, unless the creator uses AWS security credentials. Any user who has permissions to create a queue
must also have permissions to use other Amazon SQS actions in order to do anything with the created
queues.
The following is an example policy that allows a user to use all Amazon SQS actions, but only with
queues whose names are prefixed with the literal string bob_queue_.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "sqs:*",
"Resource": "arn:aws:sqs:*:123456789012:bob_queue_*"
}]
}
122
Amazon Simple Queue Service Developer Guide
Using identity-based policies
For more information, see Using identity-based policies with Amazon SQS (p. 123), and Identities
(Users, Groups, and Roles) in the IAM User Guide.
• Resource – In a policy, you use an Amazon Resource Name (ARN) to identify the resource to which the
policy applies.
• Action – You use action keywords to identify resource actions that you want to allow or deny. For
example, the sqs:CreateQueue permission allows the user to perform the Amazon Simple Queue
Service CreateQueue action.
• Effect – You specify the effect when the user requests the specific action—this can be either allow
or deny. If you don't explicitly grant access to a resource, access is implicitly denied. You can also
explicitly deny access to a resource, which you might do to make sure that a user can't access it, even if
a different policy grants access.
• Principal – In identity-based policies (IAM policies), the user that the policy is attached to is the
implicit principal. For resource-based policies, you specify the user, account, service, or other entity
that you want to receive permissions (applies to resource-based policies only).
To learn more about Amazon SQS policy syntax and descriptions, see AWS IAM Policy Reference in the
IAM User Guide.
For a table of all Amazon Simple Queue Service actions and the resources that they apply to, see Amazon
SQS API permissions: Actions and resource reference (p. 143).
To express conditions, you use predefined condition keys. There are no condition keys specific to Amazon
SQS. However, there are AWS-wide condition keys that you can use with Amazon SQS. Currently, Amazon
SQS supports only a limited subset of the condition keys available in IAM. See the section called “API
permissions reference” (p. 143).
123
Amazon Simple Queue Service Developer Guide
Using identity-based policies
With the exception of ListQueues, all Amazon SQS actions support resource-level
permissions. For more information, see Amazon SQS API permissions: Actions and resource
reference (p. 143).
Topics
• Using Amazon SQS and IAM policies (p. 124)
• Permissions required to use the Amazon SQS console (p. 125)
• AWS managed (predefined) policies for Amazon SQS (p. 126)
• Basic examples of IAM policies for Amazon SQS (p. 126)
• Basic examples of Amazon SQS policies (p. 128)
For example, the following diagram shows an IAM policy and an Amazon SQS policy equivalent to it. The
IAM policy grants the rights to the Amazon SQS ReceiveMessage and SendMessage actions for the
queue called queue_xyz in your AWS Account, and the policy is attached to users named Bob and Susan
(Bob and Susan have the permissions stated in the policy). This Amazon SQS policy also gives Bob and
Susan rights to the ReceiveMessage and SendMessage actions for the same queue.
Note
This example shows simple policies without conditions. You can specify a particular condition in
either policy and get the same result.
There is one major difference between IAM and Amazon SQS policies: the Amazon SQS policy system lets
you grant permission to other AWS Accounts, whereas IAM doesn't.
It is up to you how you use both of the systems together to manage your permissions. The following
examples show how the two policy systems work together.
• In the first example, Bob has both an IAM policy and an Amazon SQS policy that apply to his account.
The IAM policy grants his account permission for the ReceiveMessage action on queue_xyz,
whereas the Amazon SQS policy gives his account permission for the SendMessage action on the
same queue. The following diagram illustrates the concept.
124
Amazon Simple Queue Service Developer Guide
Using identity-based policies
If Bob sends a ReceiveMessage request to queue_xyz, the IAM policy allows the action. If Bob sends
a SendMessage request to queue_xyz, the Amazon SQS policy allows the action.
• In the second example, Bob abuses his access to queue_xyz, so it becomes necessary to remove
his entire access to the queue. The easiest thing to do is to add a policy that denies him access to all
actions for the queue. This policy overrides the other two because an explicit deny always overrides an
allow. For more information about policy evaluation logic, see Using custom policies with the Amazon
SQS Access Policy Language (p. 131). The following diagram illustrates the concept.
You can also add an additional statement to the Amazon SQS policy that denies Bob any type of access
to the queue. It has the same effect as adding an IAM policy that denies Bob access to the queue. For
examples of policies that cover Amazon SQS actions and resources, see Basic examples of Amazon SQS
policies (p. 128). For more information about writing Amazon SQS policies, see Using custom policies
with the Amazon SQS Access Policy Language (p. 131).
125
Amazon Simple Queue Service Developer Guide
Using identity-based policies
the permission to call the ListQueues action to be able to list queues, or the permission to call the
CreateQueue action to be able to create queues. In addition to Amazon SQS permissions, to subscribe
an Amazon SQS queue to an Amazon SNS topic, the console also requires permissions for Amazon SNS
actions.
If you create an IAM policy that is more restrictive than the minimum required permissions, the console
might not function as intended for users with that IAM policy.
You don't need to allow minimum console permissions for users that make calls only to the AWS CLI or
Amazon SQS actions.
The following AWS managed policies, which you can attach to users in your account, are for using
Amazon SQS management APIs:
• AmazonSQSFullAccess – Grants full access to Amazon SQS queues using the AWS Management
Console.
• AmazonSQSReadOnlyAccess – Grants read-only access to Amazon SQS queues using the AWS
Management Console.
You can search and review available policies on the IAM console. You can also create your own custom
IAM policies to allow permissions for Amazon SQS actions and queues. You can attach these custom
policies to the IAM users or groups that require permissions.
Amazon SQS doesn't automatically grant the creator of a queue permissions to use the queue. Therefore,
we must explicitly grant Bob permissions to use all Amazon SQS actions in addition to CreateQueue
action in the IAM policy.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "sqs:*",
"Resource": "arn:aws:sqs:*:123456789012:alice_queue_*"
}]
}
126
Amazon Simple Queue Service Developer Guide
Using identity-based policies
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:*:123456789012:MyCompanyQueue"
}]
}
You can use * instead of SendMessage to grant the following actions to a principal on a shared
queue: ChangeMessageVisibility, DeleteMessage, GetQueueAttributes, GetQueueUrl,
ReceiveMessage, and SendMessage.
Note
Although * includes access provided by other permission types, Amazon SQS considers
permissions separately. For example, it is possible to grant both * and SendMessage
permissions to a user, even though a * includes the access provided by SendMessage.
This concept also applies when you remove a permission. If a principal has only a * permission,
requesting to remove a SendMessage permission doesn't leave the principal with an everything-
but permission. Instead, the request has no effect, because the principal doesn't possess an
explicit SendMessage permission. To leave the principal with only the ReceiveMessage
permission, first add the ReceiveMessage permission and then remove the * permission.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "sqs:GetQueueAttributes",
"Resource": "*"
}]
}
127
Amazon Simple Queue Service Developer Guide
Using identity-based policies
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:*:123456789012:WidgetPartnerQueue"
}]
}
You can use the console to verify the effects of each policy as you attach the policy to the user. Initially,
the user doesn't have permissions and won't be able to do anything in the console. As you attach policies
to the user, you can verify that the user can perform various actions in the console.
Note
We recommend that you use two browser windows: one to grant permissions and the other to
sign into the AWS Management Console using the user's credentials to verify permissions as you
grant them to the user.
{
"Version": "2012-10-17",
"Id": "Queue1_Policy_UUID",
"Statement": [{
"Sid":"Queue1_SendMessage",
"Effect": "Allow",
"Principal": {
"AWS": [
"111122223333"
]
},
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:us-east-2:444455556666:queue1"
}]
}
{
"Version": "2012-10-17",
"Id": "Queue1_Policy_UUID",
"Statement": [{
"Sid":"Queue1_Send_Receive",
"Effect": "Allow",
"Principal": {
"AWS": [
"111122223333"
]
},
"Action": [
"sqs:SendMessage",
128
Amazon Simple Queue Service Developer Guide
Using identity-based policies
"sqs:ReceiveMessage"
],
"Resource": "arn:aws:sqs:*:444455556666:queue1"
}]
}
{
"Version": "2012-10-17",
"Id": "Queue1_Policy_UUID",
"Statement": [{
"Sid":"Queue1_AllActions",
"Effect": "Allow",
"Principal": {
"AWS": [
"111122223333",
"444455556666"
]
},
"Action": "sqs:*",
"Resource": "arn:aws:sqs:us-east-2:123456789012:queue1"
}]
}
• AddPermission
• CreateQueue
• DeleteQueue
• ListQueues
• ListQueueTags
• RemovePermission
• SetQueueAttributes
• TagQueue
• UntagQueue
{
"Version": "2012-10-17",
"Id": "Queue1_Policy_UUID",
"Statement": [{
"Sid":"Queue1_AllActions",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::111122223333:role/role1",
"arn:aws:iam::111122223333:user/username1"
129
Amazon Simple Queue Service Developer Guide
Using identity-based policies
]
},
"Action": "sqs:*",
"Resource": "arn:aws:sqs:us-east-2:123456789012:queue1"
}]
}
{
"Version": "2012-10-17",
"Id": "Queue1_Policy_UUID",
"Statement": [{
"Sid":"Queue1_AnonymousAccess_ReceiveMessage",
"Effect": "Allow",
"Principal": "*",
"Action": "sqs:ReceiveMessage",
"Resource": "arn:aws:sqs:*:111122223333:queue1"
}]
}
{
"Version": "2012-10-17",
"Id": "Queue1_Policy_UUID",
"Statement": [{
"Sid":"Queue1_AnonymousAccess_ReceiveMessage_TimeLimit",
"Effect": "Allow",
"Principal": "*",
"Action": "sqs:ReceiveMessage",
"Resource": "arn:aws:sqs:*:111122223333:queue1",
"Condition" : {
"DateGreaterThan" : {
"aws:CurrentTime":"2009-01-31T12:00Z"
},
"DateLessThan" : {
"aws:CurrentTime":"2009-01-31T15:00Z"
}
}
}]
}
{
"Version": "2012-10-17",
"Id": "Queue1_Policy_UUID",
"Statement": [{
"Sid":"Queue1_AnonymousAccess_AllActions_AllowlistIP",
130
Amazon Simple Queue Service Developer Guide
Using custom policies with the Access Policy Language
"Effect": "Allow",
"Principal": "*",
"Action": "sqs:*",
"Resource": "arn:aws:sqs:*:111122223333:queue1",
"Condition" : {
"IpAddress" : {
"aws:SourceIp":"192.168.143.0/24"
}
}
}]
}
Example 8: Allowlist and blocklist permissions for users in different CIDR ranges
The following example policy has two statements:
• The first statement grants all users (anonymous users) in the 192.168.143.0/24 CIDR range
(except for 192.168.143.188) permission to use the SendMessage action for the queue named
111122223333/queue1.
• The second statement blocks all users (anonymous users) in the 10.1.2.0/24 CIDR range from using
the queue.
{
"Version": "2012-10-17",
"Id": "Queue1_Policy_UUID",
"Statement": [{
"Sid":"Queue1_AnonymousAccess_SendMessage_IPLimit",
"Effect": "Allow",
"Principal": "*",
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:*:111122223333:queue1",
"Condition" : {
"IpAddress" : {
"aws:SourceIp":"192.168.143.0/24"
},
"NotIpAddress" : {
"aws:SourceIp":"192.168.143.188/32"
}
}
}, {
"Sid":"Queue1_AnonymousAccess_AllActions_IPLimit_Deny",
"Effect": "Deny",
"Principal": "*",
"Action": "sqs:*",
"Resource": "arn:aws:sqs:*:111122223333:queue1",
"Condition" : {
"IpAddress" : {
"aws:SourceIp":"10.1.2.0/24"
}
}
}]
}
131
Amazon Simple Queue Service Developer Guide
Using custom policies with the Access Policy Language
If you want to explicitly deny or allow access based on more specific conditions (such as the time the
request comes in or the IP address of the requester), you need to write your own Amazon SQS policies
and upload them to the AWS system using the Amazon SQS SetQueueAttributes action.
Topics
• Amazon SQS access control architecture (p. 132)
• Amazon SQS access control process workflow (p. 133)
• Amazon SQS Access Policy Language key concepts (p. 133)
• Amazon SQS Access Policy Language evaluation logic (p. 135)
• Relationships between explicit and default denials in the Amazon SQS Access Policy
Language (p. 137)
• Limitations of Custom Policies (p. 138)
• Custom Amazon SQS Access Policy Language examples (p. 139)
Your resources contained within the AWS service (for example, Amazon SQS queues).
Your policies. It is a good practice to have one policy per resource. The AWS service provides an API
you use to upload and manage your policies.
The access policy language evaluation code. This is the set of code within the AWS service that
evaluates incoming requests against the applicable policies and determines whether the requester is
allowed access to the resource.
132
Amazon Simple Queue Service Developer Guide
Using custom policies with the Access Policy Language
You upload your policy to AWS. The AWS service provides an API that you use to upload your policies.
For example, you use the Amazon SQS SetQueueAttributes action to upload a policy for a particular
Amazon SQS queue.
Amazon SQS examines all available Amazon SQS policies and determines which ones are applicable.
Amazon SQS evaluates the policies and determines whether the requester is allowed to use your
queue.
Based on the policy evaluation result, Amazon SQS either returns an Access denied error to the
requester or continues to process the request.
Allow
The result of a Statement (p. 134) that has Effect (p. 134) set to allow.
Action
The activity that the Principal (p. 134) has permission to perform, typically a request to AWS.
Default-deny
The result of a Statement (p. 134) that has no Allow (p. 133) or Explicit-deny (p. 134) settings.
Condition
Any restriction or detail about a Permission (p. 134). Typical conditions are related to date and
time and IP addresses.
133
Amazon Simple Queue Service Developer Guide
Using custom policies with the Access Policy Language
Effect
The result that you want the Statement (p. 134) of a Policy (p. 134) to return at evaluation
time. You specify the deny or allow value when you write the policy statement. There can be three
possible results at policy evaluation time: Default-deny (p. 133), Allow (p. 133), and Explicit-
deny (p. 134).
Explicit-deny
The result of a Statement (p. 134) that has Effect (p. 134) set to deny.
Evaluation
The process that Amazon SQS uses to determine whether an incoming request should be denied or
allowed based on a Policy (p. 134).
Issuer
The user who writes a Policy (p. 134) to grant permissions to a resource. The issuer, by definition
is always the resource owner. AWS doesn't permit Amazon SQS users to create policies for resources
they don't own.
Key
The concept of allowing or disallowing access to a resource using a Condition (p. 133) and a
Key (p. 134).
Policy
The document that acts as a container for one or more statements (p. 134).
Amazon SQS uses the policy to determine whether to grant access to a user for a resource.
Principal
The user who receives Permission (p. 134) in the Policy (p. 134).
Resource
The object that the Principal (p. 134) requests access to.
Statement
The formal description of a single permission, written in the access policy language as part of a
broader Policy (p. 134) document.
134
Amazon Simple Queue Service Developer Guide
Using custom policies with the Access Policy Language
Requester
The user who sends a request for access to a Resource (p. 134).
• By default, all requests to use your resource coming from anyone but you are denied.
• An Allow (p. 133) overrides any Default-deny (p. 133).
• An Explicit-deny (p. 134) overrides any allow.
• The order in which the policies are evaluated isn't important.
The following diagram describes in detail how Amazon SQS evaluates decisions about access
permissions.
135
Amazon Simple Queue Service Developer Guide
Using custom policies with the Access Policy Language
The enforcement code evaluates all the policies that are applicable to the request (based on the
resource, principal, action, and conditions). The order in which the enforcement code evaluates the
policies isn't important.
The enforcement code looks for an explicit-deny instruction that can apply to the request. If it finds
even one, the enforcement code returns a decision of deny and the process finishes.
136
Amazon Simple Queue Service Developer Guide
Using custom policies with the Access Policy Language
If no explicit-deny instruction is found, the enforcement code looks for any allow instructions that
can apply to the request. If it finds even one, the enforcement code returns a decision of allow and the
process finishes (the service continues to process the request).
If no allow instruction is found, then the final decision is deny (because there is no explicit-deny or
allow, this is considered a default-deny).
If a condition in a statement isn't met, the request results in a default-deny. If all conditions in a
statement are met, the request results in either an Allow (p. 133) or an Explicit-deny (p. 134) based
on the value of the Effect (p. 134) element of the policy. Policies don't specify what to do if a condition
isn't met, so the default result in this case is a default-deny. For example, you want to prevent requests
that come from Antarctica. You write Policy A1 that allows a request only if it doesn't come from
Antarctica. The following diagram illustrates the Amazon SQS policy.
If a user sends a request from the U.S., the condition is met (the request isn't from Antarctica), and the
request results in an allow. However, if a user sends a request from Antarctica, the condition isn't met
and the request defaults to a default-deny. You can change the result to an explicit-deny by writing
Policy A2 that explicitly denies a request if it comes from Antarctica. The following diagram illustrates
the policy.
If a user sends a request from Antarctica, the condition is met and the request results in an explicit-deny.
137
Amazon Simple Queue Service Developer Guide
Using custom policies with the Access Policy Language
The distinction between a default-deny and an explicit-deny is important because an allow can
overwrite the former but not the latter. For example, Policy B allows requests if they arrive on June 1,
2010. The following diagram compares combining this policy with Policy A1 and Policy A2.
In Scenario 1, Policy A1 results in a default-deny and Policy B results in an allow because the policy
allows requests that come in on June 1, 2010. The allow from Policy B overrides the default-deny from
Policy A1, and the request is allowed.
In Scenario 2, Policy B2 results in an explicit-deny and Policy B results in an allow. The explicit-deny
from Policy A2 overrides the allow from Policy B, and the request is denied.
• AddPermission
138
Amazon Simple Queue Service Developer Guide
Using custom policies with the Access Policy Language
• CreateQueue
• DeleteQueue
• ListQueues
• ListQueueTags
• RemovePermission
• SetQueueAttributes
• TagQueue
• UntagQueue
Condition keys
Currently, Amazon SQS supports only a limited subset of the condition keys available in IAM. For more
information, see Amazon SQS API permissions: Actions and resource reference (p. 143).
{
"Version": "2012-10-17",
"Id": "UseCase1",
"Statement" : [{
"Sid": "1",
"Effect": "Allow",
"Principal": {
"AWS": [
"111122223333"
]
},
"Action": [
"sqs:SendMessage",
"sqs:ReceiveMessage"
],
"Resource": "arn:aws:sqs:us-east-2:444455556666:queue2"
}]
}
{
"Version": "2012-10-17",
"Id": "UseCase2",
"Statement" : [{
"Sid": "1",
"Effect": "Allow",
139
Amazon Simple Queue Service Developer Guide
Using custom policies with the Access Policy Language
"Principal": {
"AWS": [
"111122223333",
"444455556666"
]
},
"Action": [
"sqs:SendMessage",
"sqs:ReceiveMessage"
],
"Resource": "arn:aws:sqs:us-east-2:444455556666:queue2",
"Condition": {
"DateLessThan": {
"AWS:CurrentTime": "2009-06-30T12:00Z"
}
}
}]
}
{
"Version": "2012-10-17",
"Id": "UseCase3",
"Statement" : [{
"Sid": "1",
"Effect": "Allow",
"Principal": {
"AWS": [
"111122223333"
]
},
"Action": [
"sqs:SendMessage",
"sqs:ReceiveMessage"
],
"Resource": "arn:aws:sqs:us-east-2:444455556666:queue2",
"Condition": {
"DateLessThan": {
"AWS:CurrentTime": "2009-06-30T12:00Z"
},
"IpAddress": {
"AWS:SourceIp": "203.0.113.0/24"
}
}
}]
}
140
Amazon Simple Queue Service Developer Guide
Using custom policies with the Access Policy Language
{
"Version": "2012-10-17",
"Id": "UseCase4",
"Statement" : [{
"Sid": "1",
"Effect": "Deny",
"Principal": {
"AWS": [
"111122223333"
]
},
"Action": [
"sqs:SendMessage",
"sqs:ReceiveMessage"
],
"Resource": "arn:aws:sqs:us-east-2:444455556666:queue2"
}]
}
• The aws:sourceVpce condition doesn't require an ARN for the VPC endpoint resource, only
the VPC endpoint ID.
• You can modify the following example to restrict all actions to a specific VPC endpoint by
denying all Amazon SQS actions (sqs:*) in the second statement. However, such a policy
statement would stipulate that all actions (including administrative actions needed to modify
queue permissions) must be made through the specific VPC endpoint defined in the policy,
potentially preventing the user from modifying queue permissions in the future.
{
"Version": "2012-10-17",
"Id": "UseCase5",
"Statement": [{
"Sid": "1",
"Effect": "Allow",
"Principal": {
"AWS": [
"111122223333"
]
},
"Action": [
"sqs:SendMessage",
"sqs:ReceiveMessage"
],
"Resource": "arn:aws::sqs:us-east-2:111122223333:queue1"
},
{
"Sid": "2",
"Effect": "Deny",
"Principal": "*",
"Action": [
"sqs:SendMessage",
"sqs:ReceiveMessage"
],
141
Amazon Simple Queue Service Developer Guide
Using temporary security credentials
"Resource": "arn:aws::sqs:us-east-2:111122223333:queue1",
"Condition": {
"StringNotEquals": {
"aws:sourceVpce": "vpce-1a2b3c4d"
}
}
}
]
}
You can use these temporary security credentials to make requests to Amazon SQS. The API libraries
compute the necessary signature value using those credentials to authenticate your request. If you send
requests using expired credentials, Amazon SQS denies the request.
Note
You can't set a policy based on temporary credentials.
Prerequisites
1. Use IAM to create temporary security credentials:
• Security token
• Access Key ID
• Secret Access Key
2. Prepare your string to sign with the temporary Access Key ID and the security token.
3. Use the temporary Secret Access Key instead of your own Secret Access Key to sign your Query API
request.
Note
When you submit the signed Query API request, use the temporary Access Key ID instead of your
own Access Key ID and to include the security token. For more information about IAM support
for temporary security credentials, see Granting Temporary Access to Your AWS Resources in the
IAM User Guide.
IAM returns a security token, an Access Key ID, and a Secret Access Key.
2. Prepare your query using the temporary Access Key ID instead of your own Access Key ID and include
the security token. Sign your request using the temporary Secret Access Key instead of your own.
3. Submit your signed query string with the temporary Access Key ID and the security token.
142
Amazon Simple Queue Service Developer Guide
API permissions reference
The following example demonstrates how to use temporary security credentials to authenticate an
Amazon SQS request. The structure of AUTHPARAMS depends on the signature of the API request.
For more information, see Signing AWS API Requests in the Amazon Web Services General Reference.
https://sqs.us-east-2.amazonaws.com/
?Action=CreateQueue
&DefaultVisibilityTimeout=40
&QueueName=MyQueue
&Attribute.1.Name=VisibilityTimeout
&Attribute.1.Value=40
&Expires=2020-12-18T22%3A52%3A43PST
&SecurityToken=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
&AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE
&Version=2012-11-05
&AUTHPARAMS
The following example uses temporary security credentials to send two messages using the
SendMessageBatch action.
https://sqs.us-east-2.amazonaws.com/
?Action=SendMessageBatch
&SendMessageBatchRequestEntry.1.Id=test_msg_001
&SendMessageBatchRequestEntry.1.MessageBody=test%20message%20body%201
&SendMessageBatchRequestEntry.2.Id=test_msg_002
&SendMessageBatchRequestEntry.2.MessageBody=test%20message%20body%202
&SendMessageBatchRequestEntry.2.DelaySeconds=60
&Expires=2020-12-18T22%3A52%3A43PST
&SecurityToken=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
&AWSAccessKeyId=AKIAI44QH8DHBEXAMPLE
&Version=2012-11-05
&AUTHPARAMS
Specify the actions in the policy's Action field, and the resource value in the policy's Resource field. To
specify an action, use the sqs: prefix followed by the action name (for example, sqs:CreateQueue).
Currently, Amazon SQS supports only a limited subset of the condition keys available in IAM:
• aws:CurrentTime
• aws:EpochTime
• aws:SecureTransport
• aws:SourceAccount
• aws:SourceArn
Note
This condition ensures that AWS services grant access only on behalf of resources that your
AWS account owns. You can't specify the ARN of an IAM role as source ARN, because an IAM
role is neither a source nor a service.
143
Amazon Simple Queue Service Developer Guide
API permissions reference
• aws:SourceIP
• aws:UserAgent
• aws:MultiFactorAuthAge
• aws:MultiFactorAuthPresent
• aws:PrincipalOrgID
• aws:RequestTag
• aws:sourceVpce
• aws:TagKeys
• aws:TokenAge
Amazon Simple Queue Service API and required permissions for actions
AddPermission
Action(s): sqs:AddPermission
Resource: arn:aws:sqs:region:account_id:queue_name
ChangeMessageVisibility
Action(s): sqs:ChangeMessageVisibility
Resource: arn:aws:sqs:region:account_id:queue_name
ChangeMessageVisibilityBatch
Action(s): sqs:ChangeMessageVisibilityBatch
Resource: arn:aws:sqs:region:account_id:queue_name
CreateQueue
Action(s): sqs:CreateQueue
Resource: arn:aws:sqs:region:account_id:queue_name
DeleteMessage
Action(s): sqs:DeleteMessage
Resource: arn:aws:sqs:region:account_id:queue_name
DeleteMessageBatch
Action(s): sqs:DeleteMessageBatch
Resource: arn:aws:sqs:region:account_id:queue_name
DeleteQueue
Action(s): sqs:DeleteQueue
Resource: arn:aws:sqs:region:account_id:queue_name
GetQueueAttributes
Action(s): sqs:GetQueueAttributes
Resource: arn:aws:sqs:region:account_id:queue_name
GetQueueUrl
Action(s): sqs:GetQueueUrl
Resource: arn:aws:sqs:region:account_id:queue_name
144
Amazon Simple Queue Service Developer Guide
Logging and monitoring
ListDeadLetterSourceQueues
Action(s): sqs:ListDeadLetterSourceQueues
Resource: arn:aws:sqs:region:account_id:queue_name
ListQueues
Action(s): sqs:ListQueues
Resource: arn:aws:sqs:region:account_id:queue_name
ListQueueTags
Action(s): sqs:ListQueueTags
Resource: arn:aws:sqs:region:account_id:queue_name
PurgeQueue
Action(s): sqs:PurgeQueue
Resource: arn:aws:sqs:region:account_id:queue_name
ReceiveMessage
Action(s): sqs:ReceiveMessage
Resource: arn:aws:sqs:region:account_id:queue_name
RemovePermission
Action(s): sqs:RemovePermission
Resource: arn:aws:sqs:region:account_id:queue_name
SendMessage and SendMessageBatch
Action(s): sqs:SendMessage
Resource: arn:aws:sqs:region:account_id:queue_name
SetQueueAttributes
Action(s): sqs:SetQueueAttributes
Resource: arn:aws:sqs:region:account_id:queue_name
TagQueue
Action(s): sqs:TagQueue
Resource: arn:aws:sqs:region:account_id:queue_name
UntagQueue
Action(s): sqs:UntagQueue
Resource: arn:aws:sqs:region:account_id:queue_name
Topics
• Logging Amazon SQS API calls using AWS CloudTrail (p. 146)
• Monitoring Amazon SQS queues using CloudWatch (p. 149)
145
Amazon Simple Queue Service Developer Guide
Logging API calls using CloudTrail
Using the information that CloudTrail collects, you can identify a specific request to an Amazon SQS API,
the IP address of the requester, the requester's identity, the date and time of the request, and so on. If
you configure a trail, you can enable continuous delivery of CloudTrail events to an Amazon S3 bucket.
If you don't configure a trail, you can view the most recent events in the event history in the CloudTrail
console. For more information, see Overview for Creating a Trail in the AWS CloudTrail User Guide.
A trail allows CloudTrail to deliver log files to an Amazon S3 bucket. You can create a trail to keep
an ongoing record of events in your AWS account. By default, when you create a trail using the AWS
Management Console, the trail applies to all AWS Regions. The trail logs events from all AWS Regions
and delivers log files to the specified Amazon S3 bucket. You can also configure other AWS services to
further analyze and act on the event data collected in CloudTrail logs. For more information, see the
following topics in the AWS CloudTrail User Guide:
• AddPermission
• CreateQueue
• DeleteQueue
• PurgeQueue
• RemovePermission
• SetQueueAttributes
• TagQueue
• UntagQueue
Every event or log entry contains information about the requester. This information helps you determine
the following:
146
Amazon Simple Queue Service Developer Guide
Logging API calls using CloudTrail
For more information, see CloudTrail userIdentity Element in the AWS CloudTrail User Guide.
AddPermission
The following example shows a CloudTrail log entry for an AddPermission API call.
{
"Records": [
{
"eventVersion": "1.06",
"userIdentity": {
"type": "IAMUser",
"principalId": "AKIAI44QH8DHBEXAMPLE",
"arn": "arn:aws:iam::123456789012:user/Alice",
"accountId": "123456789012",
"accessKeyId": "AKIAIOSFODNN7EXAMPLE",
"userName": "Alice"
},
"eventTime": "2018-06-28T22:23:46Z",
"eventSource": "sqs.amazonaws.com",
"eventName": "AddPermission",
"awsRegion": "us-east-2",
"sourceIPAddress": "203.0.113.0",
"userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0",
"requestParameters": {
"actions": [
"SendMessage"
],
"AWSAccountIds": [
"123456789012"
],
"label": "MyLabel",
"queueUrl": "https://sqs.us-east-2.amazon.com/123456789012/MyQueue"
},
"responseElements": null,
"requestID": "123abcde-f4gh-50ij-klmn-60o789012p30",
"eventID": "0987g654-32f1-09e8-d765-c4f3fb2109fa"
}
]
}
CreateQueue
The following example shows a CloudTrail log entry for a CreateQueue API call.
{
"Records": [
{
"eventVersion": "1.06",
"userIdentity": {
"type": "IAMUser",
"principalId": "AKIAI44QH8DHBEXAMPLE",
"arn": "arn:aws:iam::123456789012:user/Alejandro",
"accountId": "123456789012",
147
Amazon Simple Queue Service Developer Guide
Logging API calls using CloudTrail
"accessKeyId": "AKIAIOSFODNN7EXAMPLE",
"userName": "Alejandro"
},
"eventTime": "2018-06-28T22:23:46Z",
"eventSource": "sqs.amazonaws.com",
"eventName": "CreateQueue",
"awsRegion": "us-east-2",
"sourceIPAddress": "203.0.113.1",
"userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0",
"requestParameters": {
"queueName": "MyQueue"
},
"responseElements": {
"queueUrl": "https://sqs.us-east-2.amazon.com/123456789012/MyQueue"
},
"requestID": "123abcde-f4gh-50ij-klmn-60o789012p30",
"eventID": "0987g654-32f1-09e8-d765-c4f3fb2109fa"
}
]
}
DeleteQueue
The following example shows a CloudTrail log entry for a DeleteQueue API call.
{
"Records": [
{
"eventVersion": "1.06",
"userIdentity": {
"type": "IAMUser",
"principalId": "AKIAI44QH8DHBEXAMPLE",
"arn": "arn:aws:iam::123456789012:user/Carlos",
"accountId": "123456789012",
"accessKeyId": "AKIAIOSFODNN7EXAMPLE",
"userName": "Carlos"
},
"eventTime": "2018-06-28T22:23:46Z",
"eventSource": "sqs.amazonaws.com",
"eventName": "DeleteQueue",
"awsRegion": "us-east-2",
"sourceIPAddress": "203.0.113.2",
"userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0",
"requestParameters": {
"queueUrl": "https://sqs.us-east-2.amazon.com/123456789012/MyQueue"
},
"responseElements": null,
"requestID": "123abcde-f4gh-50ij-klmn-60o789012p30",
"eventID": "0987g654-32f1-09e8-d765-c4f3fb2109fa"
}
]
}
RemovePermission
The following example shows a CloudTrail log entry for a RemovePermission API call.
{
"Records": [
{
"eventVersion": "1.06",
"userIdentity": {
"type": "IAMUser",
148
Amazon Simple Queue Service Developer Guide
Monitoring queues using CloudWatch
"principalId": "AKIAI44QH8DHBEXAMPLE",
"arn": "arn:aws:iam::123456789012:user/Jane",
"accountId": "123456789012",
"accessKeyId": "AKIAIOSFODNN7EXAMPLE",
"userName": "Jane"
},
"eventTime": "2018-06-28T22:23:46Z",
"eventSource": "sqs.amazonaws.com",
"eventName": "RemovePermission",
"awsRegion": "us-east-2",
"sourceIPAddress": "203.0.113.3",
"userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0",
"requestParameters": {
"label": "label",
"queueUrl": "https://sqs.us-east-2.amazon.com/123456789012/MyQueue"
},
"responseElements": null,
"requestID": "123abcde-f4gh-50ij-klmn-60o789012p30",
"eventID": "0987g654-32f1-09e8-d765-c4f3fb2109fa"
}
]
}
SetQueueAttributes
The following example shows a CloudTrail log entry for SetQueueAttributes:
{
"Records": [
{
"eventVersion": "1.06",
"userIdentity": {
"type": "IAMUser",
"principalId": "AKIAI44QH8DHBEXAMPLE",
"arn": "arn:aws:iam::123456789012:user/Maria",
"accountId": "123456789012",
"accessKeyId": "AKIAIOSFODNN7EXAMPLE",
"userName": "Maria"
},
"eventTime": "2018-06-28T22:23:46Z",
"eventSource": "sqs.amazonaws.com",
"eventName": "SetQueueAttributes",
"awsRegion": "us-east-2",
"sourceIPAddress": "203.0.113.4",
"userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0",
"requestParameters": {
"attributes": {
"VisibilityTimeout": "100"
},
"queueUrl": "https://sqs.us-east-2.amazon.com/123456789012/MyQueue"
},
"responseElements": null,
"requestID": "123abcde-f4gh-50ij-klmn-60o789012p30",
"eventID": "0987g654-32f1-09e8-d765-c4f3fb2109fa"
}
]
}
149
Amazon Simple Queue Service Developer Guide
Monitoring queues using CloudWatch
SQS console (p. 150), the CloudWatch console (p. 151), using the AWS CLI (p. 152), or using the
CloudWatch API (p. 152). You can also set CloudWatch alarms (p. 152) for Amazon SQS metrics.
CloudWatch metrics for your Amazon SQS queues are automatically collected and pushed to CloudWatch
at one-minute intervals. These metrics are gathered on all queues that meet the CloudWatch guidelines
for being active. CloudWatch considers a queue to be active for up to six hours if it contains any
messages or if any action accesses it.
Note
Topics
• Accessing CloudWatch metrics for Amazon SQS (p. 150)
• Creating CloudWatch alarms for Amazon SQS metrics (p. 152)
• Available CloudWatch metrics for Amazon SQS (p. 153)
150
Amazon Simple Queue Service Developer Guide
Monitoring queues using CloudWatch
151
Amazon Simple Queue Service Developer Guide
Monitoring queues using CloudWatch
For more information and additional options, see Graph Metrics and Using Amazon CloudWatch
Dashboards in the Amazon CloudWatch User Guide.
For more information, see Get Statistics for a Metric in the Amazon CloudWatch User Guide.
CloudWatch API
To access Amazon SQS metrics using the CloudWatch API, use the GetMetricStatistics action.
For more information, see Get Statistics for a Metric in the Amazon CloudWatch User Guide.
1. Sign in to the AWS Management Console and open the CloudWatch console at https://
console.aws.amazon.com/cloudwatch/.
2. Choose Alarms, and then choose Create Alarm.
3. In the Select Metric section of the Create Alarm dialog box, choose Browse Metrics, SQS.
4. For SQS > Queue Metrics, choose the QueueName and Metric Name for which to set an alarm,
and then choose Next. For a list of available metrics, see Available CloudWatch metrics for Amazon
SQS (p. 153).
In the following example, the selection is for an alarm for the NumberOfMessagesSent metric for
the MyQueue queue. The alarm triggers when the number of sent messages exceeds 100.
5. In the Define Alarm section of the Create Alarm dialog box, do the following:
152
Amazon Simple Queue Service Developer Guide
Monitoring queues using CloudWatch
a. Under Alarm Threshold, type the Name and Description for the alarm.
b. Set is to > 100.
c. Set for to 1 out of 1 datapoints.
d. Under Alarm preview, set Period to 1 Hour.
e. Set Statistic to Standard, Sum.
f. Under Actions, set Whenever this alarm to State is ALARM.
If you want CloudWatch to send a notification when the alarm is triggered, select an existing
Amazon SNS topic or choose New list and enter email addresses separated by commas.
Note
If you create a new Amazon SNS topic, the email addresses must be verified before
they receive any notifications. If the alarm state changes before the email addresses are
verified, the notifications aren't delivered.
6. Choose Create Alarm.
Metric Description
153
Amazon Simple Queue Service Developer Guide
Monitoring queues using CloudWatch
Metric Description
poison-pill message is consumed
successfully.
• When the queue has a redrive
policy, the message is moved
to a dead-letter queue (p. 45)
after the configured maximum
number of receives. When
the message is moved to
the dead-letter queue, the
ApproximateAgeOfOldestMessage
metric of the dead-letter queue
represents the time when the
message was moved to the dead-
letter queue (not the original
time the message was sent).
Units: Seconds
Units: Count
Units: Count
154
Amazon Simple Queue Service Developer Guide
Monitoring queues using CloudWatch
Metric Description
Units: Count
Units: Count
155
Amazon Simple Queue Service Developer Guide
Monitoring queues using CloudWatch
Metric Description
Units: Count
Units: Count
156
Amazon Simple Queue Service Developer Guide
Compliance validation
Metric Description
Units: Count
Units: Bytes
¹ These metrics are calculated from a service perspective, and can include retries. Don't rely on the
absolute values of these metrics, or use them to estimate current queue status.
For a list of AWS services in scope of specific compliance programs, see AWS Services in Scope by
Compliance Program. For general information, see AWS Compliance Programs.
You can download third-party audit reports using AWS Artifact. For more information, see Downloading
Reports in AWS Artifact.
Your compliance responsibility when using Amazon SQS is determined by the sensitivity of your data,
your company's compliance objectives, and applicable laws and regulations. AWS provides the following
resources to help with compliance:
157
Amazon Simple Queue Service Developer Guide
Resilience
• Security and Compliance Quick Start Guides – These deployment guides discuss architectural
considerations and provide steps for deploying security- and compliance-focused baseline
environments on AWS.
• Architecting for HIPAA Security and Compliance Whitepaper – This whitepaper describes how
companies can use AWS to create HIPAA-compliant applications.
• AWS Compliance Resources – This collection of workbooks and guides might apply to your industry
and location.
• Evaluating Resources with Rules in the AWS Config Developer Guide – The AWS Config service assesses
how well your resource configurations comply with internal practices, industry guidelines, and
regulations.
• AWS Security Hub – This AWS service provides a comprehensive view of your security state within AWS
that helps you check your compliance with security industry standards and best practices.
In addition to the AWS global infrastructure, Amazon SQS offers distributed queues.
Distributed queues
There are three main parts in a distributed messaging system: the components of your distributed
system, your queue (distributed on Amazon SQS servers), and the messages in the queue.
In the following scenario, your system has several producers (components that send messages to the
queue) and consumers (components that receive messages from the queue). The queue (which holds
messages A through E) redundantly stores the messages across multiple Amazon SQS servers.
158
Amazon Simple Queue Service Developer Guide
Infrastructure security
You use AWS published API actions to access Amazon SQS through the network. Clients must support
Transport Layer Security (TLS) 1.0 or later. We recommend TLS 1.2 or later. Clients must also support
cipher suites with Perfect Forward Secrecy (PFS), such as Ephemeral Diffie-Hellman (DHE) or Elliptic
Curve Ephemeral Diffie-Hellman (ECDHE).
You must sign requests using an access key ID and a secret access key associated with an IAM principal.
Alternatively, you can use the AWS Security Token Service (AWS STS) to generate temporary security
credentials for signing requests.
You can call these API actions from any network location, but Amazon SQS supports resource-based
access policies, which can include restrictions based on the source IP address. You can also use Amazon
SQS policies to control access from specific Amazon VPC endpoints or specific VPCs. This effectively
isolates network access to a given Amazon SQS queue from only the specific VPC within the AWS
network. For more information, see Example 5: Deny access if it isn't from a VPC endpoint (p. 141).
Topics
• Make sure that queues aren't publicly accessible (p. 159)
• Implement least-privilege access (p. 160)
• Use IAM roles for applications and AWS services which require Amazon SQS access (p. 160)
• Implement server-side encryption (p. 160)
• Enforce encryption of data in transit (p. 160)
• Consider using VPC endpoints to access Amazon SQS (p. 161)
159
Amazon Simple Queue Service Developer Guide
Preventative best practices
Follow the standard security advice of granting least privilege. That is, grant only the permissions
required to perform a specific task. You can implement this using a combination of security policies.
Amazon SQS uses the producer-consumer model, requiring three types of user account access:
• Administrators – Access to creating, modifying, and deleting queues. Administrators also control
queue policies.
• Producers – Access to sending messages to queues.
• Consumers – Access to receiving and deleting messages from queues.
Use IAM roles for applications and AWS services which require
Amazon SQS access
For applications or AWS services such as Amazon EC2 to access Amazon SQS queues, they must use valid
AWS credentials in their AWS API requests. Because these credentials aren't rotated automatically, you
shouldn't store AWS credentials directly in the application or EC2 instance.
You should use an IAM role to manage temporary credentials for applications or services that need to
access Amazon SQS. When you use a role, you don't have to distribute long-term credentials (such as a
user name, password, and access keys) to an EC2 instance or AWS service such as AWS Lambda. Instead,
the role supplies temporary permissions that applications can use when they make calls to other AWS
resources.
For more information, see IAM Roles and Common Scenarios for Roles: Users, Applications, and Services
in the IAM User Guide.
For more information, see Encryption at rest (p. 109) and Key management (p. 111).
160
Amazon Simple Queue Service Developer Guide
Preventative best practices
Amazon SQS VPC endpoints provide two ways to control access to your messages:
• You can control the requests, users, or groups that are allowed through a specific VPC endpoint.
• You can control which VPCs or VPC endpoints have access to your queue using a queue policy.
For more information, see Amazon Virtual Private Cloud endpoints for Amazon SQS (p. 116) and
Creating an Amazon VPC endpoint policy for Amazon SQS (p. 117).
161
Amazon Simple Queue Service Developer Guide
Making Query API requests
To access Amazon SQS using a variety of programming languages, you can also use AWS SDKs which
contain the following automatic functionality:
For command-line tool information, see the Amazon SQS sections in the AWS CLI Command Reference
and the AWS Tools for PowerShell Cmdlet Reference.
Topics
• Making Query API requests (p. 162)
• Amazon SQS batch actions (p. 168)
Topics
• Constructing an endpoint (p. 162)
• Making a GET request (p. 163)
• Making a POST request (p. 163)
• Authenticating requests (p. 164)
• Interpreting responses (p. 167)
Constructing an endpoint
In order to work with Amazon SQS queues, you must construct an endpoint. For information about
Amazon SQS endpoints, see the following pages in the Amazon Web Services General Reference:
• Regional endpoints
• Amazon Simple Queue Service endpoints and quotas
Every Amazon SQS endpoint is independent. For example, if two queues are named MyQueue and
one has the endpoint sqs.us-east-2.amazonaws.com while the other has the endpoint sqs.eu-
west-2.amazonaws.com, the two queues don't share any data with each other.
162
Amazon Simple Queue Service Developer Guide
Making a GET request
https://sqs.eu-west-2.amazonaws.com/
?Action=CreateQueue
&DefaultVisibilityTimeout=40
&QueueName=MyQueue
&Version=2012-11-05
&AUTHPARAMS
Note
Queue names and queue URLs are case-sensitive.
The structure of AUTHPARAMS depends on the signature of the API request. For more
information, see Signing AWS API Requests in the Amazon Web Services General Reference.
• Endpoint – The resource that the request is acting on (the queue name and URL (p. 37)), for example:
https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue
• Action – The action that you want to perform on the endpoint. A question mark (?) separates the
endpoint from the action, for example: ?Action=SendMessage&MessageBody=Your%20Message
%20Text
• Parameters – Any request parameters—each parameter is separated by an ampersand (&), for
example: &Version=2012-11-05&AUTHPARAMS
The following is an example of a GET request that sends a message to an Amazon SQS queue.
https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue
?Action=SendMessage&MessageBody=Your%20message%20text
&Version=2012-11-05
&AUTHPARAMS
Note
Queue names and queue URLs are case-sensitive.
Because GET requests are URLs, you must URL-encode all parameter values. Because spaces
aren't allowed in URLs, each space is URL-encoded as %20. (The rest of the example isn't URL-
encoded to make it easier to read.)
The header is followed by a form-urlencoded POST request that sends a message to an Amazon SQS
queue. Each parameter is separated by an ampersand (&).
Action=SendMessage
163
Amazon Simple Queue Service Developer Guide
Authenticating requests
&MessageBody=Your+Message+Text
&Expires=2020-10-15T12%3A00%3A00Z
&Version=2012-11-05
&AUTHPARAMS
Note
Only the Content-Type HTTP header is required. The AUTHPARAMS is the same as for the GET
request.
Your HTTP client might add other items to the HTTP request, according to the client's HTTP
version.
Authenticating requests
Authentication is the process of identifying and verifying the party that sends a request. During the first
stage of authentication, AWS verifies the identity of the producer and whether the producer is registered
to use AWS (for more information, see Step 1: Create an AWS account (p. 4) and Step 2: Create an IAM
user (p. 4)). Next, AWS abides by the following procedure:
Topics
• Basic authentication process with HMAC-SHA (p. 164)
• Part 1: The request from the user (p. 165)
• Part 2: The response from AWS (p. 166)
• The AWS Access Key ID that identifies your AWS account, which AWS uses to look up your Secret
Access Key.
• The HMAC-SHA request signature, calculated using your Secret Access Key (a shared secret known
only to you and AWS—for more information, see RFC2104). The AWS SDK handles the signing process;
however, if you submit a query request over HTTP or HTTPS, you must include a signature in every
query request.
1. Derive a Signature Version 4 Signing Key. For more information, see Deriving the Signing Key with
Java.
Note
Amazon SQS supports Signature Version 4, which provides improved SHA256-based
security and performance over previous versions. When you create new applications that
use Amazon SQS, use Signature Version 4.
2. Base64-encode the request signature. The following sample Java code does this:
package amazon.webservices.common;
164
Amazon Simple Queue Service Developer Guide
Authenticating requests
• The timestamp (or expiration) of the request. The timestamp that you use in the request must be
a dateTime object, with the complete date, including hours, minutes, and seconds. For example:
2007-01-31T23:59:59Z Although this isn't required, we recommend providing the object using the
Coordinated Universal Time (Greenwich Mean Time) time zone.
Note
Make sure that your server time is set correctly. If you specify a timestamp (rather than an
expiration), the request automatically expires 15 minutes after the specified time (AWS
doesn't process requests with timestamps more than 15 minutes earlier than the current time
on AWS servers).
If you use .NET, you must not send overly specific timestamps (because of different
interpretations of how extra time precision should be dropped). In this case, you should
manually construct dateTime objects with precision of no more than one millisecond.
165
Amazon Simple Queue Service Developer Guide
Authenticating requests
166
Amazon Simple Queue Service Developer Guide
Interpreting responses
1. AWS uses the Access Key ID to look up your Secret Access Key.
2. AWS generates a signature from the request data and the Secret Access Key, using the same
algorithm that you used to calculate the signature you sent in the request.
3. One of the following happens:
• If the signature that AWS generates matches the one you send in the request, AWS considers the
request to be authentic.
• If the comparison fails, the request is discarded, and AWS returns an error.
Interpreting responses
In response to an action request, Amazon SQS returns an XML data structure that contains the results
of the request. For more information, see the individual actions in the Amazon Simple Queue Service API
Reference.
Topics
• Successful response structure (p. 167)
• Error response structure (p. 168)
167
Amazon Simple Queue Service Developer Guide
Batch actions
<CreateQueueResponse
xmlns=https://sqs.us-east-2.amazonaws.com/doc/2012-11-05/
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xsi:type=CreateQueueResponse>
<CreateQueueResult>
<QueueUrl>https://sqs.us-east-2.amazonaws.com/770098461991/queue2</QueueUrl>
</CreateQueueResult>
<ResponseMetadata>
<RequestId>cb919c0a-9bce-4afe-9b48-9bdf2412bb67</RequestId>
</ResponseMetadata>
</CreateQueueResponse>
<ErrorResponse>
<Error>
<Type>Sender</Type>
<Code>InvalidParameterValue</Code>
<Message>
Value (quename_nonalpha) for parameter QueueName is invalid.
Must be an alphanumeric String of 1 to 80 in length.
</Message>
</Error>
<RequestId>42d59b56-7407-4c4a-be0f-4c88daeea257</RequestId>
</ErrorResponse>
• SendMessageBatch
• DeleteMessageBatch
• ChangeMessageVisibilityBatch
168
Amazon Simple Queue Service Developer Guide
Enabling client-side buffering and request batching
You can take advantage of batch functionality using the Query API, or an AWS SDK that supports the
Amazon SQS batch actions.
Note
The total size of all messages that you send in a single SendMessageBatch call can't exceed
262,144 bytes (256 KB).
You can't set permissions for SendMessageBatch, DeleteMessageBatch, or
ChangeMessageVisibilityBatch explicitly. Setting permissions for SendMessage,
DeleteMessage, or ChangeMessageVisibility sets permissions for the corresponding
batch versions of the actions.
The Amazon SQS console doesn't support batch actions.
Topics
• Enabling client-side buffering and request batching (p. 169)
• Increasing throughput using horizontal scaling and action batching (p. 173)
Topics
• Using AmazonSQSBufferedAsyncClient (p. 169)
• Configuring AmazonSQSBufferedAsyncClient (p. 170)
Using AmazonSQSBufferedAsyncClient
Before you begin, complete the steps in Setting up Amazon SQS (p. 4).
Important
The AWS SDK for Java 2.x isn't currently compatible with the
AmazonSQSBufferedAsyncClient.
169
Amazon Simple Queue Service Developer Guide
Enabling client-side buffering and request batching
After you create the new AmazonSQSBufferedAsyncClient, you can use it to send multiple requests
to Amazon SQS (just as you can with AmazonSQSAsyncClient), for example:
Configuring AmazonSQSBufferedAsyncClient
AmazonSQSBufferedAsyncClient is preconfigured with settings that work for most use cases. You
can further configure AmazonSQSBufferedAsyncClient, for example:
1. Create an instance of the QueueBufferConfig class with the required configuration parameters.
2. Provide the instance to the AmazonSQSBufferedAsyncClient constructor.
170
Amazon Simple Queue Service Developer Guide
Enabling client-side buffering and request batching
171
Amazon Simple Queue Service Developer Guide
Enabling client-side buffering and request batching
172
Amazon Simple Queue Service Developer Guide
Increasing throughput using
horizontal scaling and action batching
To achieve high throughput, you must scale message producers and consumers horizontally (add more
producers and consumers).
Topics
• Horizontal scaling (p. 173)
• Action batching (p. 174)
• Working Java example for single-operation and batch requests (p. 174)
Horizontal scaling
Because you access Amazon SQS through an HTTP request-response protocol, the request latency (the
interval between initiating a request and receiving a response) limits the throughput that you can
achieve from a single thread using a single connection. For example, if the latency from an Amazon EC2-
based client to Amazon SQS in the same region averages 20 ms, the maximum throughput from a single
thread over a single connection averages 50 TPS.
Horizontal scaling involves increasing the number of message producers (which make SendMessage
requests) and consumers (which make ReceiveMessage and DeleteMessage requests) in order to
increase your overall queue throughput. You can scale horizontally in three ways:
When you add more clients, you achieve essentially linear gains in queue throughput. For example, if you
double the number of clients, you also double the throughput.
Note
As you scale horizontally, make sure that your Amazon SQS client has enough connections or
threads to support the number of concurrent message producers and consumers that send
requests and receive responses. For example, by default, instances of the AWS SDK for Java
173
Amazon Simple Queue Service Developer Guide
Increasing throughput using
horizontal scaling and action batching
For AmazonSQSAsyncClient, you also must make sure that enough threads are available.
Action batching
Batching performs more work during each round trip to the service (for example, when you send
multiple messages with a single SendMessageBatch request). The Amazon SQS batch actions are
SendMessageBatch, DeleteMessageBatch, and ChangeMessageVisibilityBatch. To take
advantage of batching without changing your producers or consumers, you can use the Amazon SQS
Buffered Asynchronous Client (p. 169).
Note
Because ReceiveMessage can process 10 messages at a time, there is no
ReceiveMessageBatch action.
Batching distributes the latency of the batch action over the multiple messages in a batch request, rather
than accept the entire latency for a single message (for example, a SendMessage request). Because
each round trip carries more work, batch requests make more efficient use of threads and connections,
improving throughput.
You can combine batching with horizontal scaling to provide throughput with fewer threads,
connections, and requests than individual message requests. You can use batched Amazon SQS actions
to send, receive, or delete up to 10 messages at a time. Because Amazon SQS charges by the request,
batching can substantially reduce your costs.
Batching can introduce some complexity for your application (for example, you application must
accumulate messages before sending them, or it sometimes must wait longer for a response). However,
batching can be still effective in the following cases:
• Your application generates many messages in a short time, so the delay is never very long.
• A message consumer fetches messages from a queue at its discretion, unlike typical message
producers that need to send messages in response to events they don't control.
Important
A batch request might succeed even though individual messages in the batch failed. After a
batch request, always check for individual message failures and retry the action if necessary.
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-sqs</artifactId>
174
Amazon Simple Queue Service Developer Guide
Increasing throughput using
horizontal scaling and action batching
<version>LATEST</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-ec2</artifactId>
<version>LATEST</version>
</dependency>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>LATEST</version>
</dependency>
</dependencies>
SimpleProducerConsumer.java
The following Java code example implements a simple producer-consumer pattern. The main thread
spawns a number of producer and consumer threads that process 1 KB messages for a specified time.
This example includes producers and consumers that make single-operation requests and those that
make batch requests.
/*
* Copyright 2010-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* https://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*
*/
import com.amazonaws.AmazonClientException;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.services.sqs.AmazonSQS;
import com.amazonaws.services.sqs.AmazonSQSClientBuilder;
import com.amazonaws.services.sqs.model.*;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import java.math.BigInteger;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import java.util.Scanner;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
/**
* Start a specified number of producer and consumer threads, and produce-consume
* for the least of the specified duration and 1 hour. Some messages can be left
* in the queue because producers and consumers might not be in exact balance.
*/
public class SimpleProducerConsumer {
175
Amazon Simple Queue Service Developer Guide
Increasing throughput using
horizontal scaling and action batching
/*
* Create a new instance of the builder with all defaults (credentials
* and region) set automatically. For more information, see Creating
* Service Clients in the AWS SDK for Java Developer Guide.
*/
final ClientConfiguration clientConfiguration = new ClientConfiguration()
.withMaxConnections(producerCount + consumerCount);
176
Amazon Simple Queue Service Developer Guide
Increasing throughput using
horizontal scaling and action batching
}
consumers[i].start();
}
monitor.interrupt();
monitor.join();
}
/**
* The producer thread uses {@code SendMessage}
* to send messages until it is stopped.
*/
private static class Producer extends Thread {
final AmazonSQS sqsClient;
final String queueUrl;
final AtomicInteger producedCount;
final AtomicBoolean stop;
final String theMessage;
/*
* The producedCount object tracks the number of messages produced by
* all producer threads. If there is an error, the program exits the
* run() method.
*/
public void run() {
try {
while (!stop.get()) {
sqsClient.sendMessage(new SendMessageRequest(queueUrl,
theMessage));
producedCount.incrementAndGet();
}
} catch (AmazonClientException e) {
177
Amazon Simple Queue Service Developer Guide
Increasing throughput using
horizontal scaling and action batching
/*
* By default, AmazonSQSClient retries calls 3 times before
* failing. If this unlikely condition occurs, stop.
*/
log.error("Producer: " + e.getMessage());
System.exit(1);
}
}
}
/**
* The producer thread uses {@code SendMessageBatch}
* to send messages until it is stopped.
*/
private static class BatchProducer extends Thread {
final AmazonSQS sqsClient;
final String queueUrl;
final int batchSize;
final AtomicInteger producedCount;
final AtomicBoolean stop;
final String theMessage;
/*
* Because SendMessageBatch can return successfully, but
* individual batch items fail, retry the failed batch items.
*/
if (!batchResult.getFailed().isEmpty()) {
log.warn("Producer: retrying sending "
+ batchResult.getFailed().size() + " messages");
for (int i = 0, n = batchResult.getFailed().size();
i < n; i++) {
sqsClient.sendMessage(new
SendMessageRequest(queueUrl, theMessage));
producedCount.incrementAndGet();
}
}
}
178
Amazon Simple Queue Service Developer Guide
Increasing throughput using
horizontal scaling and action batching
} catch (AmazonClientException e) {
/*
* By default, AmazonSQSClient retries calls 3 times before
* failing. If this unlikely condition occurs, stop.
*/
log.error("BatchProducer: " + e.getMessage());
System.exit(1);
}
}
}
/**
* The consumer thread uses {@code ReceiveMessage} and {@code DeleteMessage}
* to consume messages until it is stopped.
*/
private static class Consumer extends Thread {
final AmazonSQS sqsClient;
final String queueUrl;
final AtomicInteger consumedCount;
final AtomicBoolean stop;
/*
* Each consumer thread receives and deletes messages until the main
* thread stops the consumer thread. The consumedCount object tracks the
* number of messages that are consumed by all consumer threads, and the
* count is logged periodically.
*/
public void run() {
try {
while (!stop.get()) {
try {
final ReceiveMessageResult result = sqsClient
.receiveMessage(new
ReceiveMessageRequest(queueUrl));
if (!result.getMessages().isEmpty()) {
final Message m = result.getMessages().get(0);
sqsClient.deleteMessage(new
DeleteMessageRequest(queueUrl,
m.getReceiptHandle()));
consumedCount.incrementAndGet();
}
} catch (AmazonClientException e) {
log.error(e.getMessage());
}
}
} catch (AmazonClientException e) {
/*
* By default, AmazonSQSClient retries calls 3 times before
* failing. If this unlikely condition occurs, stop.
*/
log.error("Consumer: " + e.getMessage());
System.exit(1);
}
}
}
/**
179
Amazon Simple Queue Service Developer Guide
Increasing throughput using
horizontal scaling and action batching
if (!result.getMessages().isEmpty()) {
final List<Message> messages = result.getMessages();
final DeleteMessageBatchRequest batchRequest =
new DeleteMessageBatchRequest()
.withQueueUrl(queueUrl);
/*
* Because DeleteMessageBatch can return successfully,
* but individual batch items fail, retry the failed
* batch items.
*/
if (!batchResult.getFailed().isEmpty()) {
final int n = batchResult.getFailed().size();
log.warn("Producer: retrying deleting " + n
+ " messages");
for (BatchResultErrorEntry e : batchResult
.getFailed()) {
sqsClient.deleteMessage(
new DeleteMessageRequest(queueUrl,
messages.get(Integer
.parseInt(e.getId()))
.getReceiptHandle()));
consumedCount.incrementAndGet();
}
}
180
Amazon Simple Queue Service Developer Guide
Increasing throughput using
horizontal scaling and action batching
}
}
} catch (AmazonClientException e) {
/*
* By default, AmazonSQSClient retries calls 3 times before
* failing. If this unlikely condition occurs, stop.
*/
log.error("BatchConsumer: " + e.getMessage());
System.exit(1);
}
}
}
/**
* This thread prints every second the number of messages produced and
* consumed so far.
*/
private static class Monitor extends Thread {
private final AtomicInteger producedCount;
private final AtomicInteger consumedCount;
private final AtomicBoolean stop;
181
Amazon Simple Queue Service Developer Guide
Resource Description
Amazon Simple Queue Service API Descriptions of actions, parameters, and data types and a list
Reference of errors that the service returns.
Amazon SQS in the AWS CLI Command Descriptions of the AWS CLI commands that you can use to
Reference work with queues.
Regions and Endpoints Information about Amazon SQS regions and endpoints
Product Page The primary web page for information about Amazon SQS.
AWS Premium Support Information The primary web page for information about AWS Premium
Support, a one-on-one, fast-response support channel to
help you build and run applications on AWS infrastructure
services.
182
Amazon Simple Queue Service Developer Guide
Documentation history
The following table describes the important changes to the Amazon Simple Queue Service Developer
Guide since Jan 2019. For notifications about updates to this documentation, subscribe to the RSS feed.
High throughput for messages in High throughput for Amazon May 27, 2021
FIFO queues is available SQS FIFO queues provides a
higher number of transactions
per second (TPS) for messages in
FIFO queues. For information on
throughput quotas, see Quotas
related to messages.
High throughput for messages High throughput for Amazon December 17, 2020
in FIFO queues is available in SQS FIFO queues is in preview
preview release release and is subject to change.
This feature provides a higher
number of transactions per
second (TPS) for messages in
FIFO queues. For information on
throughput quotas, see Quotas
related to messages.
Amazon SQS supports You can specify the maximum June 22, 2020
pagination for listQueues and number of results to
listDeadLetterSourceQueues return from a listQueues or
listDeadLetterSourceQueues
request.
Amazon SQS supports 1-minute The one-minute CloudWatch November 25, 2019
CloudWatch metrics metric for Amazon SQS is
currently available only in the
following Regions: US East
(Ohio), Europe (Ireland), Europe
(Stockholm), and Asia Pacific
(Tokyo).
AWS Lambda triggers for You can configure messages November 25, 2019
Amazon SQS FIFO queues are arriving in a FIFO queue as a
available Lambda function trigger.
Server-side encryption (SSE) for SSE for Amazon SQS is available November 13, 2019
Amazon SQS is available in the in the China Regions.
China Regions
183
Amazon Simple Queue Service Developer Guide
FIFO queues are available in the FIFO queues are available in the October 10, 2019
Middle East (Bahrain) Region Middle East (Bahrain) Region.
Amazon Virtual Private Cloud You can send messages to September 5, 2019
(Amazon VPC) endpoints for your Amazon SQS queues
Amazon SQS are available in from Amazon VPC in the AWS
the AWS GovCloud (US-East) GovCloud (US-East) and AWS
and AWS GovCloud (US-West) GovCloud (US-West) Regions.
Regions
You can tag Amazon SQS queues You can use a single Amazon August 22, 2019
upon creation SQS API call, AWS SDK function,
or AWS Command Line
Interface (AWS CLI) command
to simultaneously create a
queue and specify its tags.
In addition, Amazon SQS
supports the aws:TagKeys and
aws:RequestTag AWS Identity
and Access Management (IAM)
keys.
The temporary queue client for Temporary queues help you July 25, 2019
Amazon SQS is now available save development time and
deployment costs when using
common message patterns such
as request-response. You can use
the Temporary Queue Client to
create high-throughput, cost-
effective, application-managed
temporary queues.
SSE for Amazon SQS is available Server-side encryption (SSE) for June 20, 2019
in the AWS GovCloud (US-East) Amazon SQS is available in the
Region AWS GovCloud (US-East) Region.
FIFO queues are available in the FIFO queues are available in the May 15, 2019
Asia Pacific (Hong Kong), China Asia Pacific (Hong Kong), China
(Beijing), AWS GovCloud (US- (Beijing), AWS GovCloud (US-
East), and AWS GovCloud (US- East), and AWS GovCloud (US-
West) Regions West) Regions.
184
Amazon Simple Queue Service Developer Guide
Amazon VPC endpoint policies You can create Amazon VPC April 4, 2019
are available for Amazon SQS endpoint policies for Amazon
SQS.
FIFO queues are available in the FIFO queues are available in the March 14, 2019
Europe (Stockholm) and China Europe (Stockholm) and China
(Ningxia) Regions (Ningxia) Regions.
FIFO queues are available in all FIFO queues are available in February 7, 2019
Regions where Amazon SQS is the US East (Ohio), US East (N.
available Virginia), US West (N. California),
US West (Oregon), Asia Pacific
(Mumbai), Asia Pacific (Seoul),
Asia Pacific (Singapore), Asia
Pacific (Sydney), Asia Pacific
(Tokyo), Canada (Central),
Europe (Frankfurt), Europe
(Ireland), Europe (London),
Europe (Paris), and South
America (São Paulo) Regions.
185
Amazon Simple Queue Service Developer Guide
AWS glossary
For the latest AWS terminology, see the AWS glossary in the AWS General Reference.
186