A New Set of APIs for Amazon SQS Lifeless-Letter Queue Redrive


Voiced by Polly

Immediately, we launch a brand new set of APIs for Amazon SQS. These new APIs mean you can handle dead-letter queue (DLQ) redrive programmatically. Now you can use the AWS SDKs or the AWS Command Line Interface (AWS CLI) to programmatically transfer messages from the DLQ to their authentic queue, or to a customized queue vacation spot, to try to course of them once more. A DLQ is a queue the place Amazon SQS mechanically strikes messages that aren’t accurately processed by your shopper utility.

To completely admire how this new API may provide help to, let’s have a fast look again at historical past.

Message queues are an integral a part of trendy utility architectures. They permit builders to decouple providers by permitting asynchronous and message-based communications between message producers and customers. In most programs, messages are continued in shared storage (the queue) till the patron processes them. Message queues enable builders to construct functions which are resilient to short-term service failure. They assist prioritize message processing and scale their fleet of employee nodes that course of the messages. Message queues are additionally common in event-driven architectures.

Asynchronous message change is just not new in utility architectures. The idea of exchanging messages asynchronously between functions appeared within the Sixties and was first made common when IBM launched TCAM for OS/360 in 1972. The final adoption got here 20 years later with IBM MQ Sequence in 1993 (now IBM MQ) and when Solar Microsystems launched Java Messaging Service (JMS) in 1998, an ordinary API for Java functions to work together with message queues.

AWS launched Amazon SQS on July 12, 2006. Amazon SQS is a extremely scalable, dependable, and elastic queuing service that “simply works.” As Werner wrote on the time: “We now have chosen a concurrency mannequin the place the method engaged on a message mechanically acquires a leased lock on that message; if the message is just not deleted earlier than the lease expires, it turns into out there for processing once more. Makes failure dealing with quite simple.

On January 29, 2014, we launched dead-letter queues (DLQ). DLQs provide help to keep away from a message that didn’t be processed from staying endlessly on prime of the queue, presumably stopping different messages within the queue from processing. With DLQs, every queue has an related property telling Amazon SQS what number of instances a message could also be offered for processing (maxReceiveCount). Every message additionally has an related obtain counter (ReceiveCount). Every time a shopper utility picks up a message for processing, the message obtain rely is incremented by 1. When ReceiveCount > maxReceiveCount, Amazon SQS strikes the message to your designated DLQ for human evaluation and debugging. You typically affiliate alarms with the DLQ to ship notifications when such occasions occur. Typical causes to maneuver a message to the DLQ are as a result of they’re incorrectly formatted, there are bugs within the shopper utility, or it takes too lengthy to course of the message.

At AWS re:Invent 2021, AWS introduced dead-letter queue redrive on the Amazon SQS console. The redrive addresses the second a part of the failed message lifecycle. It permits you to reinject the message in its authentic queue to try processing it once more. After the patron utility is mounted and able to devour the failed messages, you may redrive the messages from the DLQ again within the supply queue or a personalized queue vacation spot. It simply requires a few clicks on the console.

Immediately, we’re including APIs permitting you to write down functions and scripts that deal with the redrive programmatically. There isn’t a longer a have to have a human clicking on the console. Utilizing the API will increase the scalability of your processes and reduces the danger of human error.

Let’s See It in Motion
To check out this new API, I open a terminal for a command-line solely demo. Earlier than I get began, I be sure I’ve the newest model of the AWS CLI. On macOS I enter brew improve awscli.

I first create two queues. One is the dead-letter queue, and the opposite is my utility queue:

# First, I create the dead-letter queue (discover the -dlq I select so as to add on the finish of the queue title)
➜ ~ aws sqs create-queue 
            --queue-name awsnewsblog-dlq                                            
{
    "QueueUrl": "https://sqs.us-east-2.amazonaws.com/012345678900/awsnewsblog-dlq"
}

# second, I retrieve the Arn of the queue I simply created
➜  ~ aws sqs get-queue-attributes 
             --queue-url https://sqs.us-east-2.amazonaws.com/012345678900/awsnewsblog-dlq 
             --attribute-names QueueArn
{
    "Attributes": {
        "QueueArn": "arn:aws:sqs:us-east-2:012345678900:awsnewsblog-dlq"
    }
}

# Third, I create the appliance queue. I enter a redrive coverage: submit messages within the DLQ after three supply makes an attempt
➜  ~ aws sqs create-queue 
             --queue-name awsnewsblog 
             --attributes '{"RedrivePolicy": "{"deadLetterTargetArn":"arn:aws:sqs:us-east-2:012345678900:awsnewsblog-dlq","maxReceiveCount":"3"}"}' 
{
    "QueueUrl": "https://sqs.us-east-2.amazonaws.com/012345678900/awsnewsblog"
}

Now that the 2 queues are prepared, I submit a message to the appliance queue:

➜ ~ aws sqs send-message 
            --queue-url https://sqs.us-east-2.amazonaws.com/012345678900/awsnewsblog 
            --message-body "Hey World"
{
"MD5OfMessageBody": "b10a8db164e0754105b7a99be72e3fe5",
"MessageId": "fdc26778-ce9a-4782-9e33-ae73877cfcb2"
}

Subsequent, I devour the message, however I don’t delete it from the queue. This simulates a crash within the message shopper utility. Message customers are purported to delete the message after profitable processing. I set the maxReceivedCount property to three after I entered the redrivePolicy. I due to this fact repeat this operation thrice to pressure Amazon SQS to maneuver the message to the dead-letter queue after three supply makes an attempt. The default visibility timeout is 30 seconds, so I’ve to attend 30 seconds or extra between the retries.

➜ ~ aws sqs receive-message 
            --queue-url https://sqs.us-east-2.amazonaws.com/012345678900/awsnewsblog
{
"Messages": [
{
"MessageId": "fdc26778-ce9a-4782-9e33-ae73877cfcb2",
"ReceiptHandle": "AQEBP8yOfgBlnjlkGXjyeLROiY7xg7cZ6Znq8Aoa0d3Ar4uvTLPrHZptNotNfKRK25xm+IU8ebD3kDwZ9lja6JYs/t1kBlwiNO6TBACN5srAb/WggQiAAkYl045Tx3CvsOypbJA3y8U+MyEOQRwIz6G85i7MnR8RgKTlhOzOZOVACXC4W8J9GADaQquFaS1wVeM9VDsOxds1hDZLL0j33PIAkIrG016LOQ4sAntH0DOlEKIWZjvZIQGdlRJS65PJu+I/Ka1UPHGiFt9f8m3SR+Y34/ttRWpQANlXQi5ByA47N8UfcpFXXB5L30cUmoDtKucPewsJNG2zRCteR0bQczMMAmOPujsKq70UGOT8X2gEv2LfhlY7+5n8z3yew8sdBjWhVSegrgj6Yzwoc4kXiMddMg==",
"MD5OfBody": "b10a8db164e0754105b7a99be72e3fe5",
"Body": "Hello World"
}
]
}

# wait 30 seconds,
# then repeat two instances (for a complete of three receive-message API calls)

After three processing makes an attempt, the message is just not within the queue anymore:

➜  ~ aws sqs receive-message 
             --queue-url  https://sqs.us-east-2.amazonaws.com/012345678900/awsnewsblog
{
    "Messages": []
}

The message has been moved to the dead-letter queue. I verify the DLQ to verify (discover the queue URL ending with -dlq):

➜  ~ aws sqs receive-message 
             --queue-url  https://sqs.us-east-2.amazonaws.com/012345678900/awsnewsblog-dlq
{
    "Messages": [
        {
            "MessageId": "fdc26778-ce9a-4782-9e33-ae73877cfcb2",
            "ReceiptHandle": "AQEBCLtBMoZYVMMq7fUGNHeCliqE3mFXnkuJ+nOXLK1++uoXWBG31nDejCpxElmiBZWfbcfGJrEdKj4P9HJdrQMYDbeSqB+u1ZlB7CYzQBiQps4SEG0biEoubwqjQbmDZlPrmkFsnYgLD98D1XYWk/Ik6Z2n/wxDo9ko9rbZ15izK5RFnbwveNy8dfc6ireqVB1EGbeGkHcweHGuoeKWXEab1ynZWhNqZsQgCR6pWRkgtn59lJcLv4cJ4UMewNzvt7tMHH69GvVjXdYDYvJJI2vj+6RHvcvSHWWhTNT+CuPEXguVNuNrSya8gho1fCnKpVwQre6HhMlLPjY4wvn/tXY7+5rmte9eXagCqLQXaENB2R7qWNVPiWRIJy8/cTf37NLYVzBom030DNJlH9EeceRhCQ==",
            "MD5OfBody": "b10a8db164e0754105b7a99be72e3fe5",
            "Body": "Hello World"
        }
    ]
}

Now that the setup is prepared, let’s programmatically redrive the message to its authentic queue. Let’s assume I perceive why the patron didn’t accurately course of the message and that I mounted the patron utility code. I take advantage of start-message-move-task on the DLQ to begin the asynchronous redrive. There may be an non-compulsory attribute (MaxNumberOfMessagesPerSecond) to regulate the speed of the redrive:

➜ ~ aws sqs start-message-move-task 
            --source-arn arn:aws:sqs:us-east-2:012345678900:awsnewsblog-dlq
{
    "TaskHandle": "eyJ0YXNrSWQiOiI4ZGJmNjBiMy00MmUwLTQzYTYtYjg4Zi1iMTZjYWRjY2FkNmEiLCJzb3VyY2VBcm4iOiJhcm46YXdzOnNxczp1cy1lYXN0LTI6NDg2NjUyMDY2NjkzOmF3c25ld3NibG9nLWRscSJ9"
}

I can checklist and verify standing the of the transfer duties I initiated with list-message-move-tasks or cancel a working job by calling the cancel-message-move-task API:

➜ ~ aws sqs list-message-move-tasks 
            --source-arn arn:aws:sqs:us-east-2:012345678900:awsnewsblog-dlq
{
    "Outcomes": [
        {
            "Status": "COMPLETED",
            "SourceArn": "arn:aws:sqs:us-east-2:012345678900:awsnewsblog-dlq",
            "ApproximateNumberOfMessagesMoved": 1,
            "ApproximateNumberOfMessagesToMove": 1,
            "StartedTimestamp": 1684135792239
        }
    ]
}

Now my utility can devour the message once more from the appliance queue:

➜  ~ aws sqs receive-message 
             --queue-url  https://sqs.us-east-2.amazonaws.com/012345678900/awsnewsblog                                   
{
    "Messages": [
        {
            "MessageId": "a7ae83ca-cde4-48bf-b822-3d4bc1f4dcae",
            "ReceiptHandle": "AQEB9a+Dm2nvb3VUn9+46j9UsDidU/W6qFwJtXtNWTyfoSDOKT7h73e6ctT9RVZysEw3qqzJOx1cxblTTOSrYwwwoBA2qoJMGsqsrsRGGYojBvf9X8hqi8B8MHn9rTm8diJ2wT2b7WC+TDrx3zIvUeiSEkP+EhqyYOvOs7Q9aETR+Uz02kQxZ/cUJWsN4MMSXBejwW+c5ivv5uQtpfUrfZuCWa9B9O67Kj/q52clriPHpcqCCfJwFBSZkGTXYwTpnjxD4QM7DPS+xVeVfTyM7DsKCAOtpvFBmX5m4UNKT6TROgCnGxTRglUSMWQp8ufVxXiaUyM1dwqxYekM9uX/RCb01gEyCZHas4jeNRV5nUJlhBkkqPlw3i6w9Uuc2y9nH0Df8nH3g7KTXo4lv5Bl3ayh9w==",
            "MD5OfBody": "b10a8db164e0754105b7a99be72e3fe5",
            "Body": "Hello World"
        }
    ]
}

Availability
DLQ redrive APIs can be found immediately in all business Areas the place Amazon SQS is on the market.

Redriving the messages from the dead-letter queue to the supply queue or a customized vacation spot queue generates further API calls billed based mostly on current pricing (beginning at $0.40 per million API calls, after the primary million, which is free each month). Amazon SQS batches the messages whereas redriving them from one queue to a different. This makes shifting messages from one queue to a different a easy and low-cost choice.

To study extra about DLQ and DLQ redrive, verify our documentation.

Do not forget that we stay in an asynchronous world—so ought to your functions. Get began immediately and write your first redrive utility.

— seb



Leave a Reply

Your email address will not be published. Required fields are marked *