Using SQS Delay Queue as a Scheduler

Recently we have been migrating our pdf generation at work from Prince to Doc Raptor. Doc Raptor has an async flow which will call a web hook when the pdf generation is done. We found that if there is an error generating the pdf Doc Raptor doesn’t call the web hook.

This led to an interesting issue, we were using the webhook so we didn’t need to poll their API, now we need to poll their API on the off chance there is an error.

When looking for possible options to implement the polling I thought about using a SQS delay queue, delaying the message visibility by the polling interval. If the status isn’t completed or failed another message is queued again delaying the message visibility.

The general flow is below

The GeneratePDF posts a message to the queue containing the status_id that Doc Raptor give us to check the status, our export id and the original queue time with delay seconds set to 30 seconds. The poll lambda picks up the message and checks the sataus. If the status is completed the lambda just returns, if the status is failed it records the failed reason and returns. If the status is something else it posts another messag to the queue with same data and another 30 second delay. If we find that we have been polling for more than 15 minutes we raise an error and exit.

Lastly the code to send the delay message is very simple.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import { SQS } from 'aws-sdk';

import config from '../config';
export interface pdfCheckMessage {
  exportId: string;
  statusId: string;
  startTime: string;
}

const POLL_QUEUE = config.get('pollQueue');
const DELAY_SECONDS = config.get('delaySeconds');

const sqs = new SQS();

export const queueDelayedMessage = async (message: pdfCheckMessage) => {
  await sqs
    .sendMessage({
      QueueUrl: POLL_QUEUE,
      MessageBody: JSON.stringify(message),
      DelaySeconds: DELAY_SECONDS,
    } as SQS.Types.SendMessageRequest)
    .promise();
};

Yes this code needs upgrading to use the AWS SDK V3.