sqs can return empty messages if the message number is low (<1000), maybe this explains that why the last few hundred tasks takes longer time.
this package set the wait time to be 0, and could potentially get empty messages.
https://github.com/seung-lab/python-task-queue/blob/master/taskqueue/aws_queue_api.py#L60
here is the documentation of the long polling:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/working-with-messages.html#setting-up-long-polling
as suggested, simply setting the waittime to 20 seconds should fix this.
sqs can return empty messages if the message number is low (<1000), maybe this explains that why the last few hundred tasks takes longer time.
this package set the wait time to be 0, and could potentially get empty messages.
https://github.com/seung-lab/python-task-queue/blob/master/taskqueue/aws_queue_api.py#L60
here is the documentation of the long polling:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/working-with-messages.html#setting-up-long-polling
as suggested, simply setting the waittime to 20 seconds should fix this.