Retries & Dead-Letter Queue

Job failures are inevitable. Spooled provides automatic retries with exponential backoff and a dead-letter queue for jobs that can't be processed.

Related guides:

Jobs & Queues Building workers

How Retries Work

When a job fails, Spooled automatically schedules a retry with exponential backoff. This prevents overwhelming downstream services and gives transient failures time to resolve.

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#fef3c7', 'primaryTextColor': '#92400e', 'primaryBorderColor': '#f59e0b', 'lineColor': '#6b7280'}}}%%
stateDiagram-v2
    [*] --> pending
    pending --> claimed: Worker claims
    claimed --> completed: Success
    claimed --> failed: Error
    failed --> scheduled: Retry (attempts left)
    scheduled --> pending: Run time reached
    failed --> dlq: Max retries exceeded
    dlq --> pending: Manual replay
    completed --> [*]

Retry Flow

Worker claims a job and attempts to process it
Processing fails (exception, timeout, or explicit failure)
Spooled checks if retry attempts remain
If retries remain: the job waits as scheduled until its run time, then becomes pending
If max retries exceeded: job moves to dead-letter queue (DLQ)

Backoff

Retries use exponential backoff with a small random jitter, which spreads retries out and prevents thundering-herd problems. The delay doubles with every retry and is capped at 60 seconds: min(2^retries_so_far, 60) seconds plus 0–500 ms of jitter.

Retry	Delay before it runs	Time since first failure
1	1s	~1s
2	2s	~3s
3	4s	~7s
4	8s	~15s
5	16s	~31s
6	32s	~1 min
7	60s (cap)	~2 min
8	60s	~3 min
9	60s	~4 min
10	60s	~5 min

The delay is when the job becomes claimable again; a worker picks it up on its next poll, so observed gaps are slightly longer. With the default max_retries: 3, a job that keeps failing reaches the DLQ about 7 seconds after its first failure.

The backoff curve is fixed server-side — there is no per-job strategy, base or cap to set. If a job needs a longer wait (for example, a downstream outage measured in hours), have the worker complete it and enqueue a follow-up job with scheduled_at set to when it should run.

Retry Configuration

Configure retry behavior when creating jobs:

Job with retry configuration

curl -X POST https://api.spooled.cloud/api/v1/jobs \
  -H "Authorization: Bearer sp_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "queue_name": "my-queue",
    "payload": {
      "event": "user.created",
      "user_id": "usr_123",
      "email": "alice@example.com"
    },
    "idempotency_key": "user-created-usr_123"
  }'

import { SpooledClient } from '@spooled/sdk';

const client = new SpooledClient({
  apiKey: process.env.SPOOLED_API_KEY!,
});

const userId = 'usr_123';

// Create a job
const { id } = await client.jobs.create({
  queueName: 'email-notifications',
  payload: {
    to: 'user@example.com',
    subject: 'Welcome!',
    template: 'welcome',
  },
  idempotencyKey: `welcome-${userId}`,
  maxRetries: 5,
});

console.log(`Created job: ${id}`);

from spooled import SpooledClient
import os

client = SpooledClient(api_key=os.environ["SPOOLED_API_KEY"])

image_id = "img_123"

# Create a background job
result = client.jobs.create({
    "queue_name": "image-processing",
    "payload": {
        "image_url": "https://example.com/image.jpg",
        "operations": ["resize", "compress"],
        "output_format": "webp"
    },
    "idempotency_key": f"process-image-{image_id}",
    "max_retries": 3
})

print(f"Created job: {result.id}")

client.close()

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/spooled-cloud/spooled-sdk-go/spooled"
	"github.com/spooled-cloud/spooled-sdk-go/spooled/resources"
)

func ptr[T any](v T) *T { return &v }

func main() {
	client, err := spooled.NewClient(spooled.WithAPIKey(os.Getenv("SPOOLED_API_KEY")))
	if err != nil {
		panic(err)
	}

	resp, err := client.Jobs().Create(context.Background(), &resources.CreateJobRequest{
		QueueName:      "my-queue",
		Payload:        map[string]any{"key": "value"},
		IdempotencyKey: ptr("unique-key"),
		MaxRetries:     ptr(3),
	})
	if err != nil {
		panic(err)
	}

	fmt.Printf("Created job: %s\n", resp.ID)
}

<?php

use Spooled\SpooledClient;
use Spooled\Config\ClientOptions;

$client = new SpooledClient(new ClientOptions(
    apiKey: getenv('SPOOLED_API_KEY'),
));

$userId = 'usr_123';

// Create a job
$job = $client->jobs->create([
    'queue' => 'email-notifications',
    'payload' => [
        'to' => 'user@example.com',
        'subject' => 'Welcome!',
        'template' => 'welcome',
    ],
    'idempotencyKey' => "welcome-{$userId}",
    'maxRetries' => 5,
]);

echo "Created job: {$job->id}\n";

Configuration Options

Option	Type	Default	Description
`max_retries`	number	settings default (typically 3)	Maximum retry attempts before moving to DLQ. Omitted fields use server queue defaults. REST allows explicit `0`; gRPC maps `<=0` to the settings default.
`timeout_seconds`	number	settings default (typically 300)	Stored with the job but not enforced by the server or the SDK workers; enforce a time limit in your handler. Re-delivery after a crash is governed by the claim lease, not this field.
`priority`	number	0	Job priority (-100 to 100)

Dead-Letter Queue (DLQ)

Jobs that exhaust all retry attempts land in the dead-letter queue. The DLQ preserves the full job payload and failure history for debugging and replay.

Dashboard Tip

Dashboard → Jobs → Dead Letter Queue

What to look for:

→ Job payload and metadata
→ Last error message
→ Retry count and failure history
→ Original queue name

Actions:

Retry individual jobs
Bulk retry all DLQ jobs
Purge old failed jobs

List DLQ Jobs

# List jobs in dead-letter queue
curl -X GET "https://api.spooled.cloud/api/v1/jobs/dlq?queue_name=my-queue&limit=100" \
  -H "Authorization: Bearer sp_live_YOUR_API_KEY"

// List jobs in dead-letter queue
const dlqJobs = await client.jobs.dlq.list({
  queueName: 'payment-processing',
  limit: 100,
});

for (const job of dlqJobs) {
  console.log(`DLQ Job: ${job.id}, status: ${job.status}`);
  console.log(`Attempt: ${job.attempt}, created: ${job.createdAt}`);
}

# List jobs in dead-letter queue
dlq_jobs = client.jobs.dlq.list({
    "queue_name": "payment-processing",
    "limit": 100
})

for job in dlq_jobs:
    print(f"DLQ Job: {job.id}, status: {job.status}")
    print(f"Retry count: {job.retry_count}, created: {job.created_at}")

import (
	"github.com/spooled-cloud/spooled-sdk-go/spooled/types"
	"context"
	"github.com/spooled-cloud/spooled-sdk-go/spooled"
	"github.com/spooled-cloud/spooled-sdk-go/spooled/resources"
)

client, _ := spooled.NewClient(spooled.WithAPIKey("sp_live_YOUR_API_KEY"))

dlqJobs, err := client.Jobs().DLQ().List(context.Background(), &resources.ListDLQParams{
	QueueName: types.String("payment-processing"),
	Limit:     types.Int(100),
})
if err != nil {
	panic(err)
}

for _, job := range dlqJobs {
	fmt.Printf("DLQ Job: %s, status: %s\n", job.ID, job.Status)
}

<?php

// List jobs in dead-letter queue
$dlqJobs = $client->jobs->dlq->list([
    'queue' => 'payment-processing',
    'limit' => 100,
]);

foreach ($dlqJobs as $job) {
    echo "DLQ Job: {$job->id}, status: {$job->status}\n";
    echo "Retry count: {$job->retryCount}, created: {$job->createdAt}\n";
}

Retry DLQ Jobs

# Retry jobs from DLQ
curl -X POST https://api.spooled.cloud/api/v1/jobs/dlq/retry \
  -H "Authorization: Bearer sp_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "queue_name": "my-queue",
    "limit": 50
  }'

import (
	"github.com/spooled-cloud/spooled-sdk-go/spooled/types"
	"context"
	"github.com/spooled-cloud/spooled-sdk-go/spooled"
	"github.com/spooled-cloud/spooled-sdk-go/spooled/resources"
)

client, _ := spooled.NewClient(spooled.WithAPIKey("sp_live_YOUR_API_KEY"))

result, err := client.Jobs().DLQ().Retry(context.Background(), &resources.RetryDLQRequest{
	QueueName: types.String("payment-processing"),
})
if err != nil {
	panic(err)
}

fmt.Printf("Retried %d jobs\n", result.RetriedCount)

A retried job — whether through POST /api/v1/jobs/dlq/retry or POST /api/v1/jobs/{id}/retry — goes back to pending with retry_count reset to 0, so it gets its full max_retries budget again.

A retry sends a job.retried webhook event when you subscribe to it (data.source says whether it came from job_retry, dlq_retry or workflow_retry). To see what happened to a job before it landed here, read its history: retry_scheduled and deadlettered events carry the error and the attempt.

Purge DLQ

# Purge DLQ jobs (requires confirm: true)
curl -X POST https://api.spooled.cloud/api/v1/jobs/dlq/purge \
  -H "Authorization: Bearer sp_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "queue_name": "my-queue",
    "confirm": true
  }'

// Purge old DLQ jobs
const result = await client.jobs.dlq.purge({
  queueName: 'my-queue',
  olderThan: new Date(Date.now() - 7 * 24 * 60 * 60 * 1000), // 7 days
  confirm: true,
});

console.log(`Purged ${result.purgedCount} jobs`);

from datetime import datetime, timedelta, timezone

# Purge old DLQ jobs
result = client.jobs.dlq.purge({
    "queue_name": "my-queue",
    "older_than": datetime.now(timezone.utc) - timedelta(days=7),
    "confirm": True
})

print(f"Purged {result.purged_count} jobs")

import (
	"context"
	"github.com/spooled-cloud/spooled-sdk-go/spooled"
	"github.com/spooled-cloud/spooled-sdk-go/spooled/resources"
	"github.com/spooled-cloud/spooled-sdk-go/spooled/types"
)

client, _ := spooled.NewClient(spooled.WithAPIKey("sp_live_YOUR_API_KEY"))

result, err := client.Jobs().DLQ().Purge(context.Background(), &resources.PurgeDLQRequest{
	QueueName: types.String("my-queue"),
	Confirm:   true, // required by the API; Purge() also sets it (SDK v1.3.1+)
})
if err != nil {
	panic(err)
}

fmt.Printf("Purged %d jobs\n", result.PurgedCount)

<?php

// Purge old DLQ jobs
$result = $client->jobs->dlq->purge([
    'queueName' => 'my-queue',
    'olderThan' => (new DateTime('-7 days'))->format(DateTime::ATOM),
    'confirm' => true,
]);

echo "Purged {$result['purgedCount']} jobs\n";

DLQ Best Practices

• Get told about every dead-lettered job with an outgoing webhook on job.deadlettered (optionally in Slack, Discord or Teams format). It fires whichever way the job got there: a failure your worker reported, a lease that expired on the last retry, or a worker that went offline. job.failed with data.status deadletter fires for the same jobs, so subscribe to one of the two. There is no built-in DLQ alert beyond that, so also poll GET /api/v1/jobs/dlq or watch the dashboard
• Review DLQ jobs regularly (daily for critical queues)
• Fix the root cause before replaying jobs
• Consider archiving old DLQ jobs to cold storage

Handling Failures in Workers

Workers should explicitly mark jobs as failed with a reason. This helps with debugging and determines retry behavior.

Failing a job

# Fail a job (will retry if retries remaining; echo lease_id from claim)
curl -X POST https://api.spooled.cloud/api/v1/jobs/job_xyz123/fail \
  -H "Authorization: Bearer sp_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "worker_id": "worker-1",
    "lease_id": "lease_abc",
    "error": "Connection timeout"
  }'

// Fail the job (will retry with exponential backoff)
await client.jobs.fail(job.id, {
  workerId: 'worker-1',
  leaseId: job.leaseId ?? undefined,
  error: 'Connection timeout to downstream service',
});

import (
	"github.com/spooled-cloud/spooled-sdk-go/spooled/types"
	"context"
	"github.com/spooled-cloud/spooled-sdk-go/spooled"
	"github.com/spooled-cloud/spooled-sdk-go/spooled/resources"
)

client, _ := spooled.NewClient(spooled.WithAPIKey("sp_live_YOUR_API_KEY"))

err := client.Jobs().Fail(context.Background(), job.ID, &resources.FailJobRequest{
	WorkerID: "worker-1",
	LeaseID:  job.LeaseID,
	Error:    "Connection timeout to downstream service",
})
if err != nil {
	panic(err)
}

fmt.Println("Job failed and will retry")

<?php

// Fail the job (will retry with exponential backoff)
$client->jobs->fail($job->id, [
    'workerId' => 'worker-1',
    'leaseId' => $job->leaseId,
    'error' => 'Connection timeout to downstream service',
]);

Debug Failed Jobs

Dashboard → Jobs → Failed

What to look for:

→ Error message in last_error field
→ Retry count vs max_retries
→ Job payload for invalid data
→ Timestamps to correlate with logs

Actions:

Check worker logs for stack traces
Verify external service availability
Test payload manually

Next Steps

Jobs & queues — Understand the job lifecycle
Building workers — Production worker patterns
Real-time updates — Monitor job status in real-time