Background jobs in Node.js with BullMQ and Redis: queues, retries and scheduling

An API request should do the minimum needed to answer and return. Sending a confirmation email, generating an invoice PDF, syncing an order to an ERP or resizing images can all take seconds and can all fail for reasons outside your control. Doing them inside the request makes responses slow and turns a third-party outage into your outage.

Background jobs fix that. The request records what needs doing and returns; a separate worker process does the work, retries on failure and reports problems. In Node.js, BullMQ on Redis is the most common way to do this.

The moving partsLink to this section#

API (producer) ── add job ──► Redis (queue) ◄── fetch job ── Worker process
                                                    │
                                          retries, delays, results

• Queue: a named list of jobs stored in Redis, such as emails or order-sync.
• Producer: your API, which adds jobs.
• Worker: a separate Node process that takes jobs and runs a function for each.

Because jobs live in Redis, they survive API restarts, and you can run several workers to process more in parallel.

Setting up a queueLink to this section#

// src/queues.js
import { Queue } from 'bullmq';
import IORedis from 'ioredis';

export const connection = new IORedis(process.env.REDIS_URL, { maxRetriesPerRequest: null });

export const emailQueue = new Queue('emails', {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: 'exponential', delay: 10_000 }, // 10s, 20s, 40s, 80s…
    removeOnComplete: { age: 24 * 3600, count: 1000 },
    removeOnFail: { age: 7 * 24 * 3600 },
  },
});

maxRetriesPerRequest: null is required for BullMQ's blocking Redis commands. The cleanup options stop Redis from filling up with millions of finished jobs.

Adding jobs from the APILink to this section#

// in a route handler, after the order is saved
await emailQueue.add(
  'order-confirmation',
  { orderId: order.id },
  { jobId: `order-confirmation:${order.id}` },
);
res.status(201).json(order);

Two habits:

• Pass IDs, not whole objects. The worker loads fresh data when it runs. A job that waits in the queue for a minute shouldn't act on a stale copy.
• Use a deterministic jobId when the same work must not be queued twice. BullMQ ignores an add whose jobId already exists, which pairs nicely with the idempotency rules from the webhooks guide.

Writing a workerLink to this section#

// src/workers/email.js
import { Worker } from 'bullmq';
import { connection } from '../queues.js';
import { sendOrderConfirmation } from '../services/mail.js';
import { findOrder } from '../services/orders.js';

const worker = new Worker(
  'emails',
  async (job) => {
    if (job.name === 'order-confirmation') {
      const order = await findOrder(job.data.orderId);
      if (!order) return { skipped: 'order not found' };
      if (order.confirmationSentAt) return { skipped: 'already sent' }; // idempotent
      await sendOrderConfirmation(order);
      await markConfirmationSent(order.id);
      return { sent: true };
    }
    throw new Error(`Unknown job ${job.name}`);
  },
  { connection, concurrency: 5 },
);

worker.on('failed', (job, err) => {
  console.error({ jobId: job?.id, name: job?.name, attempts: job?.attemptsMade, err: err.message }, 'job failed');
});

Run workers as their own processes (for example node src/workers/email.js under PM2 or as a separate container) so a CPU-heavy job can't slow down API responses.

Idempotency againLink to this section#

A job can run more than once: a worker might crash after sending the email but before BullMQ records completion, and the job will be retried. Check whether the effect already happened (confirmationSentAt above) or use the provider's idempotency key when calling payment or email APIs.

Retries that help rather than hurtLink to this section#

• Exponential backoff gives a struggling API time to recover instead of hammering it.
• Don't retry permanent errors. A 400 "invalid email address" will fail every time. Throw BullMQ's UnrecoverableError to fail the job immediately:

import { UnrecoverableError } from 'bullmq';

if (err.status === 400) throw new UnrecoverableError(`Rejected by provider: ${err.message}`);
throw err; // temporary problems: retry with backoff

• Look at failed jobs. After the final attempt, jobs stay in the failed set with their error. Review them, fix the cause and retry them from a dashboard or a script.

Concurrency and rate limitsLink to this section#

concurrency controls how many jobs one worker runs at the same time. I/O-bound jobs such as API calls can use higher values; CPU-bound jobs such as image processing should stay low, or move to worker threads.

When a third-party API allows, say, 10 requests per second, enforce it in the queue rather than hoping:

new Worker('order-sync', processor, { connection, limiter: { max: 10, duration: 1000 } });

The limiter applies across every worker on that queue, so scaling out doesn't break the provider's limit.

Delayed and recurring jobsLink to this section#

Delayed: send a reminder 24 hours after sign-up.

await emailQueue.add('trial-reminder', { userId }, { delay: 24 * 3600 * 1000, jobId: `trial-reminder:${userId}` });

Recurring: instead of a separate cron daemon, use a job scheduler, which BullMQ stores in Redis so only one copy runs even with several workers:

import { Queue } from 'bullmq';
const reports = new Queue('reports', { connection });

await reports.upsertJobScheduler(
  'daily-sales-report',
  { pattern: '30 2 * * *', tz: 'Asia/Kolkata' },
  { name: 'daily-sales', data: {} },
);

The pattern uses standard cron syntax. Check what an expression means and when it will next run with the cron expression builder before deploying it, and always set tz explicitly so daylight-saving changes and server time zones don't surprise you.

Graceful shutdownLink to this section#

When a worker restarts during a deploy, let it finish the jobs it has started:

async function shutdown() {
  await worker.close(); // stops taking new jobs, waits for active ones
  await connection.quit();
  process.exit(0);
}
process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);

If a worker dies mid-job anyway, BullMQ detects the stalled job after its lock expires and hands it to another worker. That's another reason jobs must be idempotent.

MonitoringLink to this section#

At a minimum, track:

• Waiting jobs per queue: a growing backlog means workers can't keep up.
• Failed jobs per hour, with alerts on spikes.
• Job duration: a sudden increase often means a downstream API is slow.

Dashboards such as Bull Board show queues, jobs, payloads and errors, and let you retry failed jobs. Protect them behind authentication; they expose job data. Job timestamps are stored in milliseconds; the timestamp converter helps when reading raw data from Redis.

Redis in productionLink to this section#

• Use a persistent Redis (AOF enabled) or a managed Redis service. A cache-only Redis with eviction can silently drop jobs.
• Set maxmemory-policy noeviction for the Redis that holds queues.
• Keep queue Redis separate from a cache Redis if possible, so cache churn can't affect jobs.

Testing jobsLink to this section#

Background jobs deserve tests as much as API routes, and they're easier to test than they look if the processor is a plain function:

// src/jobs/order-confirmation.js
export async function orderConfirmation({ orderId }, deps) {
  const order = await deps.findOrder(orderId);
  if (!order) return { skipped: 'order not found' };
  if (order.confirmationSentAt) return { skipped: 'already sent' };
  await deps.sendOrderConfirmation(order);
  await deps.markConfirmationSent(order.id);
  return { sent: true };
}

The worker calls orderConfirmation(job.data, realDeps); tests call it with fakes. Test at least three cases: the normal path, the "already done" path that proves idempotency, and a failing dependency that should throw so BullMQ retries. None of these tests need Redis.

For a few end-to-end checks, run Redis in a container in CI, add a job with a real queue, and wait for the worker's completed event. Keep these few and focused; they're slower and can be flaky if they depend on timing.

Deploying workersLink to this section#

Treat workers as a separate service with their own deployment settings:

• Scale independently. API traffic and job volume grow at different rates. Running workers in their own processes or containers lets you add capacity where it's needed.
• Deploy compatible changes. During a rolling deploy, old and new workers run side by side and both process jobs added by old and new APIs. Add new job fields as optional, and remove old ones only after every producer has stopped sending them.
• Health checks. A worker can be running but stuck. Expose a simple health endpoint, or check that it has completed a job recently, and restart it if not.

Common problemsLink to this section#

• Jobs never run. No worker is running for that queue name, or the worker connects to a different Redis or database number.
• maxRetriesPerRequest error at startup. Pass maxRetriesPerRequest: null to the IORedis connection used by workers.
• Duplicate side effects. Jobs retried after a crash. Add idempotency checks.
• Memory grows in Redis. Completed and failed jobs are kept forever. Set removeOnComplete and removeOnFail.
• Scheduled job runs at the wrong hour. Missing tz option; the server's time zone was used.

Frequently asked questionsLink to this section#

Do I need Redis for background jobs?Link to this section#

For BullMQ, yes. Alternatives exist: database-backed queues are simpler to operate for low volumes, and managed services such as Amazon SQS remove the need to run Redis. BullMQ is a good default when you already run Redis or need delays, rate limits and schedulers.

How is this different from Laravel queues?Link to this section#

The ideas are the same: producers, workers, retries and a scheduler. If you work in both stacks, the Laravel guide shows the PHP side, and the patterns transfer directly.

Can I run workers in serverless functions?Link to this section#

Not with long-lived BullMQ workers. Use a small always-on process or container for workers, or a queue service designed for serverless.

How many workers should I run?Link to this section#

Start with one process per queue type with sensible concurrency, watch the waiting count and duration, and add processes when the backlog grows.

Need background processing for a store, an integration or an API? Let's talk.