# BullMQ — Full Content Bundle

This file concatenates the full text of every article published on
https://bullmq.io. It is generated automatically from the same source as
the rendered site, following the spirit of the
[llms.txt](https://llmstxt.org) convention's "expanded" form. The intended
audience is language models that need to ingest the project's body of
written work in a single request.

See https://bullmq.io/llms.txt for the index version.

Site: BullMQ — background jobs and message queue for Redis.
Project page: https://bullmq.io
Source code: https://github.com/taskforcesh/bullmq
Documentation: https://docs.bullmq.io

---

# Pluggable Redis Clients in BullMQ

URL: https://bullmq.io/articles/guides/pluggable-redis-clients/
Author: Manuel Astudillo
Published: 2026-05-25
Description: BullMQ now supports three Redis client backends, ioredis, node-redis, and Bun's built-in Redis client, behind a single adapter interface. Pick the one that fits your runtime and dependency footprint.
Tags: articles, nodejs, bun, redis, announcement, guide

Pluggable Redis Clients in BullMQ
Historically, BullMQ has depended on the popular Redis client ioredis.
This client has been working very reliably for us, and we can only be grateful to its authors for
their exceptional work over the years.
From time to time, however, some users wondered why we didn't also support node-redis,
which is probably the second most popular Redis client for Node.js. On top of that, with the increased popularity of Bun,
yet another Redis client appeared (Bun's native Redis client), so we felt it was time to do something about it and
provide support for these other clients.
Starting with version v5.77.0, BullMQ talks to Redis through a
small adapter interface called IRedisClient, and three drivers are supported out of the box:

ioredis, the long standing default, used automatically when you pass connection options the same way you always have.
node-redis (a.k.a. @redis/client v5), wrapped through createNodeRedisClient.
Bun's built-in Redis client, wrapped through createBunRedisClient.

The adapter interface gives BullMQ a single place to express the operations it actually needs
(Lua script registration, transactions, pipelines, hash and stream commands, key scans, blocking pops)
without caring which driver implements them. Each adapter then translates that interface to the underlying
client's idioms.
In this article we will walk through how the new adapters work, how to opt in to each one, and what
to keep in mind about dependencies.
A note on dependencies and backwards compatibility
Because we wanted to release this feature without breaking backwards compatibility, BullMQ still depends on ioredis
even if you intend to use one of the other adapters. In future versions we will rework this so that no dependency
on any specific Redis client is required, only the one you actually choose to use.

If you keep using ioredis, nothing changes. ioredis is still BullMQ's direct dependency and you do not have to install anything extra.
If you want to use node-redis, install redis@^5 in your own project. BullMQ declares it as a peer dependency for the adapter.
If you want to use Bun's built-in RedisClient, you only need to be running on Bun. The class is part of the Bun runtime itself,
there is nothing to install.

Using ioredis (the default)
Nothing has changed. If you were already doing this, keep doing it:
import { Queue, Worker } from "bullmq";

const connection = { host: "localhost", port: 6379 };

const queue = new Queue("emails", { connection });
const worker = new Worker(
"emails",
async (job) => {
await sendEmail(job.data);
},
{ connection },
);

You can also pass a pre built ioredis instance, which is the recommended pattern when you want to
share connection options across multiple queues:
import IORedis from "ioredis";
import { Queue, Worker } from "bullmq";

const connection = new IORedis({
host: "localhost",
port: 6379,
maxRetriesPerRequest: null, // required for blocking workers
});

const queue = new Queue("emails", { connection });
const worker = new Worker(
"emails",
async (job) => {
await sendEmail(job.data);
},
{ connection },
);

Using node-redis (v5)
BullMQ ships a thin wrapper, createNodeRedisClient, that exposes a node-redis raw client through the IRedisClient interface:
import { createClient } from "redis"; // your dependency
import { Queue, Worker, createNodeRedisClient } from "bullmq";

const raw = createClient({ url: "redis://localhost:6379" });
const connection = createNodeRedisClient(raw);

const queue = new Queue("emails", { connection });
const worker = new Worker(
"emails",
async (job) => {
await sendEmail(job.data);
},
{ connection },
);

A few things worth knowing:

The raw client is created and configured by your application, exactly the way node-redis's
own docs recommend. BullMQ never reaches into createClient for you.
You do not need to call await raw.connect() yourself. BullMQ will connect the client lazily the
first time it needs it (see the note on connection lifecycle below). You can still call it if you
prefer to fail fast at startup; it is harmless to do so before passing the client to BullMQ.
RESP3 is fully supported. If you want it, pass RESP: 3 when you create the client:

const raw = createClient({
url: "redis://localhost:6379",
RESP: 3,
});

Using Bun's built-in RedisClient
Bun 1.3 introduced a native RedisClient written in Zig. It is RESP3 only, has no Lua defineCommand style API,
and uses a single send(command, args) method for arbitrary Redis commands. The createBunRedisClient
adapter bridges all of that:
import { RedisClient } from "bun";
import { Queue, Worker, createBunRedisClient } from "bullmq";

const raw = new RedisClient("redis://localhost:6379");
const connection = createBunRedisClient(raw);

const queue = new Queue("emails", { connection });
const worker = new Worker(
"emails",
async (job) => {
await sendEmail(job.data);
},
{ connection },
);

You run this exactly like any other Bun program:
bun run worker.ts

There is no extra dependency to install. Bun.RedisClient is part of the runtime. Note that this only works under Bun,
running the same code under Node.js will fail because the RedisClient class does not exist there.
A note on connection lifecycle
None of the examples above call connect() on the raw client, and that is on purpose: BullMQ opens the
connection for you. The mechanism is different for each driver (ioredis auto-connects in its constructor,
createBunRedisClient does the same, and for node-redis BullMQ calls raw.connect() itself before
issuing the first command), but from the application's point of view the result is the same.
You can still call await raw.connect() yourself before passing the client to BullMQ if you want to
fail fast at startup when Redis is unreachable. BullMQ will then see an already connected client and
skip its own connect step.
Letting BullMQ create the clients for you
If you let BullMQ manage connections (by passing plain connection options instead of a pre built client),
it normally instantiates an ioredis client internally. You can override that behaviour globally by setting RedisConnection.clientFactory:
import { createClient } from "redis";
import { Queue, RedisConnection, createNodeRedisClient } from "bullmq";

RedisConnection.clientFactory = (opts) => {
const raw = createClient({
socket: { host: opts.host, port: opts.port },
username: opts.username,
password: opts.password,
database: opts.db,
});
return createNodeRedisClient(raw);
};

const queue = new Queue("emails", {
connection: { host: "myredis.taskforce.run", port: 32856 },
});

The factory receives the merged connection options BullMQ would normally hand to ioredis, and must return an IRedisClient.
From that point on, every time BullMQ needs a new connection (for Queue, Worker, QueueEvents, FlowProducer, blocking duplicates, etc.)
it will go through your factory.
Inside the factory, do not call await raw.connect() yourself. The factory is expected to return a
constructed but not yet connected client, and BullMQ will open the connection lazily as described in
the lifecycle section above. With node-redis in particular, calling connect() here would race with
BullMQ's own connect attempt and the second one would throw a Socket already opened error.
The same pattern works for Bun:
import { RedisClient } from "bun";
import { Queue, RedisConnection, createBunRedisClient } from "bullmq";

RedisConnection.clientFactory = (opts) => {
const host = opts?.host ?? "localhost";
const port = opts?.port ?? 6379;
const raw = new RedisClient(`redis://${host}:${port}`);
return createBunRedisClient(raw);
};

This is the cleanest way to flip an existing codebase from ioredis to one of the new adapters: set the factory once at startup,
and every existing new Queue(name, { connection: { host, port } }) keeps working.
When does each option make sense?
A few rules of thumb, with the understanding that "it depends on your workload" is the only fully correct answer:

Stay on ioredis if you rely on its Cluster or Sentinel implementation, its specific TLS knobs, or any of its events and options that BullMQ's
existing ecosystem already documents. It is also the safest choice while running on Node.js: the most battle tested path through BullMQ.

Try node-redis if you prefer its API, want RESP3 today on Node.js, or want to keep your application stack on a single Redis client
(node-redis is now what redis officially recommends).

Try createBunRedisClient if your service already runs on Bun and you want to drop one dependency (ioredis) in favour of the runtime built in.

You can also mix: nothing prevents one service from using ioredis and another from using createNodeRedisClient,
against the same Redis instance and against the same queues.

---

# Valkey Performance: 7.2 vs 8.1 vs 9.0 on AWS

URL: https://bullmq.io/articles/benchmarks/valkey-performance-across-versions/
Author: Manuel Astudillo
Published: 2026-04-08
Description: We benchmarked BullMQ against three Valkey versions on Intel, AMD, and Graviton instances to find the best combination of performance and cost.
Tags: articles, valkey, performance, benchmark, aws

Valkey has been moving fast since forking from Redis in early 2024. Three major releases in two
years, each promising performance improvements. But how much of that translates to real
throughput gains when running an actual workload on top of it?
We ran BullMQ against Valkey 7.2, 8.1, and 9.0 on three different AWS instance families to
find out. The goal was not just to measure raw server speed, but to understand how Valkey
version choice interacts with hardware selection, and what combination gives you the most
jobs per dollar.
Test Setup
All benchmarks ran on xlarge instances (4 vCPUs, 8 GB RAM) in us-east-1, each running
both Valkey and the BullMQ worker on the same machine to eliminate network variability:
InstanceCPUArchitectureOn-Demand $/hrc6i.xlargeIntel Xeon 8375C (Ice Lake)x86_64$0.170c7a.xlargeAMD EPYC 9R14 (Genoa)x86_64$0.153c8g.xlargeAWS Graviton 4ARM64$0.136
Valkey ran in Docker with persistence disabled (--save "" --appendonly no) to eliminate
disk I/O as a variable and isolate the comparison to pure in-memory performance.
Each test ran 5 times and we report the mean. BullMQ v5.67, Node.js v24, Amazon Linux 2023.
The full benchmark code is at bullmq-valkey-bench.
Raw Valkey Speed: PING Latency
Before looking at BullMQ workloads, we measured raw Valkey round-trip latency with a simple
PING command. This isolates the server and network stack from any application overhead.
Intel c6iAMD c7aGraviton c8g0 μs25 μs50 μs75 μs100 μs895244724634714635Valkey 7.2Valkey 8.1Valkey 9.0PING round-trip latency in microseconds (lower is better)
The improvement here is significant and consistent across all hardware. Valkey 8.1 brought a
step-change in raw latency, roughly 22-29% faster than 7.2, and 9.0 held those gains. Graviton
posts the lowest absolute latency at 0.032 ms, followed closely by AMD at 0.045 ms. Intel
trails at 0.069 ms, more than double the Graviton figure.
This tells us the Valkey engine itself got meaningfully faster. The question is whether that
translates to real workload throughput.
Bulk Job Insertion
Adding 50,000 jobs at once via addBulk(), which pipelines Redis commands for maximum
throughput:
Intel c6iAMD c7aGraviton c8g010K20K30K40K27,91435,50335,42029,20439,31238,00728,24137,06337,225Valkey 7.2Valkey 8.1Valkey 9.0Jobs per second — Bulk insertion of 50,000 jobs
AMD leads at 39,312 j/s on Valkey 8.1, with Graviton close behind at 38,007 j/s. Intel
caps at 29,204 j/s. Across versions, 8.1 consistently tops this test by 5-11%
over 7.2, while 9.0 sits in between.
Bulk insertion is heavily pipelined, so the gains from lower per-command latency get partially
hidden by the batching. Still, 8.1 manages to squeeze out a measurable improvement.
Single Job Insertion
A more realistic scenario: inserting 5,000 jobs individually with concurrent add() calls:
Intel c6iAMD c7aGraviton c8g09K18K27K36K18,40632,99427,80418,74034,97629,24618,19334,01629,136Valkey 7.2Valkey 8.1Valkey 9.0Jobs per second — Single job insertion (5,000 jobs, concurrent)
This test hits Valkey with individual Lua script executions, one per job. AMD leads
at nearly 35K j/s, with Graviton around 29K and Intel at 18K. Valkey 8.1 shows a
consistent 3-6% gain over 7.2, with 9.0 holding most of that improvement.
Processing Overhead
An important benchmark for queue users: how fast can BullMQ cycle through jobs when
the jobs themselves do no work? This measures the pure overhead of the queue machinery,
all the Lua scripts for dequeuing, locking, acknowledging, and cleaning up.
Concurrency = 1
Intel c6iAMD c7aGraviton c8g03K6K9K12K5,1007,8759,9065,1408,40910,1544,9987,7759,939Valkey 7.2Valkey 8.1Valkey 9.0Jobs per second — Processing overhead, single worker (c=1)
At concurrency 1, everything is sequential: fetch a job, process it, acknowledge it, repeat.
Graviton stands out here at over 10,000 j/s, roughly double the Intel figure. This is where
ARM's low latency pays off the most, every microsecond saved on the round-trip directly
multiplies throughput.
Version differences within each platform are small (2-5%), with 8.1 slightly ahead on AMD
and Graviton.
Concurrency = 10
Intel c6iAMD c7aGraviton c8g07.5K15K23K30K15,21828,62126,92715,30329,81426,83316,00528,01926,909Valkey 7.2Valkey 8.1Valkey 9.0Jobs per second — Processing overhead (c=10)
Scaling to 10 concurrent workers per process, AMD reaches 29,814 j/s on Valkey 8.1.
Graviton sits at ~27K j/s across all versions, remarkably stable. Both are roughly
double Intel's ~15K j/s.
An interesting detail: Intel actually sees a small gain on 9.0 at this concurrency
(16,005 j/s vs 15,218 on 7.2), while AMD shows 8.1 as the clear leader.
Concurrency = 50
Intel c6iAMD c7aGraviton c8g09K18K27K36K17,02731,67228,35418,09334,20728,21517,41730,48328,434Valkey 7.2Valkey 8.1Valkey 9.0Jobs per second — Processing overhead (c=50)
At high concurrency, AMD peaks at 34,207 j/s on Valkey 8.1, the highest throughput in
the entire benchmark. But 9.0 drops to 30,483 j/s, an 11% regression from 8.1.
Graviton tells a completely different story: all three versions land within 1% of each
other at ~28.3K j/s. This is the most stable platform across Valkey versions, and it
suggests that Graviton's low-latency memory subsystem keeps the pipeline fed efficiently
regardless of Valkey's internal changes.
Intel shows a modest improvement from 7.2 to 8.1 (17K → 18K) with 9.0 splitting the
difference.
CPU-Bound Processing
Processing jobs that perform real CPU work (1,000 sin/cos operations per job):
Intel c6iAMD c7aGraviton c8g05.5K11K17K22K9,98021,73718,9149,98921,75418,9889,63421,83818,864Valkey 7.2Valkey 8.1Valkey 9.0Jobs per second — CPU-bound processing (1,000 sin/cos per job, c=10)
When jobs do actual computation, AMD leads at ~21.8K j/s, Graviton follows at ~18.9K,
and Intel sits at ~10K. Version differences essentially vanish because the bottleneck
shifts from Valkey to the Node.js event loop. The queue overhead becomes a small
fraction of the total job time, so a 5% faster Valkey makes no measurable difference.
This is good news for most real-world workloads: your Valkey version choice will not
significantly impact throughput if your jobs do meaningful work.
A Note on io-threads
Valkey supports io-threads to parallelize network I/O across multiple cores. We tested
this configuration and found no measurable benefit for BullMQ workloads.
The reason is architectural: io-threads only parallelize the reading and writing of data
from client sockets. All command execution, including the Lua scripts that BullMQ relies
on for atomic job operations, still runs on the main thread. Since BullMQ's bottleneck is
Lua script execution rather than network I/O, adding I/O threads does not move the needle.
This is worth keeping in mind if you are evaluating multi-threaded datastores like
DragonflyDB as well. Dragonfly uses a shared-nothing architecture where each thread owns
a slice of the keyspace, but BullMQ uses hash tags to keep all keys for a given queue on
the same slot. One queue means one thread, regardless of how many cores the server has.
The only way to benefit from multi-threaded execution is to spread work across multiple
queues with different hash tags.
When you do, the performance gains can be massive: with N queues on N threads, Dragonfly
can theoretically deliver N times the single-queue throughput. Redis and Valkey Cluster
offer a similar scaling path by sharding queues across multiple nodes. We plan to explore
both approaches in a future benchmark.
The Cost Angle
Raw throughput only tells half the story. What often matters in production is throughput per dollar.
Using on-demand pricing in us-east-1 (March 2026) and the best-performing Valkey version
for each platform (8.1 in all cases):
Instance$/hrBest Overhead (c=1)Jobs per $1vs IntelIntel c6i.xlarge$0.1705,140 j/s108.8MbaselineAMD c7a.xlarge$0.1538,409 j/s197.8M1.8xGraviton c8g.xlarge$0.13610,154 j/s268.8M2.5x
Graviton delivers 2.5x more jobs per dollar than Intel for queue-overhead-bound
workloads. Even compared to AMD, Graviton is 36% more cost-efficient. The combination
of lower hourly cost and higher per-core throughput makes it difficult to justify Intel
for new BullMQ deployments on AWS.
For high-concurrency workloads (c=50), the picture using Valkey 8.1 numbers:
Instance$/hrOverhead c=50Jobs per $1vs IntelIntel c6i.xlarge$0.17018,093 j/s383.2MbaselineAMD c7a.xlarge$0.15334,207 j/s805.1M2.1xGraviton c8g.xlarge$0.13628,434 j/s752.3M2.0x
At high concurrency AMD takes the lead in both absolute throughput and cost efficiency,
with Graviton close behind. AMD's advantage at c=50 comes from its raw clock speed
advantage under load, while Graviton's lower price keeps it competitive on a per-dollar
basis.
Key Takeaways
Valkey 8.1 is the sweet spot. It consistently matches or beats both 7.2 and 9.0 across
all platforms and workloads. The PING latency improvements from 7.2 carry through to real
BullMQ throughput, while 9.0 gives back some of those gains on AMD under high concurrency.
If you are running 7.2, upgrading to 8.1 is a clear win. If you are already on 9.0, you
are not losing much in practice.
Graviton is the best value. It posts the fastest sequential processing (10,154 j/s at c=1,
double the Intel number), the lowest PING latency, and the lowest cost. For BullMQ workloads
that are queue-overhead-bound, switching from Intel c6i to Graviton c8g gives you 2.5x more
jobs per dollar. It is also the most version-stable platform: all three Valkey versions
perform within 1% of each other at c=50.
AMD is the throughput king at high concurrency. If you are running workers at c=50 and
need maximum absolute jobs per second, AMD c7a.xlarge with Valkey 8.1 hits 34,207 j/s,
the highest number in all our tests.
Version choice matters less than hardware choice. The difference between the best and
worst Valkey version on a given platform is typically 5-12%. The difference between Intel
and Graviton on the same Valkey version is 50-100%. If you are optimizing for BullMQ
throughput, pick the right instance type first, then worry about Valkey versions.
Real workloads level the field. Once jobs do actual work (CPU or I/O), Valkey version
differences disappear entirely. The queue overhead becomes a rounding error. This means
the version choice is most impactful for fire-and-forget style workloads with very
lightweight jobs.

Quick Takes
Short on time? These highlights cover the key findings:
HighlightYour BullMQ workers may be costing you 2.5x more than they shouldWe ran the numbers across Intel, AMD, and Graviton.HighlightWe upgraded to Valkey 9.0. Here is what happened.Sometimes newer isn't better. Here's what we found.HighlightWe benchmarked every Valkey version. But we asked the wrong question.The real performance gap was somewhere else entirely.

The benchmark source code, Docker Compose files, and GitHub Actions workflow for reproducing
these results are available at bullmq-valkey-bench.

---

# Step Jobs with Flows in BullMQ

URL: https://bullmq.io/articles/guides/step-jobs-with-flows/
Author: Manuel Astudillo
Published: 2026-03-26
Description: Learn how to break large jobs into sequential steps using BullMQ's Flow feature, where each step runs as an independent job with full retry, logging, and observability support.
Tags: articles, nodejs, python, elixir, flows, patterns, guide

Step Jobs with Flows in BullMQ
Some jobs are really several operations in sequence: download a file, transform it, upload the result, send a notification. When you put all of that in a single processor, any failure in the last step forces you to redo everything from scratch, and you cannot tell at a glance which step a job is stuck on.
BullMQ's Flow feature solves this by letting you model each step as its own job, linked together in a parent-child chain. Each step runs independently, with its own retries, its own logs, and its own status, while the flow guarantees they execute in order.
In this article we will cover:

Why step jobs are useful compared to monolithic processors
How to model sequential steps as a nested flow
Passing data between steps using getChildrenValues()
Dynamic flows where steps are determined at runtime
Error handling and partial progress

The Problem with Monolithic Jobs
Consider a job that processes a user-uploaded video:

Download the raw file from S3
Transcode it to the target format
Generate a thumbnail
Upload the results back to S3
Notify the user via email

A naive implementation puts everything in one processor:
const worker = new Worker( "video-pipeline", async (job) => { const file = await downloadFromS3(job.data.s3Key); const transcoded = await transcode(file, "mp4"); const thumbnail = await generateThumbnail(transcoded); await uploadToS3(transcoded, thumbnail); await notifyUser(job.data.userId, job.data.videoId); return { status: "complete" }; }, { connection },);
This works, but has several problems:

Wasted work on failure. If the upload step fails after a 10-minute transcode, the entire job retries from scratch, re-downloading and re-transcoding.
No visibility. From the outside, the job is either "active" or "completed". You can't tell whether it's stuck on the download or the transcode.
One retry policy for everything. Maybe the download should retry 5 times with exponential backoff, but the notification should only retry once. A monolithic job forces a single retry configuration.
Resource conflicts. The transcode step is CPU-heavy, so you want few concurrent workers. But the download step is I/O-bound and could run with high concurrency. In a monolithic job, the concurrency setting applies to the whole pipeline.

Flows: Sequential Steps as Nested Children
BullMQ flows let you express dependencies between jobs. A parent job will not start processing until all its children have completed successfully. By nesting children one level deep at a time, you create a chain, each step waits for the previous one:
step-5 (notify) ← parent, runs last
└─ step-4 (upload) ← runs after step-3
└─ step-3 (thumbnail) ← runs after step-2
└─ step-2 (transcode) ← runs after step-1
└─ step-1 (download) ← runs first (leaf node)

The deepest child (the leaf) runs first because it has no dependencies. When it completes, its parent becomes eligible. This continues up the chain until the root job runs last.
Creating the Flow
Use FlowProducer to add the entire chain atomically, either all jobs are created or none:

TypeScriptPythonElixir

create-flow.ts

import { FlowProducer } from 'bullmq';const flowProducer = new FlowProducer({ connection });const flow = await flowProducer.add({ name: 'notify', queueName: 'video-pipeline', data: { userId: 'user-42', videoId: 'vid-123' }, children: [ { name: 'upload', queueName: 'video-pipeline', data: { videoId: 'vid-123' }, children: [ { name: 'thumbnail', queueName: 'video-pipeline', data: { videoId: 'vid-123' }, children: [ { name: 'transcode', queueName: 'video-pipeline', data: { videoId: 'vid-123', format: 'mp4' }, children: [ { name: 'download', queueName: 'video-pipeline', data: { s3Key: 'uploads/raw/vid-123.mov' }, }, ], }, ], }, ], }, ],});

create_flow.py

from bullmq import FlowProducerflow_producer = FlowProducer(redisOpts={"host": "localhost", "port": 6379})flow = await flow_producer.add({ "name": "notify", "queueName": "video-pipeline", "data": {"userId": "user-42", "videoId": "vid-123"}, "children": [ { "name": "upload", "queueName": "video-pipeline", "data": {"videoId": "vid-123"}, "children": [ { "name": "thumbnail", "queueName": "video-pipeline", "data": {"videoId": "vid-123"}, "children": [ { "name": "transcode", "queueName": "video-pipeline", "data": {"videoId": "vid-123", "format": "mp4"}, "children": [ { "name": "download", "queueName": "video-pipeline", "data": {"s3Key": "uploads/raw/vid-123.mov"}, }, ], }, ], }, ], }, ],})

create_flow.ex

{:ok, flow} = BullMQ.FlowProducer.add( %{ name: "notify", queue_name: "video-pipeline", data: %{userId: "user-42", videoId: "vid-123"}, children: [ %{ name: "upload", queue_name: "video-pipeline", data: %{videoId: "vid-123"}, children: [ %{ name: "thumbnail", queue_name: "video-pipeline", data: %{videoId: "vid-123"}, children: [ %{ name: "transcode", queue_name: "video-pipeline", data: %{videoId: "vid-123", format: "mp4"}, children: [ %{ name: "download", queue_name: "video-pipeline", data: %{s3Key: "uploads/raw/vid-123.mov"}, }, ], }, ], }, ], }, ], }, connection: :redis)

The execution order is: download → transcode → thumbnail → upload → notify.
Processing Each Step
A single worker can handle all steps by switching on the job name:

TypeScriptPythonElixir

worker.ts

import { Worker } from 'bullmq';const worker = new Worker('video-pipeline', async (job) => { switch (job.name) { case 'download': return await handleDownload(job); case 'transcode': return await handleTranscode(job); case 'thumbnail': return await handleThumbnail(job); case 'upload': return await handleUpload(job); case 'notify': return await handleNotify(job); default: throw new Error(`Unknown step: ${job.name}`); }}, { connection });

worker.py

from bullmq import Workerasync def process(job, token): if job.name == "download": return await handle_download(job) elif job.name == "transcode": return await handle_transcode(job) elif job.name == "thumbnail": return await handle_thumbnail(job) elif job.name == "upload": return await handle_upload(job) elif job.name == "notify": return await handle_notify(job) else: raise Exception(f"Unknown step: {job.name}")worker = Worker("video-pipeline", process, { "connection": {"host": "localhost", "port": 6379}})

worker.ex

defmodule VideoPipeline do def process(%BullMQ.Job{name: "download"} = job), do: handle_download(job) def process(%BullMQ.Job{name: "transcode"} = job), do: handle_transcode(job) def process(%BullMQ.Job{name: "thumbnail"} = job), do: handle_thumbnail(job) def process(%BullMQ.Job{name: "upload"} = job), do: handle_upload(job) def process(%BullMQ.Job{name: "notify"} = job), do: handle_notify(job)end{:ok, _worker} = BullMQ.Worker.start_link( queue: "video-pipeline", connection: :redis, processor: &VideoPipeline.process/1)

Or if you prefer, use separate queues for each step. This lets you assign different
concurrency settings and different worker pools to I/O-bound vs CPU-bound steps:
const flow = await flowProducer.add({ name: "notify", queueName: "notifications", data: { userId: "user-42", videoId: "vid-123" }, children: [ { name: "upload", queueName: "uploads", // ...nested children with different queueNames }, ],});

TypeScriptPythonElixir

workers.ts

import { Worker } from 'bullmq';const downloadWorker = new Worker('downloads', handleDownload, { connection, concurrency: 20, // I/O-bound, high concurrency});const mediaWorker = new Worker('media-processing', async (job) => { switch (job.name) { case 'transcode': return await handleTranscode(job); case 'thumbnail': return await handleThumbnail(job); }}, { connection, concurrency: 2, // CPU-bound, low concurrency});const uploadWorker = new Worker('uploads', handleUpload, { connection, concurrency: 10,});const notifyWorker = new Worker('notifications', handleNotify, { connection, concurrency: 50,});

workers.py

from bullmq import Workerdownload_worker = Worker("downloads", handle_download, { "connection": connection, "concurrency": 20, # I/O-bound, high concurrency})async def process_media(job, token): if job.name == "transcode": return await handle_transcode(job) elif job.name == "thumbnail": return await handle_thumbnail(job)media_worker = Worker("media-processing", process_media, { "connection": connection, "concurrency": 2, # CPU-bound, low concurrency})upload_worker = Worker("uploads", handle_upload, { "connection": connection, "concurrency": 10,})notify_worker = Worker("notifications", handle_notify, { "connection": connection, "concurrency": 50,})

workers.ex

# In your application's supervision treechildren = [ {BullMQ.Worker, queue: "downloads", connection: :redis, processor: &handle_download/1, concurrency: 20}, {BullMQ.Worker, queue: "media-processing", connection: :redis, processor: &MediaProcessor.process/1, concurrency: 2}, {BullMQ.Worker, queue: "uploads", connection: :redis, processor: &handle_upload/1, concurrency: 10}, {BullMQ.Worker, queue: "notifications", connection: :redis, processor: &handle_notify/1, concurrency: 50},]Supervisor.start_link(children, strategy: :one_for_one)

Passing Data Between Steps
Each step's return value is available to its parent via getChildrenValues(). This is how the output of one step becomes the input for the next.

TypeScriptPythonElixir

steps.ts

async function handleDownload(job) { const localPath = await downloadFromS3(job.data.s3Key); // Return value is stored and accessible by the parent (transcode) return { localPath };}async function handleTranscode(job) { // Get results from child jobs (download step) const childrenValues = await job.getChildrenValues(); // childrenValues is keyed by the fully qualified job key // For a single child, just grab the first value const downloadResult = Object.values(childrenValues)[0]; const outputPath = await transcode(downloadResult.localPath, job.data.format); return { outputPath };}async function handleThumbnail(job) { const childrenValues = await job.getChildrenValues(); const transcodeResult = Object.values(childrenValues)[0]; const thumbnailPath = await generateThumbnail(transcodeResult.outputPath); return { thumbnailPath, videoPath: transcodeResult.outputPath };}async function handleUpload(job) { const childrenValues = await job.getChildrenValues(); const thumbnailResult = Object.values(childrenValues)[0]; const videoUrl = await uploadToS3(thumbnailResult.videoPath); const thumbUrl = await uploadToS3(thumbnailResult.thumbnailPath); return { videoUrl, thumbUrl };}async function handleNotify(job) { const childrenValues = await job.getChildrenValues(); const uploadResult = Object.values(childrenValues)[0]; await sendEmail(job.data.userId, { videoUrl: uploadResult.videoUrl, thumbnailUrl: uploadResult.thumbUrl, }); return { notified: true };}

steps.py

async def handle_download(job, token): local_path = await download_from_s3(job.data["s3Key"]) # Return value is stored and accessible by the parent (transcode) return {"localPath": local_path}async def handle_transcode(job, token): # Get results from child jobs (download step) children_values = await job.getChildrenValues() # children_values is keyed by the fully qualified job key # For a single child, just grab the first value download_result = list(children_values.values())[0] output_path = await transcode(download_result["localPath"], job.data["format"]) return {"outputPath": output_path}async def handle_thumbnail(job, token): children_values = await job.getChildrenValues() transcode_result = list(children_values.values())[0] thumbnail_path = await generate_thumbnail(transcode_result["outputPath"]) return {"thumbnailPath": thumbnail_path, "videoPath": transcode_result["outputPath"]}async def handle_upload(job, token): children_values = await job.getChildrenValues() thumbnail_result = list(children_values.values())[0] video_url = await upload_to_s3(thumbnail_result["videoPath"]) thumb_url = await upload_to_s3(thumbnail_result["thumbnailPath"]) return {"videoUrl": video_url, "thumbUrl": thumb_url}async def handle_notify(job, token): children_values = await job.getChildrenValues() upload_result = list(children_values.values())[0] await send_email(job.data["userId"], { "videoUrl": upload_result["videoUrl"], "thumbnailUrl": upload_result["thumbUrl"], }) return {"notified": True}

steps.ex

defmodule VideoPipeline do def handle_download(%BullMQ.Job{data: data}) do local_path = download_from_s3(data["s3Key"]) # Return value is stored and accessible by the parent (transcode) {:ok, %{localPath: local_path}} end def handle_transcode(%BullMQ.Job{} = job) do # Get results from child jobs (download step) {:ok, children_values} = BullMQ.Job.get_children_values(job) # children_values is keyed by the fully qualified job key # For a single child, just grab the first value [download_result | _] = Map.values(children_values) output_path = transcode(download_result["localPath"], job.data["format"]) {:ok, %{outputPath: output_path}} end def handle_thumbnail(%BullMQ.Job{} = job) do {:ok, children_values} = BullMQ.Job.get_children_values(job) [transcode_result | _] = Map.values(children_values) thumbnail_path = generate_thumbnail(transcode_result["outputPath"]) {:ok, %{thumbnailPath: thumbnail_path, videoPath: transcode_result["outputPath"]}} end def handle_upload(%BullMQ.Job{} = job) do {:ok, children_values} = BullMQ.Job.get_children_values(job) [thumbnail_result | _] = Map.values(children_values) video_url = upload_to_s3(thumbnail_result["videoPath"]) thumb_url = upload_to_s3(thumbnail_result["thumbnailPath"]) {:ok, %{videoUrl: video_url, thumbUrl: thumb_url}} end def handle_notify(%BullMQ.Job{} = job) do {:ok, children_values} = BullMQ.Job.get_children_values(job) [upload_result | _] = Map.values(children_values) send_email(job.data["userId"], %{ videoUrl: upload_result["videoUrl"], thumbnailUrl: upload_result["thumbUrl"] }) {:ok, %{notified: true}} endend

Since each step in a linear chain has exactly one child, Object.values(childrenValues)[0] (or its equivalent in your language) is a simple way to get the previous step's result.
Dynamic Flows
Sometimes you don't know the steps upfront. Maybe the pipeline depends on the file type, or certain steps are conditional. You can build the flow tree programmatically:

TypeScriptPythonElixir

dynamic-flow.ts

import { FlowProducer } from 'bullmq';interface StepDefinition { name: string; queueName: string; data: Record<string, any>;}function buildStepChain(steps: StepDefinition[]) { // Steps are listed in execution order: first step runs first. // We build the nested structure bottom-up (leaf = first step). let chain: any = undefined; // Iterate in reverse to nest from the last step inward for (let i = steps.length - 1; i >= 0; i--) { const step = steps[i]; chain = { name: step.name, queueName: step.queueName, data: step.data, ...(chain ? { children: [chain] } : {}), }; } return chain;}// Usageconst flowProducer = new FlowProducer({ connection });const steps: StepDefinition[] = [ { name: 'download', queueName: 'pipeline', data: { s3Key: 'raw/vid.mov' } }, { name: 'transcode', queueName: 'pipeline', data: { format: 'mp4' } }, { name: 'upload', queueName: 'pipeline', data: { bucket: 'output' } },];// Conditionally add a notification stepif (shouldNotify) { steps.push({ name: 'notify', queueName: 'pipeline', data: { userId: 'u-1' } });}const flow = await flowProducer.add(buildStepChain(steps));

dynamic_flow.py

from bullmq import FlowProducerdef build_step_chain(steps): """Build a nested flow from a flat list of steps (first runs first).""" chain = None for step in reversed(steps): node = { "name": step["name"], "queueName": step["queueName"], "data": step["data"], } if chain: node["children"] = [chain] chain = node return chain# Usageflow_producer = FlowProducer(redisOpts={"host": "localhost", "port": 6379})steps = [ {"name": "download", "queueName": "pipeline", "data": {"s3Key": "raw/vid.mov"}}, {"name": "transcode", "queueName": "pipeline", "data": {"format": "mp4"}}, {"name": "upload", "queueName": "pipeline", "data": {"bucket": "output"}},]# Conditionally add a notification stepif should_notify: steps.append({"name": "notify", "queueName": "pipeline", "data": {"userId": "u-1"}})flow = await flow_producer.add(build_step_chain(steps))

dynamic_flow.ex

defmodule StepChain do @doc "Build a nested flow from a flat list of steps (first runs first)." def build(steps) do steps |> Enum.reverse() |> Enum.reduce(nil, fn step, chain -> node = %{ name: step.name, queue_name: step.queue_name, data: step.data } if chain, do: Map.put(node, :children, [chain]), else: node end) endend# Usagesteps = [ %{name: "download", queue_name: "pipeline", data: %{s3Key: "raw/vid.mov"}}, %{name: "transcode", queue_name: "pipeline", data: %{format: "mp4"}}, %{name: "upload", queue_name: "pipeline", data: %{bucket: "output"}},]steps = if should_notify do steps ++ [%{name: "notify", queue_name: "pipeline", data: %{userId: "u-1"}}]else stepsend{:ok, flow} = BullMQ.FlowProducer.add( StepChain.build(steps), connection: :redis)

The buildStepChain helper takes a flat list of steps in execution order and builds the nested structure that FlowProducer expects.
Error Handling and Partial Progress
One of the main advantages of step jobs is that failures are scoped to a single step.
If the upload step fails after a successful transcode, only the upload step retries, the transcode result is preserved as the completed child's return value.
Per-Step Retry Configuration
Each job in the flow can have its own retry settings:

TypeScriptPythonElixir

retry-config.ts

const flow = await flowProducer.add({ name: 'notify', queueName: 'pipeline', data: { userId: 'user-42' }, opts: { attempts: 2, backoff: { type: 'fixed', delay: 5000 }, }, children: [ { name: 'upload', queueName: 'pipeline', data: { bucket: 'output' }, opts: { attempts: 5, backoff: { type: 'exponential', delay: 1000 }, }, children: [ { name: 'download', queueName: 'pipeline', data: { s3Key: 'raw/vid.mov' }, opts: { attempts: 5, backoff: { type: 'exponential', delay: 1000 }, }, }, ], }, ],});

retry_config.py

flow = await flow_producer.add({ "name": "notify", "queueName": "pipeline", "data": {"userId": "user-42"}, "opts": { "attempts": 2, "backoff": {"type": "fixed", "delay": 5000}, }, "children": [ { "name": "upload", "queueName": "pipeline", "data": {"bucket": "output"}, "opts": { "attempts": 5, "backoff": {"type": "exponential", "delay": 1000}, }, "children": [ { "name": "download", "queueName": "pipeline", "data": {"s3Key": "raw/vid.mov"}, "opts": { "attempts": 5, "backoff": {"type": "exponential", "delay": 1000}, }, }, ], }, ],})

retry_config.ex

{:ok, flow} = BullMQ.FlowProducer.add( %{ name: "notify", queue_name: "pipeline", data: %{userId: "user-42"}, opts: %{ attempts: 2, backoff: %{type: "fixed", delay: 5000} }, children: [ %{ name: "upload", queue_name: "pipeline", data: %{bucket: "output"}, opts: %{ attempts: 5, backoff: %{type: "exponential", delay: 1000} }, children: [ %{ name: "download", queue_name: "pipeline", data: %{s3Key: "raw/vid.mov"}, opts: %{ attempts: 5, backoff: %{type: "exponential", delay: 1000} }, }, ], }, ], }, connection: :redis)

Observing Step Progress
Since each step is a regular BullMQ job, you can inspect it with the standard APIs:
import { Queue } from "bullmq";const queue = new Queue("pipeline", { connection });// Get all active jobs to see which step is currently runningconst active = await queue.getActive();active.forEach((job) => { console.log(`Step "${job.name}" is active (job ${job.id})`);});// Get failed jobs to find which step brokeconst failed = await queue.getFailed();failed.forEach((job) => { console.log(`Step "${job.name}" failed: ${job.failedReason}`);});
You can also use the waiting-children state to see which parent step is waiting for its predecessor to complete:
const waitingChildren = await queue.getWaitingChildren();waitingChildren.forEach((job) => { console.log(`Step "${job.name}" waiting for children to complete`);});
When to Use Step Jobs vs Monolithic Jobs
Step jobs via flows are not always the right choice. Here is a quick summary:

Scenario
Recommendation

Job takes < 1 second total
Monolithic, overhead of flows isn't worth it

Steps have different retry needs
Step jobs, each step has its own attempts and backoff

You need visibility into which step failed
Step jobs, each step has its own status and logs

Steps need different concurrency / worker pools
Step jobs with separate queues

Steps can run in parallel
Flow with multiple children at the same level

Simple, fast, uniform work
Monolithic, simpler to reason about

Best Practices

Return only serializable data from each step. The return value is stored in Redis and passed to the parent via getChildrenValues(). Keep it small, return file paths or URLs, not file contents.

Use the buildStepChain helper for dynamic pipelines. It is easier to reason about a flat list than a deeply nested structure.

Separate queues for heterogeneous steps. If some steps are CPU-bound and others are I/O-bound, separate queues let you tune concurrency independently.

Use removeOnComplete to keep Redis clean. Steps in a completed flow are no longer needed. Set removeOnComplete: true on each step's options, or configure it as a queue default via queuesOptions on the FlowProducer.add() call.

Monitor the waiting-children state. This tells you at a glance where in the pipeline the flow is everything below the waiting-children job has been processed, and the next step is either active or waiting.

What's Next
This article covered sequential step jobs, each step depends on exactly one predecessor. But flows also support fan-out patterns where a parent waits for multiple children to complete in parallel. You can combine both patterns: some steps run in sequence, while others fan out and converge. See the Flows documentation for more on tree-shaped flows.

Ready to try flows? Check out the FlowProducer API reference
and the Flows guide in the official documentation.

---

# Properly Cancelling Jobs in BullMQ

URL: https://bullmq.io/articles/guides/properly-cancelling-jobs/
Author: Manuel Astudillo
Published: 2026-03-25
Description: Learn how to gracefully cancel running jobs in BullMQ using the AbortController API, including patterns for local and remote cancellation via Redis Pub/Sub.
Tags: articles, nodejs, workers, cancellation, guide, patterns

Properly Cancelling Jobs in BullMQ
Cancelling a running job sounds simple, but getting it right in a distributed system requires careful thought. A naive approach, killing a process or ignoring a running job, could lead to resource leaks, orphaned connections, and/or inconsistent state. BullMQ provides a clean, standards-based cancellation mechanism built on the AbortController Node.js API, giving you fine-grained control over how and when jobs stop.
In this article, we'll cover:

Local cancellation using the built-in AbortSignal API
Remote cancellation across workers using Redis Pub/Sub
Graceful shutdown and clean timeouts using signal composition
Best practices for cleanup and error handling

The Problem with Killing Jobs
When a long-running job needs to stop, whether because a user cancelled an operation, a timeout was exceeded, or the system is shutting down, you can't just abandon it. Consider what happens if you simply ignore a running job:

HTTP requests continue consuming bandwidth and remote server resources
Database transactions remain open, holding locks
File handles and temporary files are never cleaned up
External services are never notified of the abort
Memory allocated for the operation may never be freed

What you need is cooperative cancellation: the job processor itself decides how to wind down gracefully when asked to stop. This is exactly what BullMQ's AbortSignal integration provides.
Local Cancellation with AbortSignal
BullMQ workers pass an optional AbortSignal as the third parameter to your processor function. When you call worker.cancelJob(jobId), the signal is aborted, and your processor can react immediately.
Important: cancelJob() does not fail the job by itself. It only aborts the signal. Your processor must reject/throw when it detects cancellation; then BullMQ applies normal failure/retry rules based on the error type.
Basic Setup
Cancellation in BullMQ is cooperative: calling cancelJob() only aborts the signal, the processor must listen for that signal and stop on its own. If your processor ignores the signal, the job keeps running.
import { Worker } from "bullmq";

const worker = new Worker(
"my-queue",
async (job, token, signal) => {
// signal is an AbortSignal, your code must react to it
// for cancellation to actually work. See patterns below.
return await doWork(job.data, signal);
},
{ connection: { host: "localhost", port: 6379 } },
);

To request cancellation, call cancelJob on the worker instance. This aborts the signal but does not stop or fail the job by itself, your processor must handle it:
// Cancel a specific job (aborts its signal)
worker.cancelJob("job-123");

// Cancel with a reason (accessible via signal.reason)
worker.cancelJob("job-123", "User requested cancellation");

// Cancel all active jobs (useful during shutdown)
worker.cancelAllJobs("System shutting down");

The sections below show how to write processors that actually respond to cancellation.
Event-Based Pattern (Recommended)
The most responsive approach is to listen for the abort event on the signal. When the event fires, reject the promise directly, no flags, no polling loops:
const worker = new Worker(
"my-queue",
async (job, token, signal) => {
return new Promise((resolve, reject) => {
// Listen for abort event, reject immediately
signal?.addEventListener("abort", () => {
console.log(`Job ${job.id} cancellation requested`);

// Clean up resources
clearInterval(interval);

// Reject with error
reject(new Error("Job was cancelled"));
});

// Your processing logic
const interval = setInterval(() => {
processNextItem();
}, 100);
});
},
{ connection },
);

Why this works:

Immediate response: No polling delay; the abort listener fires the instant cancellation is requested
More efficient: No CPU wasted checking flags in loops
Cleaner code: Separation of concerns between work and cancellation
Standard pattern: Matches how Web APIs like fetch() handle AbortSignal

If cancellation is user-initiated and should not retry, throw UnrecoverableError instead of Error.
Using with fetch and Other Native APIs
One of the most useful aspects of the AbortSignal standard is that many Web APIs support it natively. You can pass the signal directly to fetch, and the HTTP request will be truly cancelled at the network level:
const worker = new Worker(
"scraper-queue",
async (job, token, signal) => {
// The signal cancels the actual HTTP request, not just the job
const response = await fetch(job.data.url, { signal });
const html = await response.text();
return { length: html.length, status: response.status };
},
{ connection },
);

When the signal fires, fetch throws an AbortError which propagates to the worker automatically.
This pattern works with any API that accepts an AbortSignal:

fetch(url, { signal }), HTTP requests
addEventListener(event, handler, { signal }), auto-removes listener on abort
Some database and HTTP clients
Node.js APIs that support AbortSignal

Controlling Retry Behavior
How you throw determines what happens to the job after cancellation:
Error TypeRetries?Use Casenew Error(...)Yes (if attempts remain)Transient cancellation, retry laternew UnrecoverableError(...)NoPermanent cancellation, user cancelled
import { Worker, UnrecoverableError } from "bullmq";

// With regular Error, job will retry if attempts remain
const worker = new Worker(
"retryQueue",
async (job, token, signal) => {
return new Promise((resolve, reject) => {
signal?.addEventListener("abort", () => {
reject(new Error("Cancelled, will retry"));
});

// Your work...
});
},
{ connection },
);

// Set attempts when adding jobs
await queue.add("task", data, { attempts: 3 });

import { Worker, UnrecoverableError } from "bullmq";

// With UnrecoverableError, no retries, cancellation is permanent
const worker = new Worker(
"noRetryQueue",
async (job, token, signal) => {
return new Promise((resolve, reject) => {
signal?.addEventListener("abort", () => {
reject(new UnrecoverableError("Cancelled permanently"));
});

// Your work...
});
},
{ connection },
);

You can also inspect signal.reason to decide at runtime:
const worker = new Worker(
"my-queue",
async (job, token, signal) => {
return new Promise((resolve, reject) => {
signal?.addEventListener("abort", () => {
const reason = signal.reason ?? "cancelled";
if (reason === "user-cancelled") {
reject(new UnrecoverableError("Cancelled by user"));
} else {
reject(new Error(`Cancelled: ${reason}`));
}
});

// Your work...
});
},
{ connection },
);

Remote Cancellation via Redis Pub/Sub
The built-in worker.cancelJob() method works on the local worker instance, so you need a reference to the worker object that is processing the job. But what if the cancellation request comes from a different process? For example:

A web server receives a user's cancel request, but the job is running on a separate worker process
You have multiple worker servers and don't know which one is processing a specific job
A monitoring dashboard needs to cancel jobs across the cluster

The solution is to use Redis Pub/Sub as a signaling channel between the process that wants to cancel the job and the worker that is running it.
Architecture
┌─────────────┐ Redis Pub/Sub ┌──────────────┐
│ API Server │ ──── PUBLISH ────────► │ Worker 1 │
│ (cancel req)│ {action, jobId, │ (processing │
│ │ reason, ...} │ jobs) │
└─────────────┘ └──────────────┘
┌──────────────┐
──── PUBLISH ────────► │ Worker 2 │
same payload │ (processing │
│ jobs) │
└──────────────┘

Implementation
We recommend using a single control channel per queue, following the same prefix and queue name that BullMQ already uses for its Redis keys: {prefix}:{queueName}:control. An action field in each message distinguishes the operation type, here we use "cancel", but the same channel could carry other control actions in the future (e.g. worker concurrency changes).
By default the prefix is bull, so for a queue named my-queue the channel would be bull:my-queue:control.
If you only have a handful of queues, you can simplify further by using a single shared channel for all of them, for example {prefix}:control, and including a queueName field in each message so workers can filter. This avoids one subscription per queue and keeps the setup minimal.
The per-queue channel only starts to matter when you have many queues and want to avoid dispatching irrelevant messages, but since control messages are infrequent in practice, either approach works well.
Important: Pub/Sub is fire-and-forget. If no subscriber is listening when a message is published, it is lost, Redis does not buffer Pub/Sub messages. This means:

If all workers for a queue are down when you publish a cancel request, nobody will receive it.
The publisher gets no confirmation that the message was actually processed.

For most cancellation use cases this is fine, and is always possible to send this message several times to increase the chances the actual worker processing the job receives it.
First, set up the worker to subscribe to the control channel:
import { Worker } from "bullmq";
import Redis from "ioredis";

const connection = { host: "localhost", port: 6379 };
const queueName = "my-queue";
const prefix = "bull"; // default BullMQ prefix

// Create a dedicated Redis subscriber for control signals.
// Important: A Redis connection in subscribe mode cannot be used
// for other commands, so we need a separate connection.s
const subscriber = new Redis(connection);
const CONTROL_CHANNEL = `${prefix}:${queueName}:control`;

const worker = new Worker(
queueName,
async (job, token, signal) => {
// Pass the signal to operations that support it.
// If cancelled, the error propagates and the worker fails the job automatically.
return await doExpensiveWork(job.data, signal);
},
{ connection },
);

// Listen for remote control messages
subscriber.subscribe(CONTROL_CHANNEL);
subscriber.on("message", (channel, message) => {
if (channel === CONTROL_CHANNEL) {
try {
const payload = JSON.parse(message);
if (payload.action !== "cancel") {
return;
}

const { jobId, reason } = payload;
// cancelJob returns true if this worker was processing that job
const cancelled = worker.cancelJob(jobId, reason);
if (cancelled) {
console.log(`Cancelled job ${jobId} from remote request`);
}
} catch (err) {
console.error("Invalid cancel message:", err);
}
}
});

Then, from any other process (API server, admin dashboard, CLI tool, etc.), publish a cancel message:
import Redis from "ioredis";

const redis = new Redis({ host: "localhost", port: 6379 });
const queueName = "my-queue";
const prefix = "bull"; // must match the worker's prefix
const CONTROL_CHANNEL = `${prefix}:${queueName}:control`;

async function cancelJobRemotely(jobId: string, reason?: string) {
await redis.publish(
CONTROL_CHANNEL,
JSON.stringify({
action: "cancel",
jobId,
reason: reason ?? "Remote cancellation",
}),
);
console.log(`Cancel request published for job ${jobId}`);
}

// Cancel from an API endpoint
app.post("/api/jobs/:id/cancel", async (req, res) => {
await cancelJobRemotely(req.params.id, "Cancelled by user");
res.json({ status: "cancel-requested" });
});

How It Works

Every worker process subscribes to its queue-scoped control channel (for example: bull:my-queue:control)
When a cancellation request arrives from any source, it publishes a message to that channel
All worker processes receive the message and call worker.cancelJob(jobId)
Only the worker actually processing that job will find a match, and cancelJob returns true
The AbortSignal fires in the processor, and the job cleans up gracefully

Since Redis Pub/Sub delivers messages to all subscribers, this works regardless of how many worker processes you have or which one is handling the job. The overhead is usually negligible: the message is a small JSON payload, and cancelJob on a non-matching worker is an in-memory no-op.
Production-Ready Version
For a production deployment, you'll could encapsulate this pattern into a reusable helper:
import { Job, Worker, WorkerOptions } from "bullmq";
import Redis from "ioredis";

interface CancellableWorkerOptions extends WorkerOptions {
controlChannel?: string;
}

function createCancellableWorker<T, R>(
queueName: string,
processor: (
job: Job<T, R>,
token?: string,
signal?: AbortSignal,
) => Promise<R>,
opts: CancellableWorkerOptions,
) {
const prefix = String(opts.prefix ?? "bull");
const controlChannel =
opts.controlChannel ?? `${prefix}:${queueName}:control`;
const worker = new Worker(queueName, processor, opts);

// Dedicated subscriber connection
const subscriber = new Redis(opts.connection as Redis.RedisOptions);
subscriber.subscribe(controlChannel);

subscriber.on("message", (_channel, message) => {
try {
const payload = JSON.parse(message);
if (payload.action !== "cancel") {
return;
}

const { jobId, reason } = payload;
worker.cancelJob(jobId, reason);
} catch (err) {
console.error("Invalid cancel message:", err);
}
});

// Clean up subscriber when worker closes
const originalClose = worker.close.bind(worker);
worker.close = async (force?: boolean) => {
await subscriber.unsubscribe(controlChannel);
await subscriber.quit();
return originalClose(force);
};

return { worker, controlChannel };
}

Usage:
const { worker, controlChannel } = createCancellableWorker(
"video-processing",
async (job, token, signal) => {
return await transcodeVideo(job.data.videoUrl, signal);
},
{ connection },
);

And to cancel from anywhere:
const redis = new Redis(connection);
await redis.publish(
controlChannel,
JSON.stringify({
action: "cancel",
jobId: "job-123",
reason: "User cancelled upload",
}),
);

Combining Cancellation with Graceful Shutdown
BullMQ's worker.close() is already graceful: it stops fetching new jobs and waits for all currently active processors to finish before closing Redis connections and cleaning up. In many cases that's all you need, the in-flight jobs complete normally, and the worker shuts down cleanly.
Cancellation becomes useful when you want to speed up that graceful shutdown. If your processors run long tasks (video transcoding, large data imports, etc.) and you can't afford to wait minutes for them to finish, cancelling active jobs lets the processors abort early so worker.close() returns faster. But this is a trade-off: cancelled jobs will be failed (and possibly retried), whereas letting them finish means no work is wasted.
Choose the approach that fits your use case:
const worker = new Worker(
"my-queue",
async (job, token, signal) => {
return await doWork(job.data, signal);
},
{ connection },
);

process.on("SIGTERM", async () => {
// Option 1: Just close gracefully — wait for active jobs to finish
await worker.close();

// Option 2: Cancel active jobs to speed up shutdown.
// Pause first so the worker doesn't pick up the cancelled jobs again
// before close() finishes.
// await worker.pause();
// worker.cancelAllJobs('Process shutting down');
// await worker.close();

process.exit(0);
});

Implementing Clean Job Timeouts
A job that runs forever is just as problematic as one that crashes. Timeouts and cancellation are two sides of the same coin: the processor still needs to stop cooperatively and clean up. The AbortSignal API makes this straightforward with AbortSignal.any().
Composing Signals with AbortSignal.timeout()
Modern Node.js (v17.3+) provides AbortSignal.timeout(), which creates a signal that auto-aborts after a specified duration. Combined with AbortSignal.any(), you can merge the worker's cancellation signal with a timeout signal:
const worker = new Worker(
"my-queue",
async (job, token, signal) => {
// Create a timeout signal (30 seconds)
const timeoutSignal = AbortSignal.timeout(30_000);

// Combine timeout + external cancellation (when provided)
const signals: AbortSignal[] = [timeoutSignal];
if (signal) {
signals.push(signal);
}
const combinedSignal = AbortSignal.any(signals);

// Pass the combined signal. If it fires, the operation throws and
// the worker catches it automatically to fail the job.
const response = await fetch(job.data.url, { signal: combinedSignal });
return await response.json();
},
{ connection },
);

Per-Job Configurable Timeouts
In practice, different jobs need different timeouts. A video transcode shouldn't have the
same limit as a thumbnail resize. Store the timeout in job data and create the signal dynamically:
// When adding jobs, specify the timeout
await queue.add("process-video", {
videoUrl: "https://...",
timeoutMs: 300_000, // 5 minutes for large videos
});

await queue.add("process-thumbnail", {
imageUrl: "https://...",
timeoutMs: 10_000, // 10 seconds for thumbnails
});

// In the worker
const worker = new Worker(
"media-queue",
async (job, token, signal) => {
const timeoutMs = job.data.timeoutMs ?? 60_000; // Default: 1 minute
const timeoutSignal = AbortSignal.timeout(timeoutMs);
const signals: AbortSignal[] = [timeoutSignal];
if (signal) {
signals.push(signal);
}
const combinedSignal = AbortSignal.any(signals);

return await processMedia(job.data, combinedSignal);
},
{ connection },
);

Reusable Timeout Helper
Create a reusable utility that composes the worker signal with a timeout and passes the resulting signal to the underlying operation:
async function withTimeout<T>(
operation: (signal: AbortSignal) => Promise<T>,
timeoutMs: number,
parentSignal?: AbortSignal,
): Promise<T> {
const timeoutSignal = AbortSignal.timeout(timeoutMs);
const combinedSignal = parentSignal
? AbortSignal.any([parentSignal, timeoutSignal])
: timeoutSignal;

return await operation(combinedSignal);
}

// Usage in a worker
const worker = new Worker(
"my-queue",
async (job, token, signal) => {
// Wrap any async operation with a timeout
const result = await withTimeout(
(combinedSignal) => fetchAndProcess(job.data.url, combinedSignal),
30_000, // 30 second timeout
signal,
);
return result;
},
{ connection },
);

The helper is composable, so you can apply different timeouts to each phase of a job:
const worker = new Worker(
"pipeline-queue",
async (job, token, signal) => {
// Different timeouts for different phases
const downloaded = await withTimeout(
(combinedSignal) => download(job.data.url, combinedSignal),
10_000,
signal,
);
const processed = await withTimeout(
(combinedSignal) => process(downloaded, combinedSignal),
60_000,
signal,
);
const uploaded = await withTimeout(
(combinedSignal) => upload(processed, combinedSignal),
30_000,
signal,
);

return { downloadSize: downloaded.length, outputUrl: uploaded.url };
},
{ connection },
);

Best Practices

Always use the event-based pattern for immediate responsiveness. Polling introduces latency between the cancel request and the processor reacting.

Clean up resources in the abort handler. Close database connections, cancel HTTP requests, delete temporary files, and release anything else the job was using.

Use UnrecoverableError for user-initiated cancellations. If a user explicitly cancelled a job, retrying it automatically would be unexpected behavior.

Use regular Error for system-initiated cancellations. Shutdowns and transient issues should allow the job to be retried by another worker.

Use a single control channel per queue. The convention {prefix}:{queueName}:control with an action field in each message keeps things simple now and extensible later. Any service that knows the prefix and queue name can publish control messages without custom wiring.

Handle cleanup errors gracefully. If cleanup fails, log the error but don't let it mask the original cancellation.

Combine with the lockRenewalFailed event. If a worker loses its Redis lock (due to network issues), cancel the affected jobs so they can be picked up by another worker:

worker.on("lockRenewalFailed", (jobIds) => {
jobIds.forEach((id) => worker.cancelJob(id, "Lock renewal failed"));
});

What's Next
This article focused on Node.js, but job cancellation is equally important in Python and Elixir workers. In upcoming articles, we'll cover:

Cancelling jobs in BullMQ Python, using asyncio.CancelledError and task cancellation
Cancelling jobs in BullMQ Elixir, using OTP process signals and GenServer patterns

The core pattern remains the same across all languages: cooperative cancellation with proper resource cleanup, and Redis Pub/Sub for cross-process signaling.

Ready to try BullMQ? Check out the documentation to get started,
or explore the cancellation guide
for the full API reference.

---

# BullMQ Python vs RQ Performance Benchmark

URL: https://bullmq.io/articles/benchmarks/bullmq-python-vs-rq/
Author: Manuel Astudillo
Published: 2026-02-12
Description: Benchmark comparing BullMQ Python vs RQ (Redis Queue) for job queue performance in Python.
Tags: articles, python, performance, benchmark, rq

Python developers have long relied on RQ (Redis Queue) as the go-to solution
for background job processing with Redis. It's simple, well-documented, and has been around since 2012.
But as Python applications scale, the question arises: is there a faster alternative?
BullMQ Python is a native Python port of the popular
BullMQ library, bringing the same battle-tested architecture that powers millions
of jobs in production Node.js applications. With its async-first design and optimized Lua scripts,
BullMQ Python promises significant performance improvements over traditional synchronous queues.
In this benchmark, we'll put both libraries head-to-head to see how they perform in real-world scenarios.
Architecture Differences
Before diving into the numbers, it's important to understand the fundamental architectural differences
between these two libraries:
RQ (Redis Queue)

Synchronous, blocking architecture
One job processed at a time per worker
Simple and straightforward design
To scale, you run multiple worker processes

BullMQ Python

Asynchronous, non-blocking architecture (built on asyncio)
Configurable concurrency per worker
Lua scripts for atomic Redis operations
Single worker can handle many concurrent jobs

These design choices have real implications. RQ's simplicity is its strength for basic use cases,
but BullMQ's async architecture allows a single worker process to handle multiple jobs concurrently,
significantly improving resource efficiency.
Feature Comparison
Beyond performance, the two libraries differ significantly in available features:
FeatureBullMQ PythonRQAsync supportNative (asyncio)No (synchronous)Concurrency per workerConfigurable (1–1000+)1 (scale via processes)Job prioritiesYes (numeric)Separate queues onlyRate limitingYes (global, per-queue)NoDelayed jobsYes (built-in)Via rq-schedulerRepeatable / cron jobsYes (built-in)Via rq-schedulerRetries with backoffYes (exponential, custom)Basic (fixed count)Parent–child flowsYes (FlowProducer)Basic dependenciesJob progress trackingYesNoGlobal eventsYesNoStalled job recoveryAutomaticManualSandboxed processorsYesVia forking workerDashboard / UITaskforce.sh / Bull Boardrq-dashboardAtomic operationsLua scriptsPython + Redis calls
Benchmark Configuration
I ran these benchmarks on my local machine:

Machine: MacBook Pro with M2 Pro chip, 16GB RAM
Python: 3.13
Redis: 6.4.0 (local Docker instance)
BullMQ Python: 2.19.5
RQ: 2.6.1
Methodology: 5 runs per test, reporting mean values

For RQ, we used SimpleWorker which processes jobs in the same process without forking, providing
the fairest comparison since BullMQ also processes jobs in-process. For processing tests with
simulated I/O work, we also tested RQ with multiple worker processes to match BullMQ's concurrency level.
Benchmark Results
Bulk Job Insertion
Adding 50,000 jobs at once using bulk insertion:
BullMQRQ05.5K11K17K22K21,90018,700Jobs per second - Bulk job insertion (50,000 jobs)
For bulk insertions, BullMQ holds a slight ~17% edge at ~21,900 vs ~18,700 jobs/sec.
Both libraries efficiently batch Redis round-trips, BullMQ uses pipelined Lua scripts while
RQ uses enqueue_many() with its own pipelining, so the results are in the same ballpark.
Single Job Insertion (Concurrent)
Adding 5,000 jobs with concurrent add() calls (concurrency=10):
BullMQRQ02K4K6K8K6,2002,800Jobs per second - Single job insertion with concurrency
Here BullMQ shows a ~2.2x advantage (6,200 vs 2,800 jobs/sec). The async architecture
allows multiple add() calls to be in-flight simultaneously, while RQ's synchronous design
means each insertion blocks until complete. This is a significant win for applications that
need to enqueue jobs from async web frameworks like FastAPI or Starlette.
Processing with Simulated I/O Work
Most real-world jobs involve some I/O: calling APIs, querying databases, reading files.
To simulate this, each job performs a 10ms async sleep (BullMQ) or 10ms thread sleep (RQ).
For a fair scaling comparison, we match BullMQ's concurrency level with an equal number
of RQ worker processes. At concurrency=10:
BullMQ (c=10)RQ (10 workers)0200400600800654471Jobs per second - 10ms I/O work (BullMQ c=10 vs RQ with 10 workers)
With 10 concurrent workers, BullMQ is 39% faster (654 vs 471 jobs/sec). RQ performs
respectably here: 10 OS processes provide real parallelism. But the gap widens dramatically
when we scale to 50:
BullMQ (c=50)RQ (50 workers)08001.6K2.4K3.2K3,1001,200Jobs per second - 10ms I/O work (BullMQ c=50 vs RQ with 50 workers)
At concurrency=50, BullMQ is 2.6x faster (3,100 vs 1,200 jobs/sec). BullMQ scales
near-linearly (654 → 3,100, a 4.7x increase for 5x more concurrency), while RQ shows
sub-linear scaling (471 → 1,200, only 2.5x for 5x more workers). The reason? 50 RQ
processes all compete for the same Redis queue, creating contention. BullMQ handles all 50
concurrent jobs in a single process with zero contention.
Pure Queue Overhead
To isolate the queue machinery cost from the job work itself, we process no-op jobs
(jobs that return immediately without doing any work). This test deliberately uses a
single RQ worker to measure the true per-job overhead floor (note that this is lower
than the I/O test above, which uses 10 parallel RQ worker processes):
BullMQ (c=10)RQ (1 worker)08501.7K2.6K3.4K3,400335Jobs per second - No-op jobs (pure queue overhead, single RQ worker)
BullMQ is ~10x faster for raw job turnover (3,400 vs 335 jobs/sec). This explains
why RQ struggles with high-throughput workloads: each RQ job cycle involves approximately
24 sequential Redis round-trips (dequeue, deserialize, status updates, result storage,
cleanup). At ~0.14ms per Redis round-trip on localhost, that's ~3.3ms of unavoidable per-job
overhead, capping a single worker at ~300 jobs/sec regardless of how lightweight the job is.
BullMQ uses optimized Lua scripts that batch multiple Redis operations into atomic calls,
reducing per-job overhead to a fraction of RQ's.
CPU-Bound Processing
Processing jobs with CPU work (1,000 sin/cos operations per job, matching the Elixir benchmark methodology):
BullMQ (c=10)RQ (1 worker)07501.5K2.3K3K2,900351Jobs per second - CPU-bound processing (1000 sin/cos per job)
BullMQ is ~8x faster for CPU-bound work (2,900 vs 351 jobs/sec). Since both libraries
are single-threaded for CPU work (Python's GIL limits parallelism), the difference comes
entirely from per-job overhead: BullMQ's ~0.3ms vs RQ's ~3ms. When the job work itself
is lightweight (~0.3ms for 1000 sin/cos), BullMQ's lower overhead translates directly
into higher throughput.
Summary
BenchmarkBullMQ PythonRQWinnerBulk Insert21,900/sec18,700/secBullMQ (1.2x)Single Insert6,200/sec2,800/secBullMQ (2.2x)10ms I/O (c=10)654/sec471/sec (10 workers)BullMQ (1.4x)10ms I/O (c=50)3,100/sec1,200/sec (50 workers)BullMQ (2.6x)Pure Overhead3,400/sec335/sec (1 worker)BullMQ (10x)CPU Processing2,900/sec351/secBullMQ (8x)
Understanding Python's Performance Ceiling
Readers familiar with BullMQ's Node.js implementation may notice that the Python version tops out
at ~3,400 jobs/sec for pure overhead, while the Node.js counterpart routinely exceeds 30,000 jobs/sec.
The gap is not a design issue it is a platform-level constraint rooted in the performance of
Python's Redis client.
To pinpoint the bottleneck we profiled every layer of BullMQ Python's job-processing pipeline:
OperationTime per callRedis round-trip (PING)0.22 msLua script eval (simple)0.23 msFull job cycle (c=1)0.51 msJSON encode / decode< 0.002 msasyncio scheduling0.05 ms
The dominant cost is the Redis round-trip itself. BullMQ Python uses
redis-py, the standard async Redis client for Python.
Node.js uses ioredis, which benefits from several
structural advantages:

C++ networking via libuv: ioredis performs socket I/O through Node.js's libuv layer,
written in C/C++. redis-py uses Python's asyncio stream layer, which is implemented in
pure Python.
Native protocol parsing: ioredis can use hiredis (a C library) to parse Redis protocol
responses. redis-py parses them in Python.
Lower event-loop overhead: Node.js's libuv dispatches callbacks roughly 10× faster than
Python's asyncio event loop.

The net effect is that a single Redis round-trip takes ~0.22 ms in Python versus
~0.05–0.08 ms in Node.js: a 3–4× difference. Because every job requires at least one
Redis Lua-script call, this per-call gap directly caps throughput.
Unfortunately, redis-py is the only mature async Redis client available for Python, so this
ceiling is effectively a platform limitation rather than a BullMQ design issue. Despite this
constraint, BullMQ Python still delivers the fastest job processing in the Python ecosystem,
up to 10× faster than RQ and considerably ahead of any other Python queue library we've tested.
The Resource Efficiency Story
The raw numbers tell only part of the story. Consider what these results mean in practice:
Single Process Comparison:

1 BullMQ worker (concurrency=50): ~3,100 jobs/sec (10ms I/O work)
50 RQ worker processes: ~1,200 jobs/sec (same 10ms I/O work)

BullMQ achieves 2.6x higher throughput in a single process compared to 50 RQ processes.
Each RQ process consumes memory independently, needs its own Redis connection, and adds operational
complexity. BullMQ's async architecture eliminates this overhead entirely.
Scaling Behavior:

BullMQ scales near-linearly with concurrency: c=10 → c=50 yields 4.7x throughput
RQ scales sub-linearly with workers: 10 → 50 workers yields only 2.5x throughput

The sub-linear scaling for RQ comes from Redis contention: 50 processes all polling the same
queue simultaneously. BullMQ avoids this by scheduling all work in a single event loop.
For a production workload processing 100,000 jobs per hour (with real I/O work):

BullMQ: 1 worker process
RQ: 25-30 worker processes

This translates directly to infrastructure savings, simpler deployments, and reduced resource consumption.
When to Choose Each
Choose RQ when:

You need simplicity above all else
Your job volume is moderate (less than 1,000 jobs/min)
You prefer synchronous Python code
You're already running multiple worker processes anyway

Choose BullMQ Python when:

You need high throughput from minimal workers
You're using async Python (FastAPI, Starlette, etc.)
You want advanced features (priorities, rate limiting, job dependencies)
Resource efficiency matters for your infrastructure costs

Conclusion
BullMQ Python delivers substantial performance improvements over RQ across all benchmark categories.
The async architecture and optimized Lua scripts provide up to 10x speedups for pure job throughput,
2.6x gains for realistic I/O workloads at scale, 8x for CPU-bound work, and 2.2x for
concurrent single insertions. BullMQ also scales near-linearly with concurrency, while RQ's
multi-process scaling hits diminishing returns from Redis contention.
The key architectural insight: RQ's SimpleWorker performs ~24 sequential Redis round-trips per job,
creating an unavoidable ~3ms overhead floor. BullMQ batches these operations into atomic Lua scripts,
keeping per-job overhead under 0.3ms, a 10x reduction that compounds across every job processed.
For Python applications that need to process high volumes of background jobs efficiently, BullMQ Python
offers a compelling alternative to traditional synchronous queues. The ability to handle thousands of
jobs per second with a single worker process can significantly reduce infrastructure complexity and costs.
The benchmark code is available at bullmq-python-bench
if you'd like to run these tests yourself.

Ready to try BullMQ Python? Check out the documentation
to get started.

---

# BullMQ Elixir vs Oban Performance Benchmark

URL: https://bullmq.io/articles/benchmarks/bullmq-elixir-vs-oban/
Author: Manuel Astudillo
Published: 2026-02-06
Description: Benchmark comparing BullMQ Elixir (Redis) vs Oban (PostgreSQL) for job queue performance.
Tags: articles, elixir, performance, benchmark, oban

With the release of BullMQ for Elixir, we ran some benchmarks against
Oban, the most popular job queue in the Elixir ecosystem. I think that
it is an interesting comparison because they use different backends: BullMQ uses Redis, Oban uses PostgreSQL.
About BullMQ Elixir
While BullMQ Elixir is new to the ecosystem, it's built on years of battle-tested code from the
Node.js BullMQ library. The Elixir implementation itself is just
a thin layer on top of the same Lua scripts that power the Node.js version, scripts that have been
refined over years of production use across thousands of deployments.
The heavy lifting happens in Redis via these Lua scripts. The Elixir code handles connection management,
job serialization, and the worker lifecycle, but the core queue logic is shared with Node.js.
We've also ported most of the comprehensive test suite from Node.js, so despite being young in the Elixir world,
BullMQ Elixir has a solid test coverage.
Keep in mind that these are still the early days for BullMQ in Elixir, and we're actively working
to squeeze more performance out of it. We expect these numbers to improve as we optimize the Elixir-specific
code paths. We're also planning benchmarks for distributed Elixir deployments, running workers across
multiple nodes, which is where things can get even more interesting. Stay tuned.
Under the hood
When you add a job in Oban, it runs a single SQL INSERT (or batch INSERT for bulk operations). The job lands
in an oban_jobs table. Simple and straightforward.
BullMQ does more per job. Each insert runs a Lua script that atomically roughly does the following:

Generates a unique job ID
Stores job data in a Redis hash
Adds the job to the right data structure (LIST for waiting, ZSET for delayed/prioritized)
Checks for duplicates
Handles parent-child dependencies if you're using flows
Publishes events for real-time subscribers
Updates rate limiting state if configured

Feature Comparison
FeatureBullMQObanDelayed Jobs✅✅Priority Queues✅✅Deduplication✅✅Rate Limiting✅💰 ProFlows/Dependencies✅💰 ProReal-time Events✅❌Cron/Schedulers✅✅BackendRedisPostgreSQLCross-language✅ Node.js, Python, PHP, ElixirElixir, Python
Rate limiting and job flows are included in BullMQ's open-source version. Oban requires the paid Pro tier for these.
Benchmark Environment

MacBook Pro M2 Pro, 16GB RAM
Redis 7.x (Docker, localhost) with AOF enabled (appendfsync everysec)
PostgreSQL 16 (standalone, localhost) with connection pool of 150
Elixir 1.18 / OTP 27
BullMQ 1.2.6, Oban 2.20.3

All tests were run 5 times, and we report the mean result.
Tests used 50,000 jobs to ensure statistically meaningful run times (several seconds per test).
Results
Single Job Insertion
BullMQOban01.5K3K4.5K6K5,8002,900Jobs per second - Single job insertion (50,000 jobs)
For one-at-a-time inserts, BullMQ achieves ~5,800 jobs/sec vs Oban's ~2,900 jobs/sec—about 100% faster.
This matters in scenarios where you're adding jobs from request handlers or event callbacks, where each
request enqueues one job.
Concurrent Single Job Insertion
BullMQOban04.5K9K14K18K17,70011,200Jobs per second - Concurrent single job insertion (50,000 jobs, 10 concurrent inserters)
When 10 processes insert jobs simultaneously (simulating multiple request handlers):

BullMQ: ~17,700 jobs/sec
Oban: ~11,200 jobs/sec

BullMQ is ~57% faster. Both scale well with concurrency—BullMQ from 5.8K to 17.7K (3.1x), Oban from 2.9K to 11.2K (3.9x).
Bulk Job Insertion
BullMQOban015K30K45K60K51,40036,800Jobs per second - Bulk insert 50,000 jobs (batches of 1,000)
For sequential bulk inserts with 1,000-job batches, BullMQ wins: 51.4K jobs/sec vs 36.8K jobs/sec (+40%).
Concurrent Bulk Job Insertion
BullMQOban025K50K75K100K63,40089,600Jobs per second - Concurrent bulk insert (50,000 jobs, batches of 1,000, 10 concurrent inserters)
Here's where Oban shines. When 10 processes do bulk inserts simultaneously:

BullMQ: ~63.4K jobs/sec
Oban: ~89.6K jobs/sec

Oban is 41% faster at concurrent bulk inserts. PostgreSQL's connection pool efficiently parallelizes
multiple bulk INSERT statements, each running in its own transaction. Redis pipelines, while fast,
still serialize through a single-threaded event loop.
Batch size still matters:
Batch SizeBullMQObanWinner10046.3K57.4KOban (+24%)25054.4K63.7KOban (+17%)50058.6K57.8KTie100057.0K51.0KBullMQ (+12%)200057.3K40.4KBullMQ (+42%)
Smaller batches favor Oban due to PostgreSQL's efficient transaction handling. Larger batches favor BullMQ
because PostgreSQL's overhead (WAL writes, index updates) compounds while Redis stays consistent.
Note: PostgreSQL has a hard limit of 65,535 parameters per query, which caps Oban's batch size to ~7,000 jobs
(depending on column count).
Job Processing (10ms work)
BullMQOban02.5K5K7.5K10K8,3004,400Jobs per second - Processing with 10ms simulated work (1 worker, concurrency=100)
Each test uses a single BullMQ Worker (or a single Oban queue) with a matching connection pool size.
We tested two concurrency levels to show how throughput scales:
ConcurrencyBullMQObanDifference10911 jobs/sec523 jobs/secBullMQ +74%1008,300 jobs/sec4,400 jobs/secBullMQ +88%
BullMQ scales from 911 to 8,300 jobs/sec (9.1x) when going from 10 to 100 concurrent processors;
Oban scales from 523 to 4,400 (8.4x). Both scale well, but BullMQ maintains its lead at every level.
To put the 100-concurrency numbers in context: with 10ms of work per job, the theoretical maximum is
10,000 jobs/sec (100 × 100 jobs/sec). BullMQ achieves 83% of theoretical max; Oban achieves 44%.
The difference is queue overhead — time spent fetching jobs, updating state, and marking completion.
BullMQ's lower overhead means more time doing actual work.
CPU-Bound Processing
BullMQOban06.5K13K20K26K24,3006,800Jobs per second - CPU-bound work with 1000 sin/cos operations per job (1 worker, concurrency=100)
To measure throughput with lightweight but real CPU work, each job performs 1,000 sin/cos calculations
(~1ms of CPU time). This uses 1 worker with a matching connection pool:
ConcurrencyBullMQObanDifference1012,400 jobs/sec1,200 jobs/secBullMQ +944%10024,300 jobs/sec6,800 jobs/secBullMQ +257%
The gap is dramatic — especially at concurrency 10, where BullMQ is nearly 10x faster. Since each job
only takes ~1ms of CPU time, the bottleneck is almost entirely queue overhead: how fast the system can
dequeue, track, and mark jobs complete. This is where BullMQ's Redis pipelines and atomic Lua scripts shine.
BullMQ scales from 12.4K to 24.3K (2x) going from 10 to 100 concurrent processors, peaking at 25.8K jobs/sec.
Oban scales from 1.2K to 6.8K (5.7x) — a steeper curve that suggests its per-job overhead dominates at lower concurrency.
These numbers scale with concurrency and hardware. On a 12-core machine like the M2 Pro used here,
the BEAM VM schedules lightweight processes across all available cores automatically, so adding more
concurrent processors translates directly into higher throughput — until Redis or network I/O becomes the bottleneck.
Pure Queue Overhead
BullMQOban06.5K13K20K26K25,6007,100Jobs per second - Minimal work per job, measuring raw queue overhead (1 worker, concurrency=100)
To isolate the queue machinery itself, we ran a test where each job does essentially nothing —
just enqueue, dequeue, and mark complete. This measures the raw overhead of the queue system:
ConcurrencyBullMQObanDifference1014,600 jobs/sec1,200 jobs/secBullMQ +1106%10025,600 jobs/sec7,100 jobs/secBullMQ +262%
With no job work to amortize, this test exposes the full cost of each queue round-trip. BullMQ peaks at
27.2K jobs/sec — the ceiling for single-worker throughput on this hardware. At concurrency 10,
BullMQ is over 12x faster than Oban, showing how Redis's in-memory operations and pipelined commands
minimize per-job overhead compared to PostgreSQL's disk-based query cycle.
Poll-Free Architecture
Both BullMQ and Oban use blocking commands to wait for jobs, but the implementations differ significantly.
How BullMQ fetches jobs:
BullMQ uses a marker-based system with BZPOPMIN. Instead of blocking directly on job lists, workers block
on a separate "marker" ZSET. When jobs are added, a marker with a timestamp is pushed to this ZSET:

When jobs are available: BZPOPMIN returns immediately with a marker → worker runs "moveToActive" Lua script → fetches the next job based on priority, delay, rate limiting, etc.
When idle: BZPOPMIN blocks until a marker arrives or timeout.

This unified mechanism handles standard jobs, priority jobs, delayed jobs, and rate-limited jobs through
the same code path. The timestamp on the marker tells workers when the next job should be processed—for
immediate jobs it's 0, for delayed jobs it's the scheduled time.
The Elixir architectural win:
In Node.js BullMQ, each worker process maintains its own Redis connection for blocking operations. With 100 idle workers:

100 Node.js workers idle = 100 Redis connections blocking on BZPOPMIN

In Elixir BullMQ, a single coordinator process manages all concurrent job processors:

1 blocking connection for BZPOPMIN waits (shared by all job processors)
Shared connection pool for job operations (moveToActive, complete, etc.)

When idle, that's 100x fewer blocking connections vs Node.js. During active processing, workers share
a configurable connection pool rather than each holding dedicated connections. This architecture
scales efficiently—at thousands of workers across many queues, the connection savings are substantial.
BEAM Advantage: Effortless Horizontal Scaling
The BEAM VM was designed for distributed systems. Elixir nodes can connect to each other and communicate
seamlessly—this is what powers Phoenix's real-time features across clusters.
For BullMQ Elixir, scaling is straightforward:

Spin up more BEAM nodes - each runs its own BullMQ worker coordinator
Point them at the same Redis - that's it
Redis handles work distribution - jobs are automatically distributed across all workers

No load balancers, no orchestration layer, no coordination overhead. Each node's workers compete fairly
for jobs through Redis's atomic operations. If you need more throughput you can just add another node,
for example using Kubernetes or similar orchestrators, by increasing the replica count to increase
your processing capacity linearly.
If your load is large enough you will end up saturating Redis, which can also be updated to
larger instances, or you could start dividing jobs into smaller queues and take advantage of
Redis clustering.
Why does BullMQ win at processing?
BullMQ's architecture minimizes round-trips. When a worker fetches a job, processes it, and marks it complete,
BullMQ batches these operations into efficient Redis pipelines. The Lua scripts that run in Redis are
executed atomically with minimal overhead.
PostgreSQL, by contrast, requires more round-trips: fetch job, update state to "running", complete job
with state update. Each operation is a separate query, and PostgreSQL's MVCC and WAL machinery add
overhead that Redis avoids by keeping everything in memory.
Why does Oban win at concurrent bulk insert?
PostgreSQL's bulk INSERT is remarkably efficient. A single INSERT INTO ... VALUES (row1), (row2), ...
statement handles thousands of rows with:

One transaction
One WAL write
Batched index updates

When multiple processes do bulk inserts simultaneously, PostgreSQL's connection pool runs each INSERT
in parallel across separate connections. Each transaction runs independently, and PostgreSQL can
parallelize the work across CPU cores.
Redis, by contrast, is single-threaded. BullMQ's Redis pipelines batch commands to minimize network
round-trips, but Redis still processes each job's Lua script sequentially—even when multiple clients
send pipelines simultaneously. The pipelines queue up and execute one at a time.
For sequential bulk inserts, BullMQ's low per-job overhead wins. For concurrent bulk inserts, PostgreSQL's
parallelism wins.
Durability trade-offs
Oban gives you PostgreSQL's ACID guarantees out of the box. Jobs are on disk, transactions are atomic,
and you get all the durability you'd expect.
BullMQ's durability depends on how you configure Redis. With AOF persistence and appendfsync always,
you get similar guarantees but at a slight performance cost. Most deployments use appendfsync everysec as
a reasonable middle ground—which is what we used in these benchmarks.
If losing a second of jobs during a Redis crash is unacceptable, Oban is the safer choice.
If you need the throughput and can tolerate that risk (or have Redis replication), BullMQ makes sense.
Conclusions
The benchmarks show both libraries have strengths in different scenarios:
BullMQ excels at:

Single job insertion (~100% faster, ~57% faster with 10 concurrent inserters)
Job processing throughput (~88% faster for I/O work, up to ~257% faster for CPU-bound jobs)
Pure queue overhead (up to 12x faster at low concurrency, ~3.6x faster at high concurrency)
Sequential bulk inserts (40% faster at 1000-job batches)

Oban excels at:

Concurrent bulk inserts (41% faster with 10 concurrent inserters)
Small batch inserts (up to ~500 jobs per batch)

The concurrent bulk insert result is notable: when multiple processes bulk-insert simultaneously,
PostgreSQL's connection pool parallelizes these operations efficiently, while Redis's single-threaded
event loop serializes them.
BullMQ makes sense when:

Processing throughput matters. Up to ~257% faster processing means lower latency and better resource utilization.
You want features without paying extra. Rate limiting, job flows, and real-time events are all open source.
You're already running Redis. No new infrastructure to manage.
Your stack spans multiple languages. The same queues work across Node.js, Python, PHP, and Elixir.
Add jobs from your Python ML service, and process them in Elixir.
You value battle-tested code. BullMQ Elixir runs the same Lua scripts that power thousands of Node.js deployments. Years of edge cases already handled.

Oban makes sense when:

You want everything in PostgreSQL. One database, one backup strategy, one ops story.
Strict durability is non-negotiable. ACID guarantees out of the box with no configuration.
You're in the Elixir/Python ecosystem. Oban now supports both, with tight Ecto and Phoenix integration for Elixir.
You do concurrent bulk inserts. Multiple processes bulk-inserting jobs in parallel is where Oban shines.

Both are solid choices. Oban has excellent documentation and a strong community.
BullMQ brings faster processing, cross-platform reach, and a decade of production hardening.

Methodology Notes
To ensure fair comparison:

PostgreSQL pool_size was set to 150 connections to match concurrency needs (earlier versions of this benchmark used pool_size=20, which unfairly bottlenecked Oban)
50,000 jobs per test ensures tests run long enough (several seconds) for meaningful measurements
5 runs per test, mean result reported
Same machine for all tests, with Redis and PostgreSQL both running locally
Default configurations for both libraries where possible

The benchmark code is open source and we welcome feedback and improvements.

Benchmark code is available at GitHub

---

# BunJS vs NodeJS Benchmark for BullMQ

URL: https://bullmq.io/articles/benchmarks/bunjs-vs-nodejs/
Author: Manuel Astudillo
Published: 2026-01-30
Description: Benchmark comparing BullMQ performance on BunJS vs NodeJS runtimes.
Tags: articles, bun, performance, benchmark

Yesterday we managed to run the complete BullMQ testsuite on BunJS, and is now part of our continuous
integration process to ensure compatibility with this new runtime. This is our way to ensure that it is
supported and officially compatible with BullMQ.
BunJS has experienced a lot of traction the last couple of years and has now reached
a level of maturity comparable to NodeJS for many use cases, but with a focus on performance and
developer experience.
In my early experiments with BunJS around a year ago, I found that while BunJS could run BullMQ quite reliably,
it was not really that much faster than NodeJS, at least for synthetic benchmarks. Today I wanted to
perform new benchmarks to see how things stand now that BunJS has matured, and compare it with the latest
NodeJS version (v24.13.0 at the time of writing).
I am running the benchmarks on my local machine. This is not an ideal benchmark environment, but
hopefully it can give us some idea of the performance differences between the two runtimes.
My machine is a MacBook Pro with M2 Pro chip, 16GB RAM, running macOS Ventura 13.4.1, with a local Redis server running
with default configuration on Docker.
I am not doing any fancy tests, just some of the most standard operations in BullMQ. You can find the repo with the benchmark code and
instructions to run it yourself at bullmq-bun-bench.
In real world scenarios we usually have a dedicated fleet of workers so in this benchmark we are just
trying to ascertain what overhead results from the queue management itself, whereas the actual heavy duty of
processing the job is what at the end of the day is going to limit how many jobs you can process per second.
In this regard BunJS could be much faster depending on the type of workload.
Benchmark Configuration

Jobs per test: 50,000
Worker concurrency: 100
Redis: Local Docker instance
Results: Best of 5 runs

Benchmark Results (50,000 Jobs)
Job Addition (Queue.add)
Adding 50,000 jobs individually using Queue.add():
Node.js v24Bun 1.3.3020K40K60K80K54,11367,751Jobs per second - Individual job addition (1,000 parallel batches)
Bun is about 25% faster than Node.js for individual job additions in this test. Both runtimes benefit from ioredis's auto-pipelining
which batches concurrent Redis commands. We run 1,000 parallel add() calls at a time to simulate realistic high-throughput scenarios.
Bulk Job Addition (Queue.addBulk)
Adding 50,000 jobs at once using Queue.addBulk():
Node.js v24Bun 1.3.3015K30K45K60K45,29057,013Jobs per second - Bulk job addition
For bulk operations, Bun is 26% faster. The addBulk method explicitly batches Redis commands into pipelines, making
it highly efficient.
Interestingly, individual add() calls with auto-pipelining can match or slightly exceed addBulk performance when
there's enough concurrency.
Job Processing
Processing 50,000 jobs with a worker at concurrency 100:
Node.js v24Bun 1.3.308.5K17K26K34K30,10233,670Jobs per second - Job processing (concurrency=100)
For pure job processing, Bun is about 12% faster than Node.js. Job processing is heavily I/O bound (Redis round-trips), so
the runtime's JavaScript performance matters less here.
Job Processing with CPU Work
Processing 50,000 jobs that include CPU-intensive work (recursive Fibonacci calculation):
Node.js v24Bun 1.3.304K8K12K16K11,11614,810Jobs per second - Processing with CPU work
When jobs include actual CPU work (recursive Fibonacci calculation), Bun is 33% faster. This suggests
that Bun's JavaScriptCore engine performs better than Node's V8 for this type of CPU-bound workload.
Flow Producer
Creating 16,666 flows, each with a parent job and 2 children (49,998 total jobs):
Node.js v24Bun 1.3.309.5K19K29K38K26,16336,022Jobs per second - Flow producer (parent + 2 children)
Flow creation is 38% faster with Bun. Creating flows involves multiple Redis operations per flow, so this
benchmark tests both JavaScript execution and Redis client performance.
Summary (50,000 Jobs)
BenchmarkNode.js v24Bun 1.3.3DifferenceJob Addition54,113/sec67,751/sec+25%Bulk Addition45,290/sec57,013/sec+26%Job Processing30,102/sec33,670/sec+12%CPU Work11,116/sec14,810/sec+33%Flow Producer26,163/sec36,022/sec+38%
Scaling to 100,000 Jobs
To see how performance holds up at larger scale, we also ran benchmarks with 100,000 jobs:
Job Addition (100K)
Node.js v24Bun 1.3.3020K40K60K80K36,76567,385Jobs per second - Individual job addition (100K jobs)
At 100K jobs, Bun's advantage grows to 83% for individual job additions. Node.js performance dropped noticeably at this scale.
Bulk Addition (100K)
Node.js v24Bun 1.3.3015K30K45K60K44,15055,866Jobs per second - Bulk job addition (100K jobs)
Bulk addition shows a 27% advantage for Bun, similar to the 50K results.
Job Processing (100K)
Node.js v24Bun 1.3.309K18K27K36K34,17634,364Jobs per second - Job processing (100K jobs)
At 100K jobs, both runtimes perform nearly identically for job processing (~0.5% difference).
CPU Work (100K)
Node.js v24Bun 1.3.304K8K12K16K11,31914,817Jobs per second - Processing with CPU work (100K jobs)
CPU-intensive work shows a 31% advantage for Bun, consistent with the 50K results.
Flow Producer (100K)
Node.js v24Bun 1.3.309K18K27K36K23,88934,340Jobs per second - Flow producer (100K jobs)
Flow creation is 44% faster with Bun at larger scale.
Summary (100,000 Jobs)
BenchmarkNode.js v24Bun 1.3.3DifferenceJob Addition36,765/sec67,385/sec+83%Bulk Addition44,150/sec55,866/sec+27%Job Processing34,176/sec34,364/sec~sameCPU Work11,319/sec14,817/sec+31%Flow Producer23,889/sec34,340/sec+44%
Understanding the 100K Results
The 100K benchmark reveals some interesting patterns worth discussing.
Node.js Job Addition Slowdown:
The most striking difference is Node.js dropping from 54K to 37K jobs/sec for individual job additions,
while Bun maintains a consistent ~67K jobs/sec at both scales. I am not sure but I think
this could be attributed to V8's garbage collection behavior, with 100K concurrent promises and job objects,
maybe GC pauses become more frequent and impactful. Bun's JavaScriptCore engine on the other hand appears
to handle memory pressure more gracefully in this scenario.
Job Processing Convergence:
Interestingly, job processing performance converges at 100K jobs
(from 12% difference to essentially identical).
This reinforces that job processing is fundamentally I/O bound, the longer the benchmark runs, the more Redis round-trip latency dominates,
masking any runtime differences.
Consistent CPU Performance:
The CPU-intensive workload shows consistent ~30% advantage for Bun at both scales, suggesting this
difference is inherent to the JavaScript engine performance rather than memory management.
Conclusions
In these benchmarks, Bun was faster than Node.js across most operations, with differences ranging from nearly identical
up to 83% depending on the workload and scale.
Key findings:

Job Addition shows the largest variance. Bun maintains consistent performance at both scales while Node.js
slows down significantly at 100K jobs (+25% at 50K, +83% at 100K)
CPU-intensive work consistently favors Bun by ~30-33% at both scales
Job Processing is nearly identical at larger scale, confirming it's I/O bound
Flow Producer shows 38-44% improvement with Bun

Just remember that these are synthetic benchmarks running on a local machine with Redis on Docker, so expect differences
in a production environment where Redis network latency may play a bigger role and where most likely you will have
dedicated workers. In any case, it seems you can get a bit more juice out of your machines when using BunJS as it stands
right now, specially considering that BunJS is probably also faster at doing the actual work in the workers, not just
the queue handling which is basically what we are testing here.
NodeJS has proved to be a stable and quite fast runtime all these years, but now with BunJS being more mature,
BullMQ users now have another runtime option, which is always a good thing in my opinion.
If you're already using Node.js and it works well for you, there's no pressing need to switch. But if you're starting
a new project or curious about Bun, you can expect comparable or slightly better performance for BullMQ workloads.
If you try BullMQ with Bun, we'd be happy to hear about your experience!

---

# Top Redis™ Alternatives for 2025

URL: https://bullmq.io/articles/redis/top-redis-alternatives-2025/
Author: Manuel Astudillo
Published: 2025-02-05
Description: How to Choose the Best Redis™ Replacement for Your Application
Tags: articles, redis, message queues, valkey, upstash, dragonflydb, elasticache, memorydb

Redis™ was released by Salvatore Sanfilippo in 2009. From its inception, it was a fully open-source product that rapidly gained popularity by providing a fast, robust in-memory database that was both easy to use and simple to manage.
Today, Redis™ is ubiquitous in web applications—it’s used as a caching layer to accelerate performance, as an ultra-fast in-memory database, as a reliable pub/sub system, as a session manager, and even as a message queue.
In March 2024, however, Redis™ switched its license from open source to a source-available model. This move left many longtime users wondering whether they should continue using the “good old” Redis™ or migrate to a different vendor.
Fortunately, Redis™ drop-in replacements have been available for over a decade, making a switch with minimal code changes entirely possible. In recent years, the number of alternatives has grown significantly, and each vendor now offers unique features that might make their solution more suitable for certain use cases.
In this post, I provide an overview of well-known Redis™ alternatives to help you choose the one that best fits your use case and requirements. I’ve strived to keep this review as objective as possible so you can make an informed decision for your project. If you notice any omissions or inaccuracies, please email me and I’ll update the article accordingly. While most vendors claim that their solution is the best, I aim to offer a neutral perspective based on my own experience and expertise.
Redis™

Redis™ is the original implementation, ensuring 100% compatibility with its documented behavior and features. Over the years, Redis™ has proven to be extremely stable and has built a massive community. Whether you need help from community members or support from Redis™ Labs, assistance is never far away.
However, Redis™’s conservative development approach means it has not fully embraced some of the latest trends in hardware and software design. For example, being single-threaded, Redis™ can utilize only one CPU core—limiting its ability to leverage modern multi-core architectures.
If stability and reliability are your primary concerns and you don’t require bleeding-edge features or maximum performance, Redis™ remains an excellent choice.
KeyDB

KeyDB was launched in 2019 and is often cited as the first Redis™ drop-in replacement. This fork of Redis™ has been optimized for multi-threading and includes several performance enhancements over the original. While early versions claimed 100% Redis™ compatibility, some incompatibilities existed initially—but many have been addressed over time.
In my experience, KeyDB is less stable than Redis™ and isn’t maintained as actively. Its smaller community means that issues might take longer to resolve—or, in some cases, you may need to implement fixes yourself.
Update: On January 1, 2025, KeyDB’s main maintainer, John Sully, announced that he would be leaving the project announcement.
Valkey

Valkey

Valkey was created shortly after Redis™ transitioned to a source-available license. As a fork of Redis™ (originating from version 7.2.4), Valkey was 100% compatible from the start. In less than a year, Valkey released a major update featuring unique enhancements such as RDMA support, improved multi-core utilization, and enhanced I/O multithreading- benchmarks suggest throughput increases of over three times compared to its previous version.
Valkey is a promising project that has been actively maintained and continues to evolve. Although its cutting-edge features like RDMA may introduce some stability trade-offs compared to Redis™, my experience so far has been very positive. If you value an open-source, Redis™-compatible service and want to leverage the latest hardware and software advancements, Valkey is a strong contender.
DragonflyDB

DragonflyDB is a Redis™-compatible service developed by a commercial company. Unlike Redis™—which is written in C—DragonflyDB is built from scratch in C++, a significant architectural shift. DragonflyDB claims full Redis™ compatibility, and in my experience, this holds true. The development team is highly active, with a clear focus on maximizing performance and stability.
One of DragonflyDB’s key advantages is its advanced multithreading capabilities, enabling it to harness multi-core CPUs for significant performance gains. It also offers unique features like improved memory utilization (up to 60% better) and a snapshotting mechanism that can be up to 30 times faster than Redis™.
At the time of writing, DragonflyDB lacks support for Append-Only Files (AOF). This means you must rely on RDB snapshotting for persistence—a method that, while reliable, may not provide the same level of data safety as AOF in the event of a crash. In my opinion, deploying DragonflyDB in a high-availability setup with replication is the best way to mitigate this risk.
Regarding licensing, DragonflyDB is offered under a source-available license similar to recent Redis™ releases. This license allows free usage but restricts cloud providers from offering DragonflyDB as a managed service. In other words, you can deploy it on your own infrastructure—but if you want a fully managed service, you must use DragonflyDB’s cloud offering.
If you’re looking for a Redis™ drop-in replacement that prioritizes performance and stability—and you’re comfortable with a source-available license and the current lack of AOF support—DragonflyDB is an excellent option.
Upstash

Upstash is a commercial, Redis™-compatible service offered exclusively as a fully managed cloud solution. Unlike some alternatives, you cannot deploy Upstash on your own infrastructure; you must use their cloud service.
One of Upstash’s primary advantages is its serverless architecture, which automatically scales based on your application’s needs. For instance, you don’t have to worry about memory constraints—Upstash dynamically adjusts memory allocation as usage increases.
Upstash is an attractive option if you prefer a fully managed service without the overhead of infrastructure management, and if automatic scaling is a priority. However, its pricing model—charging per Redis™ command executed rather than a fixed monthly fee—can make it more expensive, particularly for high-throughput applications. In my experience, while Upstash is stable and reliable, it may not deliver the raw performance of some other alternatives. If performance isn’t your primary concern and you value a hassle-free, automatically scaling service, Upstash is a solid choice.
Amazon Web Services (AWS)

Amazon has significantly expanded its Redis™-compatible offerings in recent years, now providing a diverse array of services with varying features and pricing. Let’s review each of these options.
ElastiCache for Redis™
Amazon’s ElastiCache for Redis™ is a fully managed Redis™ service with a long track record of stability and reliability. Its key benefits include high availability, automatic failover, and reduced infrastructure management.
While performance is generally good, ElastiCache may not match the speed of some newer alternatives optimized for modern hardware. Its pricing is competitive—costs are based on the memory allocated to your Redis™ instances, allowing for flexible scaling. One important consideration is that ElastiCache is accessible only from within the Amazon network. If you require external access, you might need to set up a VPN or AWS Direct Connect, which can complicate the configuration and add extra costs.
ElastiCache Valkey
Due to licensing changes that prevented Amazon from continuing with the latest original Redis™ releases, they decided to support the Valkey project. This Valkey-based version of ElastiCache is fully compatible with Redis™ and offers the same features—such as high availability, automatic failover, and clustering—as the original ElastiCache for Redis™, with the added advantage of leveraging the latest enhancements found in Valkey.
ElastiCache Serverless
The latest addition to the ElastiCache family is the Serverless Redis™ service. Similar to Upstash, this service is serverless, offers 99.99% uptime, and scales automatically based on your application’s demands. However, unlike Upstash, ElastiCache Serverless is only accessible within the Amazon network, which may limit external access. The pricing model is straightforward: you pay for the total memory used along with the number of ElastiCache Processing Units (ECPUs). Since understanding the practical implications of ECPUs for your application can be challenging, it’s advisable to conduct tests to accurately gauge costs for your specific use case. Notably, this serverless option is available for both open-source Redis™ and Valkey, incorporating the latest features of the Valkey project.
MemoryDB
MemoryDB, released by Amazon in 2021, is designed with a strong emphasis on data durability and availability, making it one of the safest Redis™-compatible options available. It is offered solely as a fully managed service and is intended for deployment on large EC2 instances—generally making it one of the more expensive alternatives.
MemoryDB is an excellent choice if data durability, high performance, and high availability are paramount and you’re willing to invest in a premium solution. Apart from its enhanced guarantees, MemoryDB shares many similarities with other ElastiCache services and, like them, is accessible only from within the Amazon network.
Other Cloud Providers
While AWS dominates the cloud market, other providers offer robust Redis™-compatible solutions:

Azure Cache for Redis: Fully managed service with enterprise-grade security and Redis™ 7.0 compatibility.
Google Cloud Memorystore: Supports Redis™ clustering and seamless integration with GCP services.

Feature Comparison

Solution
License
Managed Option
Multithreading
Unique Features
Best For

Redis™
Source-available
Yes (Redis Ltd)
No
Proven stability, large community
Legacy systems, simplicity

KeyDB
BSD-3
No
Yes
Multi-threading, performance improvements
Performance-critical workloads

Valkey
BSD-3
Yes (AWS)
Yes
RDMA support, active development
High-throughput, open-source fans

DragonflyDB
BSL 1.1
Yes (Vendor)
Yes
Memory efficiency, horizontal scaling
Performance-critical workloads

Upstash
Proprietary
Yes (vendor)
No
Serverless, pay-per-request
Ephemeral workloads, auto-scaling

AWS ElastiCache
Proprietary
Yes
Varies
Deep AWS integration, serverless option
AWS-centric teams

AWS MemoryDB
Proprietary
Yes
No
Enhanced data durability, high performance
High-availability, data safety

Conclusion
Choosing the right Redis™ replacement ultimately depends on your application’s requirements, budget, and your preference between managing your own infrastructure versus using a fully managed service. Here’s a quick recap:

Redis™ remains a rock-solid, community-backed option if stability is your top priority and you don’t need the latest performance optimizations.
KeyDB offers multi-threading and performance improvements but may lag behind in stability and community support.
Valkey provides modern features like RDMA and enhanced multi-core utilization, making it ideal for those who want to leverage the latest hardware advancements, and want to stick with an open-source solution.
DragonflyDB excels in performance and memory utilization with advanced multithreading—if you can work within its current licensing model and persistence limitations.
Upstash is perfect for those who prefer a fully managed, serverless solution, despite its potentially higher cost per command.
Amazon’s offerings (ElastiCache for Redis™, ElastiCache Valkey, ElastiCache Serverless, and MemoryDB) cater to a wide range of needs, from cost efficiency and scalability to enhanced data durability and performance, albeit with the caveat of network accessibility constraints.

The landscape of Redis™-compatible solutions is evolving rapidly. What might be the best choice today could shift as projects mature and new features emerge. It’s important to stay informed, run performance tests, and consider your specific use case before making a final decision.
References

Redis™ Wikipedia
Redis™ Labs
KeyDB
Valkey
DragonflyDB
Upstash
Amazon ElastiCache
Amazon MemoryDB

About the Author
Manuel Astudillo is a software engineer and entrepreneur. He is the founder of Taskforce.sh Inc., a company that provides consulting services for software development and cloud infrastructure. He is also the creator of BullMQ, a Node.js library for handling distributed job queues. You can find him on GitHub and Twitter.

---