Retries, durable waits, and idempotency keys

Picture a task that exports an organization’s invoices. It runs for about ten minutes, paging through the database five hundred rows at a time, and emails a download link when it finishes. In production, plenty goes wrong. At minute eight, the platform redeploys and recycles the worker out from under the running job. Or the box runs out of memory and gets killed. Or the third invoice page hits a downstream service that returns 429 Too Many Requests because you’ve been hammering it.

Some of this Trigger.dev handles for free; the rest is on you. The line between them turns on one idea: durability lives in the seams between steps, not inside them. Once that clicks, retries, idempotency keys, and durable waits become one model with three knobs instead of three APIs to memorize.

By the end you can write a multi-step task that survives every failure above. It resumes from where it died instead of starting over, and it never sends the “your export is ready” email twice or re-charges a customer because a retry replayed a side effect. The previous lesson covered defining, typing, triggering, and queuing a task; this one makes a defined task crash-proof.

Durability lives between steps, not inside them

A durable run survives the worker dying: redeploy mid-run, OOM-kill the box, or let the platform recycle the machine, and the run picks back up and finishes.

The runtime makes that work by serializing your run’s state, where it is and what it has produced so far, at three moments: every await wait.* call, every await *.triggerAndWait call, and the end of every attempt. Each snapshot is a checkpoint . When the worker dies, the runtime starts a new one, rehydrates it from the last checkpoint, and continues.

Checkpoints sit between operations, never inside one, so a single stretch of work is not snapshotted line by line. Suppose your task body is one nine-minute synchronous loop with no await wait.* or triggerAndWait inside it. If the worker dies at minute five, the last checkpoint was the start, so the whole loop runs again from zero. That task runs on a durable platform but is not durable.

So split long work into small steps separated by a wait or a triggerAndWait. Each completed piece becomes a save point you never re-cross: crash on page seven, and pages one through six stay done. Durability follows from where you place the seams, and placing them is your job.

Worker · running

Step A — write page 1

executing

⚐ checkpoint after page 1

Step B — write page 2

waiting

⚐ checkpoint after page 2

The run starts on a worker. Step A, writing page 1, is executing. The checkpoint after it is dim, because that boundary hasn't been crossed yet.

Worker · running

Step A — write page 1

done

🚩 checkpoint after page 1

Step B — write page 2

waiting

⚐ checkpoint after page 2

Step A finishes and the runtime writes a checkpoint. Page 1 is now durably done.

Worker · running

Step A — write page 1

done

🚩 checkpoint after page 1

Step B — write page 2

executing

⚐ checkpoint after page 2

The run moves past the checkpoint into Step B, writing page 2.

Worker · dead ⚡

Step A — write page 1

done

🚩 checkpoint after page 1

Step B — write page 2

aborted

⚐ checkpoint after page 2

The worker dies mid-Step-B. Step B is aborted, but checkpoint A is still lit.

New worker · running new

Step A — write page 1

skipped — cached

🚩 checkpoint after page 1

Step B — write page 2

executing

⚐ checkpoint after page 2

A new worker rehydrates from checkpoint A. Step A is not re-run; it returns its cached result, greyed out. Step B re-executes from the top.

Worker · running

Step A — write page 1

done

🚩 checkpoint after page 1

Step B — write page 2

done

🚩 checkpoint after page 2

Step B completes, a checkpoint is written, and the run finishes. The step in flight at crash time is the only one that repeats.

Prev

Next

Retries are the runtime’s job, declared not coded

A transient blip, an overloaded service or a reset database connection, kills a run that would have succeeded on a second try. You don’t write that “try again” logic by hand: the runtime owns it, and you configure it once on the task.

src/trigger/export-invoices.ts

export const exportInvoices = schemaTask({
  id: 'export-invoices',
  retry: {
    maxAttempts: 5,
    factor: 1.8,
    minTimeoutInMs: 1_000,
    maxTimeoutInMs: 60_000,
    randomize: true,
  },
  run: async (payload, { ctx }) => {
    // ...export the invoices...
  },
});

Each field shapes the back-off curve. maxAttempts counts every try including the first, so 5 is the original run plus four retries. factor is the exponential multiplier between waits: at 1.8, each retry waits roughly 1.8× as long as the last. minTimeoutInMs floors the first wait and maxTimeoutInMs caps every wait; past the cap, retries keep firing but stop spreading further apart. randomize: true adds jitter.

Jitter is why exponential backoff with jitter is non-negotiable. When a downstream API falls over, a thousand of your runs can fail at the same instant. Without jitter they all compute the identical back-off and retry in lockstep, a thundering herd that knocks the service back down the moment it recovers. Jitter scatters the retries across a window so the service sees a trickle. Leave it on.

Watch video

Arpit Bhayani walks the thundering-herd failure end to end (~11 min) and shows why backoff alone isn’t enough without jitter.

Note

Set retries project-wide in trigger.config.ts, and add a per-task retry block only when a task needs a different curve.

A throw inside your task triggers a retry on the configured back-off, for free, so don’t wrap an external call in your own try/catch retry loop. A second hand-rolled layer multiplies with the runtime’s, turning “five attempts” into twenty-five and dissolving your back-off curve. Let it throw. If the body needs to know which attempt it’s on, ctx.attempt.number tells you, but reaching for it should be rare.

Retry on transients, abort on permanents

Every throw retries, which is the right default. But some failures will never succeed, and retrying them only burns the full run of attempts before failing anyway, burying the real error. For those, throw an AbortTaskRunError: the run fails immediately, skipping every remaining retry. The heuristic to internalize is retry on transients, abort on permanents.

A transient failure clears on its own: a 5xx from an overloaded service, a dropped connection, a 429 rate-limit. A permanent failure won’t: a malformed payload, a validation error, a bug in your own code.

Transient → retry

const res = await fetch(downstreamUrl);
if (res.status === 429 || res.status >= 500) {
  throw new Error(`Downstream unavailable: ${res.status}`);
}

A bare throw hands the failure to the back-off, and a later attempt will likely succeed.

Permanent → abort

const parsed = payloadSchema.safeParse(input);
if (!parsed.success) {
  throw new AbortTaskRunError('Invalid payload — retrying will not fix it');
}

A payload that failed validation fails identically every time, so AbortTaskRunError ends the run now instead of wasting four back-off windows.

Throw AbortTaskRunError for permanent failures; everything else throws normally and rides the back-off.

A task just hit each of these failures. Decide whether the runtime should retry it, or whether you should throw `AbortTaskRunError` and stop now. Drag each item into the bucket it belongs to, then press Check.

Retry (transient) Likely to clear on its own — let it throw

Abort (permanent) Will never succeed — throw AbortTaskRunError

Resend returns 429 Too Many Requests

Postgres connection reset mid-query

Stripe responds 500

The payload fails Zod validation

404 from a resource that will never exist

A downstream service responds 503 Service Unavailable

Check Reset

Run-level retries and the duplicate-side-effect trap

What a retry re-runs depends on which of two layers you mean.

A run-level retry is the one you just configured: on an unhandled throw, the runtime re-runs the task from its most recent checkpoint. When the task body has no checkpoints between its start and the line that threw, that checkpoint is the very beginning, so every line runs again.

A call-level retry is narrower: an SDK or HTTP client re-attempts a single failed request, like a wrapper that quietly retries a 429’d fetch. That restarts the one call, not the run. You configure run-level retries; call-level retries run underneath.

The danger is when a replayed line has a side effect: anything that touches the outside world and can’t be quietly taken back, like a row written, an email sent, or a card charged. Picture a task that loops over an organization’s members and emails each one. It throws on member two hundred after a transient blip, and the retry restarts from the top, re-sending to members one through one hundred and ninety-nine, who already got the email on the first pass. That is how most duplicate-email incidents happen.

Predict what the program below prints.

This is a sketch of a task body, not runnable code — reason it through. The task logs a line per member, then throws on the third iteration. Run-level retries are configured to allow two attempts in total, and there are three members. What does the worker's log show across both attempts? Predict what this program prints, then press Check.

async function run() {
  for (const member of ['Ada', 'Bo', 'Cy']) {
    if (member === 'Cy') throw new Error('transient blip');
    console.log(`sent to ${member}`);
  }
}

Check

Idempotency keys make a retried step run once

You guarded Stripe webhooks against double-processing with a stable key and a unique constraint. Trigger.dev gives you that same guarantee as a runtime primitive: a stable key collapses repeats into one run.

Every trigger, triggerAndWait, and batchTriggerAndWait accepts an idempotency key . Within a time window, the same key returns the same run: no new run starts, no body re-executes, and you get a handle to the original run, finished or in flight, with its result. Re-trigger that key a thousand times and the work happens once.

await chargeCustomer.trigger(payload, {
  idempotencyKey,
  idempotencyKeyTTL: '24h',
});

That window is idempotencyKeyTTL , a duration string like '60s', '5m', or '3d', not a number of milliseconds. Leave it off and it defaults to thirty days. Inside the window a late duplicate, a retried POST or a double-click, maps to the same run; past it, the key is free to start a fresh one.

Rather than splice strings by hand, call idempotencyKeys.create(parts, { scope }) with an array of parts like [organizationId, 'export', day], which it hashes into one stable key, as if organizationId:export:day were glued together safely.

Scope decides what the key is namespaced against:

• scope: 'run', the default, hashes the key with the parent run id. Use it for keys inside a retrying task: a retry regenerates the same keys, and the runtime recognizes the child work as already done. It replaces manually prefixing ctx.run.id onto your keys.
• scope: 'global' hashes the key alone, namespaced against nothing: “this runs once, ever”, a business key triggered from your app. One export per organization per day, however many times the button is clicked.
• scope: 'attempt' re-allows the work on each retry, rarely what you want.

// In a Server Action
const day = todayInTimeZone(org.timeZone);
const key = await idempotencyKeys.create([org.id, 'export', day], {
  scope: 'global',
});
await exportInvoices.trigger({ organizationId: org.id }, {
  idempotencyKey: key,
  idempotencyKeyTTL: '3d',
});

// Inside the task
for (const member of members) {
  const memberKey = await idempotencyKeys.create([member.id, 'notify'], {
    scope: 'run',
  });
  await sendOne.triggerAndWait({ memberId: member.id }, {
    idempotencyKey: memberKey,
  });
}

The app-side scope: 'global' key, built from [org.id, 'export', day] and namespaced against nothing: “one export for this org on this day, period.” Double-click the button or fire the action ten times, the first wins and the rest return that same run. The business rule lives in the key.

// In a Server Action
const day = todayInTimeZone(org.timeZone);
const key = await idempotencyKeys.create([org.id, 'export', day], {
  scope: 'global',
});
await exportInvoices.trigger({ organizationId: org.id }, {
  idempotencyKey: key,
  idempotencyKeyTTL: '3d',
});

// Inside the task
for (const member of members) {
  const memberKey = await idempotencyKeys.create([member.id, 'notify'], {
    scope: 'run',
  });
  await sendOne.triggerAndWait({ memberId: member.id }, {
    idempotencyKey: memberKey,
  });
}

The in-task scope: 'run' key, built per member from [member.id, 'notify']. This is the fix for the duplicate-send trap: hashed with the parent run id, a retry regenerates the identical key for each member, so the runtime returns the already-completed sends cached instead of mailing them again. Only the member it never reached actually sends.

// In a Server Action
const day = todayInTimeZone(org.timeZone);
const key = await idempotencyKeys.create([org.id, 'export', day], {
  scope: 'global',
});
await exportInvoices.trigger({ organizationId: org.id }, {
  idempotencyKey: key,
  idempotencyKeyTTL: '3d',
});

// Inside the task
for (const member of members) {
  const memberKey = await idempotencyKeys.create([member.id, 'notify'], {
    scope: 'run',
  });
  await sendOne.triggerAndWait({ memberId: member.id }, {
    idempotencyKey: memberKey,
  });
}

'3d' means a duplicate export request arriving up to three days later still collapses onto the first run. Leave it off and you’d get the thirty-day default.

← Previous 1 / 1 Next →

Treat an idempotency key as required on every trigger, triggerAndWait, and wait.forToken, the way a Server Action’s input schema is. A trigger without one is a duplicate side effect waiting for its first retry.

Inside a task that loops over members and may itself be retried at the run level, which key correctly sends each member exactly one notification — even across a retry of the parent?

const recipientKey = await idempotencyKeys.create(
  [member.id, 'notify'],
  { scope: 'global' },
);

const recipientKey = await idempotencyKeys.create(
  [member.id, 'notify'],
  { scope: 'run' },
);

const recipientKey = await idempotencyKeys.create(
  ['notify'],
  { scope: 'run' },
);

Durable pauses: wait.for and wait.until

Re-runs and checkpoints exist because something put a boundary in the task. Durable waits let you place those boundaries on purpose, and each comes with a gotcha.

wait.for pauses for a duration: await wait.for({ seconds: 2 }) or await wait.for({ minutes: 5 }). It does three things: checkpoints, frees the worker so you pay nothing while the run sleeps, and resumes after the duration on a possibly-new worker. Because no live process holds the pause, only a checkpoint and a wake-up time, it survives a crash. Reach for it to pace a loop between export pages, or to back off until a rate-limit window reopens.

The trap is that wait.for looks like setTimeout, the wrong tool here.

setTimeout — wrong

await new Promise((resolve) => setTimeout(resolve, 2_000));

Wrong on a durable platform. The worker sits idle for the full two seconds, so you pay for the wait, and the timer lives in process memory, so a crash evaporates it and the run never resumes.

wait.for — durable

await wait.for({ seconds: 2 });

Right. The pause is a checkpoint, not a live timer: the worker is freed, nothing is billed while it sleeps, and a crash mid-wait resumes cleanly on a new worker.

wait.until waits to an absolute wall-clock moment instead of a duration: await wait.until({ date }). The semantics match wait.for. Use it for “send the welcome email twenty-four hours after signup” or “act exactly at the trial’s period end.”

Its gotcha is quieter: if the date is already past, wait.until resolves immediately. It does not error and does not skip the rest of the task, it just falls through as if there were no wait. So if you mean “do nothing once this date has passed,” check the date yourself before acting.

Caution

A rule from the next lesson, worth planting now: any wait on an external signal must carry a timeout, because a signal that never arrives is an indefinite wait, and an indefinite wait is a resource leak. wait.for and wait.until bound themselves by their own duration; the timeout rule matters once you wait on tokens.

A paginated export that survives a crash

This task ties the four mechanisms together: it exports an organization’s invoices in pages of five hundred, publishes progress, then emails a download link. It’s the shape the CSV export takes in the next chapter.

export const exportInvoices = schemaTask({
  id: 'export-invoices',
  schema: z.object({ organizationId: z.string(), totalPages: z.number() }),
  retry: { maxAttempts: 5, factor: 1.8, randomize: true },
  run: async ({ organizationId, totalPages }) => {
    for (let page = 1; page <= totalPages; page++) {
      const key = await idempotencyKeys.create([organizationId, 'page', page]);
      await writePage.triggerAndWait({ organizationId, page }, { idempotencyKey: key });
      metadata.set('page', page);
      await wait.for({ seconds: 2 });
    }
    const doneKey = await idempotencyKeys.create([organizationId, 'export-email']);
    await sendReadyEmail.trigger({ organizationId }, { idempotencyKey: doneKey });
  },
});

The task has no session, so tenancy rides in on the payload as organizationId and travels into every child (writePage scopes its own queries from it). The org context is cargo, re-derived from data, never ambient.

export const exportInvoices = schemaTask({
  id: 'export-invoices',
  schema: z.object({ organizationId: z.string(), totalPages: z.number() }),
  retry: { maxAttempts: 5, factor: 1.8, randomize: true },
  run: async ({ organizationId, totalPages }) => {
    for (let page = 1; page <= totalPages; page++) {
      const key = await idempotencyKeys.create([organizationId, 'page', page]);
      await writePage.triggerAndWait({ organizationId, page }, { idempotencyKey: key });
      metadata.set('page', page);
      await wait.for({ seconds: 2 });
    }
    const doneKey = await idempotencyKeys.create([organizationId, 'export-email']);
    await sendReadyEmail.trigger({ organizationId }, { idempotencyKey: doneKey });
  },
});

The page loop. Each page gets its own scope: 'run' key (the default) built from the page number, then writePage.triggerAndWait runs it as a durable child step. A retry of the export regenerates these keys, so already-written pages return cached instead of re-exporting.

export const exportInvoices = schemaTask({
  id: 'export-invoices',
  schema: z.object({ organizationId: z.string(), totalPages: z.number() }),
  retry: { maxAttempts: 5, factor: 1.8, randomize: true },
  run: async ({ organizationId, totalPages }) => {
    for (let page = 1; page <= totalPages; page++) {
      const key = await idempotencyKeys.create([organizationId, 'page', page]);
      await writePage.triggerAndWait({ organizationId, page }, { idempotencyKey: key });
      metadata.set('page', page);
      await wait.for({ seconds: 2 });
    }
    const doneKey = await idempotencyKeys.create([organizationId, 'export-email']);
    await sendReadyEmail.trigger({ organizationId }, { idempotencyKey: doneKey });
  },
});

After each page, metadata.set publishes progress to the dashboard and the in-app inspector: a live “page 7 of 20” with no extra plumbing.

export const exportInvoices = schemaTask({
  id: 'export-invoices',
  schema: z.object({ organizationId: z.string(), totalPages: z.number() }),
  retry: { maxAttempts: 5, factor: 1.8, randomize: true },
  run: async ({ organizationId, totalPages }) => {
    for (let page = 1; page <= totalPages; page++) {
      const key = await idempotencyKeys.create([organizationId, 'page', page]);
      await writePage.triggerAndWait({ organizationId, page }, { idempotencyKey: key });
      metadata.set('page', page);
      await wait.for({ seconds: 2 });
    }
    const doneKey = await idempotencyKeys.create([organizationId, 'export-email']);
    await sendReadyEmail.trigger({ organizationId }, { idempotencyKey: doneKey });
  },
});

The pause between pages does double duty. It’s a checkpoint boundary, the save point that makes a mid-export crash resumable, and it paces the load on the database and downstream so the export doesn’t hammer them.

export const exportInvoices = schemaTask({
  id: 'export-invoices',
  schema: z.object({ organizationId: z.string(), totalPages: z.number() }),
  retry: { maxAttempts: 5, factor: 1.8, randomize: true },
  run: async ({ organizationId, totalPages }) => {
    for (let page = 1; page <= totalPages; page++) {
      const key = await idempotencyKeys.create([organizationId, 'page', page]);
      await writePage.triggerAndWait({ organizationId, page }, { idempotencyKey: key });
      metadata.set('page', page);
      await wait.for({ seconds: 2 });
    }
    const doneKey = await idempotencyKeys.create([organizationId, 'export-email']);
    await sendReadyEmail.trigger({ organizationId }, { idempotencyKey: doneKey });
  },
});

The final email carries its own idempotency key, so if the whole task retries after the last page, the “your export is ready” email sends once, not twice. Every side effect in the task is key-guarded.

← Previous 1 / 1 Next →

Now kill it. The worker dies at page seven of twenty. A new worker rehydrates from the last checkpoint, the wait.for after page six. Pages one through six don’t re-export: their triggerAndWait calls re-issue the same scope: 'run' keys, so the runtime returns the cached results of those completed child runs. Page seven re-executes from the top, which is fine, because writing a page is the work, not a duplicate side effect. The loop runs on to page twenty, the final email fires once under its own key, and the run finishes.

Cancellation is cooperative

Sometimes a run must stop before it finishes: a user cancels an export, or an admin kills a runaway job. You cancel from the Trigger.dev dashboard or with runs.cancel(runId).

Cancellation is cooperative : the runtime stops scheduling new steps at once, but a step already running stops only if you wired it to. The run exposes an AbortSignal that fires on cancel; forward it into your fetch and SDK calls so an in-flight request aborts instead of running to completion.

External resources

Trigger.dev v4 is the current line, and the web is still full of v3 examples that will quietly mis-teach you. For anything this lesson skipped, go to the current docs, not a blog post.

Trigger.dev — Idempotency

trigger.dev

Idempotency keys, TTL, and scopes: the canonical reference for the primitive this lesson centers on.

Trigger.dev — Errors & Retrying

trigger.dev

The retry config, AbortTaskRunError, and the full back-off surface, current as of v4.

AWS Builders' Library — Timeouts, retries, and backoff with jitter

aws.amazon.com

Marc Brooker's gold-standard write-up of why backoff needs jitter: the timeless theory under this lesson's retry knobs.

The next lesson adds the one wait this one skipped: pausing on an external signal, such as a human clicking approve or a third party calling back.