Prevent multiple task queuing for same concurrency key

[gautamsi]

I am trying to evaluate his for pg-boss replacement, have a question.

pg-boss allows to prevent duplicate task while in queue, that means if something is already processed already it can be queued, otherwise it will just be discarded if the item is already in queue. Idempotency is not helpful as it prevents running that task forever in the same task scope

I like to be

• queue task like scrape a website
• prevent same website being queued while previous one has not finished
• able to queue this again (if needed) only if previous run has already completed.
• able to scrape other website in parallel

is this possible?

[morgs32]

I think this is called a "dropping queue": https://effect.website/docs/concurrency/queue/#dropping-queue

[KeuqS]

For the “one pending scrape per website, while different websites run in parallel” behavior, combine a per-key concurrency limit with debounce or a small deduplication layer.

In the current SDK, the concurrency part looks like this:

import { task } from "@trigger.dev/sdk";

export const scrapeWebsite = task({
  id: "scrape-website",
  concurrency: { perKey: 1 },
  run: async (payload: { url: string }) => {
    // scrape payload.url
  },
});

await scrapeWebsite.trigger(
  { url },
  { concurrencyKey: normalizeUrl(url) }
);

That allows different websites to run concurrently, while runs sharing the same normalized URL are serialized. The important caveat is that perKey: 1 limits execution; it does not discard an already queued duplicate.

If repeated triggers arrive in a burst, use a debounce key for that website so later triggers reschedule the pending run instead of creating more runs. For a strict “drop while queued, then allow the next trigger after completion” policy, keep a short-lived application-level claim keyed by the normalized URL (or use idempotency with an appropriate TTL), and release/expire it when the run finishes. That avoids the “idempotency forever” problem while preserving parallelism across websites.

Current references:

• https://trigger.dev/changelog/multiple-concurrency-limits
• https://trigger.dev/changelog/debounced-task-runs
• https://trigger.dev/vs/bullmq

[KeuqS]

For the “one pending scrape per website, while different websites run in parallel” behavior, combine a per-key concurrency limit with debounce or a small deduplication layer.

In the current SDK, the concurrency part looks like this:

import { task } from "@trigger.dev/sdk";

export const scrapeWebsite = task({
  id: "scrape-website",
  concurrency: { perKey: 1 },
  run: async (payload: { url: string }) => {
    // scrape payload.url
  },
});

await scrapeWebsite.trigger(
  { url },
  { concurrencyKey: normalizeUrl(url) }
);

That allows different websites to run concurrently, while runs sharing the same normalized URL are serialized. The important caveat is that perKey: 1 limits execution; it does not discard an already queued duplicate.

If repeated triggers arrive in a burst, use a debounce key for that website so later triggers reschedule the pending run instead of creating more runs. For a strict “drop while queued, then allow the next trigger after completion” policy, keep a short-lived application-level claim keyed by the normalized URL (or use idempotency with an appropriate TTL), and release/expire it when the run finishes. That avoids the “idempotency forever” problem while preserving parallelism across websites.

Current references:

• https://trigger.dev/changelog/multiple-concurrency-limits
• https://trigger.dev/changelog/debounced-task-runs
• https://trigger.dev/vs/bullmq