Duplicate workflow jobs are processed. How do I fix this?

Short explanation:

Duplicate jobs can occur when a producer resends work or a consumer crashes before acknowledging a completed action. Many reliable delivery systems intentionally allow redelivery to avoid losing work. Correct processing therefore identifies the logical operation and makes repeated attempts safe.

Key points:

• Give each logical operation a stable idempotency key.

• Enforce uniqueness atomically with the state change; a check-then-insert can race.

• Use provider idempotency for external actions and acknowledge jobs after durable handling.

• A “processed” flag checked outside the same atomic state change can still race with another worker.

• For external effects, store durable intent and use provider-side idempotency or reconciliation rather than claiming a local flag solves every crash window.

Interview point:

• Walk through a crash between the effect and acknowledgement; the recovery path should preserve one intended result.