# How to Run Background Jobs Without Losing Work > How to run background jobs without losing work: pick a job queue, make jobs idempotent, retry with backoff, watch failed jobs and keep cron visible. ```yaml url: "https://makerhunt.io/studio/how-to-run-background-jobs-without-losing-work" markdown: "https://makerhunt.io/studio/how-to-run-background-jobs-without-losing-work.md" title: How to Run Background Jobs Without Losing Work type: studio_article category: Growth tags: "background jobs, job queue, idempotent jobs, exponential backoff, Postgres job queue, Sidekiq, cron vs queue, dead letter queue, makers, Growth" featured_image: "https://txmhk1zrnc.ufs.sh/f/xSkWTCqmKWx9OFk1E7oKIa721XeYDqdAk50ThsHNwmQz9Bvu" published_at: "2026-11-04T09:55:26.918Z" updated_at: "2026-11-04T09:55:27.503Z" ``` ## About this studio article How to run background jobs without losing work: pick a job queue, make jobs idempotent, retry with backoff, watch failed jobs and keep cron visible. ## Article To run background jobs without losing work, move anything slow, flaky or not needed for the response out of the web request and into a job queue, make every job safe to run twice, retry failures with exponential backoff and a hard cap, send whatever finally fails to a list you actually look at, and keep scheduled jobs and their history visible. The rest of this post is the detail that keeps a queue from becoming a new place where work disappears. This guide is for solo makers and teams of two or three whose app sends emails, builds exports, calls an AI model or another API, or processes imports inside the request, and has started to time out, double-send or drop work. The worked example near the end is made up. Tool behaviour comes from each project's own documentation, linked below and checked on 4 November 2026. ## What counts as a background job? A background job is work your app records now and runs a moment later in a separate process, called a worker, so the user gets a response without waiting. You probably need one when you see any of these: - A button that sometimes spins until your host gives up, usually because it calls a third-party API. - Customers who get two welcome emails, or none, after a signup that was retried. - A CSV import that works for 200 rows and dies at 20,000. - A cron script on a server that nobody has checked since you wrote it. ## When do you not need a job queue yet? If the slow part is fast enough that nobody notices and failure costs nothing, leave it in the request. How fast is fast enough is a judgment call, not a rule. A queue adds a worker process and a new kind of bug, so add it when the pain is real. The one exception is anything that talks to an outside service you do not control. ## What should a job actually contain? Store IDs, not objects. The [Sidekiq best practices page](https://github.com/sidekiq/sidekiq/wiki/Best-Practices) puts it bluntly: arguments are saved as JSON, complex objects do not survive the trip, and even if they did, the object may have changed by the time the job runs. Pass `invoice_id` and load the invoice inside the job. Then think about when the job is enqueued. Enqueue before the database transaction commits and a fast worker can look for a row that does not exist yet, or that gets rolled back. Enqueue after it commits and a crash in between leaves the record saved but the job never queued. The [River guide to transactional enqueueing](https://riverqueue.com/docs/transactional-enqueueing) walks through both. A queue that lives in the same database as your data can insert the job inside the same transaction, so both happen or neither does. If your queue lives elsewhere, enqueue after commit and run a small sweeper that finds records still waiting for their job. ## Which background job queue should you pick? Pick the one that fits the stack you already run: a Postgres job queue if you have Postgres, a Redis-backed queue if Redis is already there, or a hosted service if you are on serverless and do not want to run a worker at all. ### Postgres-backed queues These store jobs as rows in your existing database. What makes them practical is the `SKIP LOCKED` option on row locks, documented in the [Postgres SELECT reference](https://www.postgresql.org/docs/current/sql-select.html): a worker locks the job it takes and other workers skip that row instead of waiting. - **Rails:** the [Solid Queue README](https://github.com/rails/solid_queue) describes a database-based backend for Active Job, configured by default in new Rails 8 apps, with recurring jobs built in. GoodJob is a Postgres-based alternative with cron-like jobs and a dashboard. - **Elixir:** [Oban](https://hexdocs.pm/oban/Oban.html), which stores jobs in PostgreSQL, MySQL or SQLite. - **Node:** pg-boss and [Graphile Worker](https://worker.graphile.org/docs), both built on SKIP LOCKED. - **Go and Python:** River for Go, and Procrastinate for Python with Django integration. ### Redis-backed queues Sidekiq for Ruby, BullMQ for Node and other languages, Celery and RQ for Python. If you already run Redis, they add nothing new to host. [BullMQ's own overview](https://docs.bullmq.io/) now says it can run on PostgreSQL as well as Redis and compatible stores. ### Hosted and serverless options Inngest runs your functions as steps with retries built in. AWS SQS and Cloudflare Queues are message queues you pair with your own consumer code. You trade the worker you would babysit for another vendor and pricing you should check the day you sign up. If this is you Start with Why One server, already on Postgres A Postgres job queue for your language No new service; jobs can share your transactions Already running Redis Sidekiq, BullMQ, Celery or RQ Mature tooling, nothing new to host Serverless, no long-running process A hosted queue or step-function service Nothing to keep alive between requests Rails 8 on default settings Solid Queue It is already configured Not sure, low volume Whatever uses the database you have Fewest moving parts People search "Sidekiq vs" every other option, but for a small app the backend matters less than the habits below. ## What is an idempotent job, and why does every job need one? An idempotent job is one you can run twice with the same input and end up where one run would leave you. You need this because almost every queue promises to deliver a job at least once, not exactly once. The [AWS SQS at-least-once delivery page](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html) tells you to design for receiving the same message more than once, and the [Cloudflare Queues delivery guarantees page](https://developers.cloudflare.com/queues/reference/delivery-guarantees/) says the same. In plain words: a worker can finish the work, crash before it reports "done", and the queue hands the job to another worker. If that job sends an email, your customer gets two. Three fixes: 1. **Check, then act, then record.** Before sending the receipt, check a `receipt_sent_at` column; after sending, set it. Lock the row so two workers cannot both pass. 2. **Unique keys.** Give the side effect a natural key (one export per user per day, one reminder per invoice per stage) and let a unique index reject the duplicate. 3. **Provider idempotency keys.** The [Stripe idempotent requests reference](https://docs.stripe.com/api/idempotent_requests) accepts keys up to 255 characters and says keys can be removed once they are at least 24 hours old. Create the key once per action and reuse it on every retry (store it with the job, or build it from the invoice ID plus the purpose), and keep emails and other personal data out of it. It is the same idea as in our post on [shipping a public API your first customers will use](https://makerhunt.io/studio/how-to-ship-a-public-api-your-first-customers-will-use), pointed the other way. If the job was started by a webhook, the receiving side has its own traps, covered in this walkthrough of [how Stripe webhooks break in production](https://dev.to/posting-dude/your-stripe-webhook-works-in-test-mode-heres-how-it-breaks-in-production-198). ![An open notebook with handwritten lists and a black pen on a wooden desk, beside a laptop keyboard and a black external drive](https://images.pexels.com/photos/8652316/pexels-photo-8652316.jpeg?auto=compress&cs=tinysrgb&w=1200&h=800&fit=crop) ## How should background jobs retry? Retry with exponential backoff and jitter, cap the number of attempts, and do not retry errors that can never succeed. Backoff makes each wait longer than the last; jitter adds randomness so a hundred failed jobs do not all retry in the same second. Marc Brooker's post on [exponential backoff and jitter](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) is still the clearest explanation of why. Defaults differ a lot, so look yours up. The [Sidekiq error handling page](https://github.com/sidekiq/sidekiq/wiki/Error-Handling) describes 25 retries over roughly three weeks before a job moves to the Dead set. The [BullMQ retry guide](https://docs.bullmq.io/guide/retrying-failing-jobs) only retries when you set `attempts` above 1. The [Inngest retries page](https://www.inngest.com/docs/features/inngest-functions/error-retries/retries) gives a default of four retries after the first attempt. Three weeks suits a job that can wait for your bug fix, not a password reset email that is useless an hour later. Set attempts per job, based on how long the result stays useful. ### Which errors to retry and which to fail fast - **Retry:** timeouts, connection errors, 5xx responses from another service, and 429 Too Many Requests. - **Fail fast:** most other 4xx responses (bad input, missing permission, a deleted record). Retrying just repeats the same answer. - **Respect the server:** when a 429 or 503 includes a `Retry-After` header, MDN's [Retry-After reference](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Retry-After) explains it is telling you how long to wait. Wait at least that long. ### Give every job a timeout A job with no timeout can hang on one slow API call and hold a worker forever. Set a timeout on the outbound call and a limit on the job as a whole, and treat hitting it as a retryable failure. ## Where do failed jobs go? When a job runs out of attempts it should land somewhere you can see it, with its error and payload and a way to run it again after you fix the cause. Sidekiq calls this the Dead set and keeps jobs there for six months. AWS SQS moves a message to a dead-letter queue after the number of receives you set in `maxReceiveCount`, per the [SQS dead-letter queue guide](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html). Cloudflare Queues does the same once its retry limit is reached. The failed list only helps if someone reads it. Book fifteen minutes a week. Fix the cause of each failure, then retry the batch. Watch for poison messages, payloads that crash the worker every time; a hard attempt limit plus the failed list is the cure. ## Should you use cron or a job queue for scheduled work? Use the scheduler in your job queue if it has one. The cron vs queue question comes down to three things a lone cron line does not handle: - **One instance only.** Run two app servers and a naive scheduler fires the nightly job twice. The [Oban periodic jobs docs](https://hexdocs.pm/oban/periodic_jobs.html) insert scheduled jobs from a single leader node for exactly this reason. - **Catch-up after downtime.** If the server was down at 02:00, does that run happen later or get skipped? Decide, and make the job safe either way. - **History.** Scheduled jobs in the queue get the same retries, failed list and logs. Knowing that a scheduled job has stopped is a monitoring problem, covered in the SideHunt guide to [monitoring a side project during your day job](https://sidehunt.io/studio/how-to-monitor-a-side-project-during-your-day-job). ## How do you handle jobs that take a long time? Split them. A 20,000-row import should be one job that processes a batch, saves where it got to and enqueues itself for the next batch, or a job that fans out small ones. A crash then costs you one batch, not the whole import. Save progress after each chunk (last processed ID or page token), make each chunk idempotent, and store total and done counts so the user can see progress. The same batching applies to data backfills, covered in our post on [changing your database schema without breaking production](https://makerhunt.io/studio/how-to-change-your-database-schema-without-breaking-production). ## What should users see while a job runs? Give every user-triggered job a visible status: queued, processing, done or failed. Store it on the record the user cares about (the export, the import), not only in the queue. Show progress for short jobs; for long ones, let them leave and email them when it is ready. On failure, say what happened and offer a retry button. Weak copy: "Something went wrong." Better copy: "Your export did not finish because the file was too large to process at once. Try a shorter date range, or reply to this email and we will run it for you." ## How do you deploy without killing jobs in flight? Tell workers to stop taking new jobs, give running jobs time to finish, then stop them. The [Sidekiq signals page](https://github.com/sidekiq/sidekiq/wiki/Signals) is a good model even if you use something else: `TSTP` at the start of a deploy stops it fetching new jobs, `TERM` at the end asks it to shut down within a timeout (25 seconds by default), and anything still running is pushed back to run again. One more reason every job must be idempotent. The other deploy trap is payload shape: jobs queued by old code run on new code. Make new arguments optional for one release, or add a second job class and let the old one drain. On ops, start with one worker process next to your web process. Keep concurrency low and limit jobs that call rate-limited APIs, so a burst of queued emails does not hammer your provider. ## A job inventory sheet Before you move anything, list every piece of async or scheduled work your app does. It usually turns up one job nobody remembered. Column What to write Job A short name, like "send receipt" or "nightly usage sync" Trigger User action, webhook, schedule or another job Payload The IDs it needs, nothing more Safe to run twice? Yes, or how you will make it so Attempts and timeout Based on how long the result stays useful If it finally fails Who finds out, and what the user sees Queue Fast or bulk ## A made-up example: Plinthbox moves proposal PDFs into a queue Plinthbox is invented for this post: a two-person app that lets independent furniture makers send clients branded PDF proposals. Bastian, the founder, generated the PDF inside the "Send proposal" request, then emailed it. Big proposals timed out, so makers clicked again and clients got two emails. A nightly reminder script for unsigned proposals had stopped after a server move, and nobody noticed for weeks. Bastian already ran Postgres, so he added a Postgres job queue and one worker. "Send proposal" now saves the proposal as "queued" and enqueues a job with only the proposal ID, in the same transaction. The job builds the PDF, sends the email with a key made from the proposal ID, and sets `sent_at`; if `sent_at` is already set, it stops. PDF jobs get five attempts with backoff, and the email call fails fast on a 4xx. The reminder script became a scheduled job in the queue. The proposal page shows "Preparing your PDF", then "Sent to the client", or a plain failure message with a retry button. ## What does weak vs better look like? Area Weak Better Payload The whole user object, serialized The user ID, loaded fresh in the job Enqueue timing Before the save commits Same transaction, or after commit with a sweeper Duplicates "It only runs once" A sent_at check, a unique key or a provider idempotency key Retries One default for every job Attempts per job, backoff with jitter, 4xx fails fast Failures Logged and forgotten A failed list read weekly, with a retry button Scheduled work A cron line on one server The queue's scheduler, one instance, with history User status "Something went wrong." Queued, processing, done or failed, with next steps ![Blank pale pink and yellow-green sticky notes scattered on a dark wall in slanting sunlight and shadow](https://images.pexels.com/photos/36352378/pexels-photo-36352378.jpeg?auto=compress&cs=tinysrgb&w=1200&h=800&fit=crop) ## What traps catch small teams? - **Treating the queue as a database.** Job history gets pruned. Keep status on your own records. - **One giant queue.** A bulk export at the front of the line delays every login email behind it. - **No worker in production.** The deploy started the web process but not the worker, and jobs pile up quietly. Check queue length after every deploy for the first week. - **Testing only the happy path.** Run each new job twice on purpose and check nothing doubled. ## A 7-day plan to add background jobs This is a planning guess for an hour or two a day, not a benchmark. Day Do this 1 Fill in the job inventory sheet 2 Pick a backend that fits your stack and run one worker locally 3 Move the flakiest job (usually an outside API call) and make it idempotent 4 Set attempts, backoff and timeouts per job; decide which errors fail fast 5 Add user-facing status and a retry button 6 Move scheduled scripts into the queue's scheduler 7 Deploy with a worker, check the failed list, book the weekly look ## Background jobs FAQ ### Do I need Redis for background jobs? No. If you already run Postgres, a Postgres-backed queue such as Solid Queue, GoodJob, Oban, pg-boss, Graphile Worker, River or Procrastinate keeps everything in one database. ### What is an idempotent job? A job that leaves the same result whether it runs once or several times with the same input. Most queues deliver at least once, so any job that charges, sends or creates something must check whether it already did. ### How many times should a job retry? As many times as the result stays useful. A password reset might get a few quick attempts; a nightly sync can retry for hours. Always add backoff, jitter and a cap. ### Should I use cron or a job queue? Use your queue's scheduler for recurring work when it has one, because you get retries, history and protection against running twice. Either way, make the job idempotent. ## Move one job this week Pick the request that fails most often, move it into a job you can safely run twice, and give it a place to fail where you will see it. When your product is steadier and ready for more users, [MakerHunt](https://makerhunt.io/) is a good place to show it to other makers. ## Links - Article (HTML): https://makerhunt.io/studio/how-to-run-background-jobs-without-losing-work - AI-friendly Markdown: https://makerhunt.io/studio/how-to-run-background-jobs-without-losing-work.md ## Explore - [Studio index](https://makerhunt.io/studio) - [Blog](https://makerhunt.io/blog) - [Browse projects](https://makerhunt.io/projects) _HTML version: https://makerhunt.io/studio/how-to-run-background-jobs-without-losing-work_