Coder Legion - Recent posts tagged sidekiq

Better Sidekiq Classes

Thu, 28 Aug 2025 23:31:09 +0000

Sidekiq is pretty much the go-to solution for enqueuing jobs for background processing when working on a Ruby-based project. It's simple to implement, has a clear DSL, and is well-supported by common testing frameworks like RSpec. That being said, and I may be nitpicking, but the standard implementation examples do add a little extra overhead if you wish to be more flexible with your job business logic, and additional test setup is required.

The "normal" way

Given we want a little background job that updates a user's name and any other business logic needed (to keep the example simple), we could create a standard Sidekiq job with that logic and call it from our code as such:

class UpdateUserJob
  include Sidekiq::Job

  def perform(user_id, name)
    user = User.find(user_id)
    user.update!(name: name)
    # .. additional business logic here
  end
end

UpdateUserJob.perform_async(1, "bob") # perform the job as soon as possible
UpdateUserJob.perform_in(5.minutes, 1, "bob") # delay the processing by 5 minutes

This is simple enough. But what if we want to test the logic being performed inside this job? Firstly, we'd need to update our spec_helper to let it know what to do with a job. If you're using RSpec, you might even need to add the rspec-sidekiq gem for additional configuration and matcher options. A classic-looking spec for this job might look something like this:

it "updates the user" do
  UpdateUserJob.perform_async(1, "bob")
  expect(UpdateUserJob.jobs.size).to eq(1)
  UpdateUserJob.drain
  expect(User.find(1).name).to eq("bob")
end

We need to drain the job before we can test the result of its work. Alternatively, we could wrap the test inside one of the Sidekiq testing helper methods:

it "updates the user" do
  Sidekiq::Testing.inline! do
    UpdateUserJob.perform_async(1, "bob")
    expect(User.find(1).name).to eq("bob")
  end
end

This does work as expected however it does leave a little to be desired. We have to remember to do this whenever testing job logic (the number of times I've had to search through previous specs to remind myself of the syntax), it adds a few extra lines to the tests which are not necessary and there is a spec performance hit as well.

The "use case" way

So, what options do we have? Well, the first option would be to extract the logic performed by the job into a class which we would then pass the job attributes to for processing. That way we can independently test the business logic without having to mess with background workers and the like. I have used various names for these classes, Use Case, Resource, or Library to name a few (use whatever best suits you and your conventions). Our new job would look something like this:

class UpdateUserJob
  include Sidekiq::Job

  def perform(user_id, name)
    UpdateUserUseCase.new(user_id, name).call
  end
end

class UpdateUserUseCase
  def initialize(user_id, name)
    @user_id = user_id
    @name = name
  end

  def call
    user = User.find(@user_id)
    user.update!(name: @name)
    # .. additional business logic here
  end
end

Now, when it comes to testing, we could just forgo testing the actual job class as it does nothing except pass on some arguments to a plain old Ruby object. We can just test the class directly. Sidekiq is also already extensively tested so there is little point in testing whether the jobs on the queue have increased or not unless you are expecting new jobs to be created as a knock-on effect of your business logic.

it "updates the user" do
  UpdateUserUseCase.new(1, "bob").call
  expect(User.find(1).name).to eq("bob")
end

The added benefit to doing it this way is that you have a plain old Ruby object that you can call independently and inline in your code elsewhere should the need arise. For instance, you may want to defer the processing to the background when handling a mass import of data, however, when being performed as a single HTTP request to your API, you could wish to do it inline.

The downside to this is that we now, however, have had to create an additional class to provide this flexibility. That means, if we wish to keep this convention going, for each Job class, there would be a corresponding UseCase class. What if we could take this idea one step further? What if we could turn our Job class into a plain old Ruby object right from the start? Let's rewrite our UpdateUserJob class.

The "runner" way (with POROs)

Let's pretend you had never heard of Sidekiq or background queues, and you want to write some code to perform a specific task. For the sake of convention, I have ended the following class name with "Job" to match previous examples however by syntax definition, it is no longer what we would call a classic "job". It is now just a Plain Old Ruby Object (PORO; identical to our previous UpdateUserUseCase).

class UpdateUserJob
  def initialize(user_id, name)
    @user_id = user_id
    @name = name
  end

  def call
    user = User.find(@user_id)
    user.update!(name: @name)
    # .. additional business logic here
  end
end

With that in place, we need to set up some sort of mechanism to have this code be run by Sidekiq in our background queue. We can introduce a simple class to wrap the sidekiq calls, allowing us to inject our custom job class with its arguments. To not conflict with the Sidekiq namespace, I have created the Runners module which will run this code for me (side note: no matter what project I've worked on in the past, I always seem to use these "runner" style classes for various things).

 # Cannot figure out why this code block fails to format correctly here on CoderLegion
module Runners
  class Sidekiq
    include ::Sidekiq::Job

    sidekiq_options(retry: 3, backtrace: true, queue: "default")

    def perform(job_class_name, *args)
      Kernel.const_get(job_class_name).new(*args).call
    end

    def run(job_class, *args)
      self.class.perform_async(job_class.to_s, *args)
    end

    def run_in(delay_in_seconds, job_class, *args)
      self.class.perform_in(delay_in_seconds, job_class.to_s, *args)
    end
  end
end

And with that, we have everything we need to be able to throw plain old Ruby classes into the background for processing. If we wish to run our business logic inline, we simply run

UpdateUserJob.new(1, "bob").call

but if we want to delegate the processing to the background queue, we can send it off immediately or delay it

Runners::Sidekiq.new.run(UpdateUserJob, 1, "bob") # perform the job as soon as possible
Runners::Sidekiq.new.run_in(5.minutes, UpdateUserJob, 1, "bob") # delay the processing by 5 minutes

Bells & Whistles

Now that we have our lovely Sidekiq runner class, we can embellish it with a couple final bells and whistles. We might want to add / override the sidekiq_options on demand instead of being stuck with the previous hard-coded options. Perhaps we prefer using keyword arguments in our method definitions for the added clarity and ability to insert the arguments in any order. And maybe, we would like the job names in the Sidekiq UI to be a little more descriptive as to the actual job they are running so that we don't have to go digging through the arguments list to find specific jobs.

 # Cannot figure out why this code block fails to format correctly here on CoderLegion
module Runners
  class Sidekiq
    include ::Sidekiq::Job

    DEFAULT_OPTIONS = { retry: 3, backtrace: true, queue: "default" }

    def perform(job_class_name, *args)
      if args in [Hash]
        # This handles the case of keyword arguments
        job_class_name.constantize.new(**args.first.symbolize_keys!).call
      else
        # This handles the case of basic arguments
        job_class_name.constantize.new(*args).call
      end
    end

    def initialize(options = {})
      self.class.sidekiq_options(DEFAULT_OPTIONS.merge(options))
    end

    def run(job_class, *args)
      update_display_class(job_class)
      self.class.perform_async(job_class.to_s, *args)
    end

    def run_in(delay_in_seconds, job_class, *args)
      update_display_class(job_class)
      self.class.perform_in(delay_in_seconds, job_class.to_s, *args)
    end

    private

    def update_display_class(job_class)
      self.class.sidekiq_options.merge!("display_class" => "#{job_class} [#{self.class.name}]")
    end
  end
end

In essence, you can now define your background job classes as basic Ruby objects, execute them inline, or transfer them to Sidekiq for background processing via our Runner. The main requirement is maintaining a consistent DSL with your job classes, such as implementing a mandatory call (or equivalent: perform or execute) method. (Kudos to my friend Kris for the original idea.)

Scaling Sidekiq in Ruby on Rails: Best Practices, War Stories, and WTF Moments

Mon, 25 Aug 2025 00:27:16 +0000

Imagine that your Rails application is running smoothly. Users are satisfied, payments are processed, emails are sent out, and reports are produced. Sidekiq feels like magic.

Then one Friday evening (because production issues always happen on Fridays), you check the Sidekiq dashboard ????

Jobs are piling up faster than your interns can say, “Did we forget to add sidekiq_options retry: false again?”

Welcome to Sidekiq at scale—where background jobs stop being background and start being your entire life.

Here are some lessons I've learned from implementing Sidekiq at scale in production, along with metrics and real-world examples.

1. Keep Jobs Small and Fast

Jobs should be atomic and quick. A 5–10 second job is already too long in most production systems.

Bad example:

class ReportJob
  def perform(user_id)
    user = User.find(user_id)
    generate_pdf(user)          # CPU-heavy
    upload_to_s3(user)          # I/O-heavy
    send_email(user)            # External API
  end
end

If this fails halfway, the entire job retries.

✅ Better:

class GeneratePdfJob < ApplicationJob
  def perform(user_id)
    PdfGenerator.call(User.find(user_id))
  end
end

class UploadReportJob < ApplicationJob
  def perform(file_path)
    S3Uploader.call(file_path)
  end
end

class SendReportJob < ApplicationJob
  def perform(user_id, report_id)
    ReportMailer.with(user_id:, report_id:).deliver_now
  end
end

Chain jobs with callbacks or Sidekiq batches.

2. Idempotency is Non-Negotiable

Jobs will run at least once, sometimes more. If jobs aren’t idempotent, you’ll double-charge customers or send duplicate emails.

Instead of:

order.mark_as_paid!
send_receipt(order)

Do:

order.update!(status: "paid") unless order.paid?
ReceiptMailer.send(order).deliver_later unless order.receipt_sent?

3. Use Queues Strategically

Don’t dump everything into default. Separate workloads:

critical → user-facing (emails, webhooks, payments)
default → normal jobs (notifications, syncs)
low → heavy, non-urgent (reports, exports, ETL)

sidekiq.yml:

:queues:
  - [critical, 5]
  - [default, 3]
  - [low, 1]

4. Monitor Everything

Visibility is critical at scale:

Sidekiq Web UI for queue depth & retries

SQL snippet to monitor retries:

SELECT count(*)
FROM sidekiq_jobs
WHERE queue = 'default'
AND retry = true;

5. Control Concurrency

More threads ≠ better throughput. High concurrency can overwhelm Postgres or external APIs.

✅ Start with concurrency = 2–5x your CPU cores.

:concurrency: 15

6. Redis is Your Heartbeat

Redis is the single point of truth for Sidekiq.

Use dedicated Redis (not shared with cache/session store)
Monitor memory & latency

7. Smart Retries Only

Default retries (up to 25) can flood your system. Customize them.

class MyJob
  sidekiq_options retry: 3

  def perform
    ExternalService.call!
  rescue ExternalService::InvalidCredentials
    # Don’t retry, just fail fast
    raise Sidekiq::Shutdown
  end
end

8. Deploy with Process Separation

Don’t run one mega Sidekiq instance. Separate by queue type.

bundle exec sidekiq -q critical,5 -q default,2
bundle exec sidekiq -q low

9. Dead Jobs are Zombie Failures

Dead jobs = silent failures.

bundle exec sidekiqctl clean --all

Or query them in Redis:

Sidekiq::DeadSet.new.size

10. Autoscale or Suffer

In Kubernetes, ECS, or Heroku, autoscale workers on queue latency:

Scale up if latency > 30s
Scale down if latency < 5s

11. Chain Jobs for External API Pagination

Sometimes, the firehose comes from outside your app. A classic case: syncing a huge dataset from a third-party API that only gives you paginated results.

If you naively fetch all pages in one job, you’ll hit rate limits, blow up memory, and make retries a nightmare.

✅ Better: chain jobs page by page. Each job handles one page, then enqueues the next, until the API says you’re done. That way:
Jobs stay tiny and idempotent.

Failures retry gracefully without restarting the whole sync.
You respect API rate limits with backoff and jitter.

You can fan out each page’s items to dedicated jobs for massive throughput.

Example:

class SyncExternalPage
  include Sidekiq::Worker
  sidekiq_options queue: :sync, retry: 10

  def perform(cursor = nil)
    response = ExternalClient.fetch_page(cursor: cursor)

    response[:items].each do |raw|
      UpsertExternalItem.perform_async(raw) # idempotent per item
    end

    if response[:next_cursor]
      # Chain the next page
      self.class.perform_in(1, response[:next_cursor])
    else
      Rails.logger.info("Sync complete ✅")
    end
  end
end

class UpsertExternalItem
  include Sidekiq::Worker

  def perform(raw)
    ExternalRecord.upsert(
      { external_id: raw["id"], name: raw["name"] },
      unique_by: :index_external_records_on_external_id
    )
  end
end

This pattern lets you chew through millions of external records without drowning your system in retries or hitting API bans.

Conclusion

Sidekiq is incredibly robust, but at scale, you need production-grade practices:

Keep jobs small, fast, idempotent
Prioritize with queues
Monitor latency & retries
Tune concurrency & Redis
Use autoscaling to survive traffic spikes
Chain jobs for external API pagination to handle massive datasets gracefully

With these practices, you can confidently handle millions of jobs per day without breaking a sweat.

✍️ Your turn: what’s your worst Sidekiq horror story? Did you also accidentally email your entire user base at 3am?