Rails 8 SolidQueue Database Connection Pool Configuration on Heroku

So, I broke production today. Not in a small way either. I’m writing this because if you’re running Rails 8 with SolidQueue on Heroku, you might run into the same thing, and I’d rather you learn from my mistake than make it yourself.

The Setup

I’ve been working on a Rails 8 app that uses SolidQueue for background jobs. I like Rails because it makes sense to me. I’m an Ops person who happens to write Ruby, not the other way around. Rails just clicks for me in a way other frameworks don’t.

I have everything running on Heroku. The app uses Puma as the web server and SolidQueue for processing background jobs. Everything had been running fine for a while.

The Original Problem (That I Was Trying to Fix)

Yesterday afternoon, I noticed that background jobs were being enqueued successfully, but they weren’t actually running. You’d submit a job, see it go into the queue, and then…nothing. The job would just sit there and never seemed to progress.

This is the kind of problem that makes you question everything. Is the queue adapter configured correctly? Are workers running? Is there some silent failure happening?

I started digging into the configuration. In development, I’m using AsyncAdapter which is fine for local work that processes jobs in-memory immediately. But in production, I’m using SolidQueue, which persists jobs to the database and processes them with workers.

The Config that Broke SolidQueue

I committed a few changes meant to target some memory issues, pushed to GitHub, then deployed to Heroku. The deployment went through fine. Build succeeded. Assets compiled. Everything looked good.

Then I checked the logs…

Solid Queue is configured to use 4 threads but the database connection pool is 3. Increase it in `config/database.yml`
Exiting...

The app crashed immediately on boot. Nothing graceful about this shutdown. Production was down.

The Panic

My first thought was: “Eek! I need to revert immediately.” So I did. I reverted the Puma config changes back to exactly what was there before. Pushed again. Deployed again.

Still crashed. Same error.

That’s when I realized the problem wasn’t just my Puma changes. The database connection pool was too small. My changes had exposed a configuration issue that was already there, but it wasn’t breaking things because SolidQueue wasn’t actually running before.

The Real Problem

SolidQueue needs database connections. Each thread that SolidQueue uses needs its own connection from the pool. If you configure SolidQueue to use 4 threads but only give it a pool of 3 connections, it can’t start. It fails immediately.

Here’s what our config/queue.yml looked like:

default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues: "*"
      threads: <%= ENV.fetch("SOLID_QUEUE_THREADS", 2).to_i %>
      processes: <%= ENV.fetch("JOB_CONCURRENCY", 1).to_i %>
      polling_interval: 0.1

The default is 2 threads per worker. But SolidQueue was saying it needed 4 threads. I’m still not 100% sure why that is a specific threshold. It might be internal threads for the dispatcher, or it might be calculating something differently. The point is, it needed 4.

My config/database.yml had this for the queue connection:

production:
  queue:
    <<: *primary_production
    url: <%= ENV["DATABASE_URL"] %>
    migrations_paths: db/queue_migrate
    pool: <%= ENV.fetch("SOLID_QUEUE_POOL", ENV.fetch("RAILS_MAX_THREADS", 2).to_i + 1).to_i %>

It doesn’t take Dr. Richard Nash to figure out the math I was on the wrong side of:

RAILS_MAX_THREADS defaults to 2
Pool size = 2 + 1 = 3
SolidQueue needs 4
3 < 4 = RIP my app

I didn’t account for SolidQueue’s actual pool size requirements. It worked before because SolidQueue wasn’t actually starting (the plugin wasn’t loading), so the pool size didn’t matter. When I “fixed” the Puma config to explicitly enable SolidQueue, it tried to start, hit the pool size limit, and crashed.

The Fix

I did two things:

Reverted the Puma config to exactly what was working before
Fixed the database pool size to actually meet SolidQueue’s requirements

Here’s the fix for config/database.yml:

production:
  queue:
    <<: *primary_production
    url: <%= ENV["DATABASE_URL"] %>
    migrations_paths: db/queue_migrate
    # Solid Queue needs pool size >= number of threads (default is 4 threads from queue.yml)
    pool: <%= ENV.fetch("SOLID_QUEUE_POOL", 5).to_i %>

Changed from RAILS_MAX_THREADS + 1 (which was 3) to a fixed default of 5. This gives SolidQueue enough connections with a bit of headroom. You can still override it with the SOLID_QUEUE_POOL environment variable if you need to.

The Puma config went back to:

plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"]

on_worker_boot do
  ActiveRecord::Base.establish_connection if defined?(ActiveRecord)
end

The original, working version. No “improvements.”

Why This Happened

I made several mistakes:

I changed working code without understanding why it worked. My “improvement” worked without issue locally because the queuing configuration for the development environment is entirely different.
I didn’t test the configuration change. I should have tested locally with RAILS_ENV=production to see if SolidQueue would actually start. But I didn’t. I just assumed it would work.
I didn’t understand the database pool requirements. I saw the calculation RAILS_MAX_THREADS + 1 and assumed it was correct without checking what SolidQueue actually needed.
I deployed to production without testing in staging. We don’t have a staging environment that matches production exactly, but I should have at least tried to simulate it locally.

What I Should Have Done

If I had to do this over again, here’s what I’d do:

Understand the existing configuration first. Why was it written that way? What edge cases does it handle?
Test locally with production-like settings:RAILS_ENV=production SOLID_QUEUE_IN_PUMA=1 rails server This would have caught the pool size issue immediately.
Check the actual requirements. SolidQueue’s documentation is pretty clear about pool size requirements. I should have read it.
Add validation. I could have added an initializer that checks the pool size on startup and fails fast with a clear error message if it’s too small.

The Math Behind Pool Sizing

For anyone else dealing with this, here’s how to calculate the pool size you need:

According to SolidQueue’s requirements, the pool must be at least:

pool_size >= (threads × processes) + dispatcher_threads

For our setup:

threads: 2 (from queue.yml)
processes: 1 (from queue.yml)
Dispatcher threads: typically 1-2
Minimum: 2 × 1 + 2 = 4

We set it to 5 to give some headroom. You might need more if you’re running multiple worker processes or have higher thread counts.

Heroku Considerations

If you’re running on Heroku, you need to think about total connection usage. Heroku Postgres has connection limits based on your plan (see their docs for details).

For our setup:

Puma: 2 threads × 1 worker = 2 connections
SolidQueue pool: 5 connections (even though it only uses 2-4 at a time)
Total: ~7 connections

That’s well within even a Hobby Dev plan’s 20 connection limit, but if you scale up workers or threads, you’ll need to watch it.

You can check your connection usage with:

heroku pg:info

Prevention: Add Validation

After this incident, I added a startup validation to catch this kind of thing early:

# config/initializers/solid_queue_validation.rb
if defined?(SolidQueue) && Rails.env.production?
  Rails.application.config.after_initialize do
    queue_config = SolidQueue::Configuration.new
    pool_config = ActiveRecord::Base.configurations.configs_for(env_name: Rails.env, name: :queue)
    
    if pool_config
      pool_size = pool_config.pool
      # Calculate minimum required pool size
      required_pool = queue_config.workers.sum { |w| w[:threads] * w[:processes] } + 2
      
      if pool_size < required_pool
        raise "SolidQueue requires pool size of at least #{required_pool}, but database.yml has #{pool_size}. " \
              "Update config/database.yml production.queue.pool to at least #{required_pool}"
      end
    end
  end
end

This will fail fast on startup with a clear error message if the pool is too small. Better to fail in development or staging than in production.

Lessons Learned

Don’t change working code without understanding why it works. This is ops 101, but I violated it anyway.
Test configuration changes locally first. Even if you don’t have a perfect staging environment, you can simulate production locally.
Read the documentation. SolidQueue’s docs are clear about pool size requirements. I should have checked.
Add validation for critical configuration. If something can break production, validate it on startup.
Revert immediately when production breaks. Don’t try to debug in production. Revert first, then figure out what went wrong.

SolidQueue Pool Size Lessons Learned the Hard Way

The fix was simple: change pool: 3 to pool: 5. But the change led to a complete production outage for two hours.

This is a reminder that even small configuration changes can have big consequences. As an ops person, I should know better. But sometimes you get complacent, or you think you’re “improving” something, and you break production.

The good news is that it’s fixed now, and I’ve added validation to prevent it from happening again. The bad news is that I broke production, and that’s on me.

If you’re running SolidQueue on Heroku, check your config/database.yml and make sure your queue connection pool is at least 5 (or calculate it based on your actual thread configuration). Don’t make the same mistake I did.

References

SolidQueue GitHub – The actual library we’re using
SolidQueue Configuration – How to configure it properly
Heroku Postgres Connection Limits – Know your limits
Rails Database Configuration – The Rails way