I'm running a Rails 5.1 application on google AppEngine.
Below is my config
Procfile
worker: bundle exec sidekiq -C config/sidekiq.yaml
pubsub: bundle exec rake run_analytics_queue_processor
sidekiq.yaml
:concurrency: 2
:timeout: 30
:queues:
- default
config/initializers/sidekiq.rb
# frozen_string_literal: true
require 'sidekiq/web'
url = CREDENTIALS[:redis_url]
Sidekiq::Web.use(Rack::Auth::Basic) do |user, password|
[user, password] == [CREDENTIALS[:sidekiq_username], CREDENTIALS[:sidekiq_password]]
end
Sidekiq.configure_server do |config|
config.redis = { url: url, id: nil }
end
Sidekiq.configure_client do |config|
config.redis = { url: url, id: nil, size: 12 }
end
puma.rb
# frozen_string_literal: true
# Puma can serve each request in a thread from an internal thread pool.
# The `threads` method setting takes two numbers: a minimum and maximum.
# Any libraries that use thread pools should be configured to match
# the maximum value specified for Puma. Default is set to 5 threads for minimum
# and maximum; this matches the default thread size of Active Record.
#
workers 2
threads 6, 6
preload_app!
rackup DefaultRackup
# Specifies the `port` that Puma will listen on to receive requests; default is 3000.
#
port ENV.fetch('PORT') { 3000 }
# Specifies the `environment` that Puma will run in.
#
environment ENV.fetch('RAILS_ENV') { 'local' }
on_worker_boot do
# Worker specific setup for Rails 4.1+
# See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
begin
ActiveRecord::Base.connection.disconnect!
rescue StandardError
ActiveRecord::ConnectionNotEstablished
end
ActiveRecord::Base.establish_connection
end
# Specifies the number of `workers` to boot in clustered mode.
# Workers are forked webserver processes. If using threads and workers together
# the concurrency of the application would be max `threads` * `workers`.
# Workers do not work on JRuby or Windows (both of which do not support
# processes).
#
# workers ENV.fetch("WEB_CONCURRENCY") { 2 }
# Use the `preload_app!` method when specifying a `workers` number.
# This directive tells Puma to first boot the application and load code
# before forking the application. This takes advantage of Copy On Write
# process behavior so workers use less memory.
#
# preload_app!
# Allow puma to be restarted by `rails restart` command.
plugin :tmp_restart
In my procfile the bundle exec rake run_analytics_queue_processor
is calling a rake job that endlessly runs and takes in messages from our PubSub service and queues them to a sidekiq/ActiveJob background job
I have been getting multiple Timeout Timeout::Error: Waited 1 sec
issues when calling perform_later
with sidekiq jobs. Looking up the issue it seems like the connection pooling for redis was incorrect. I believe it is threads * concurrency which should be 12
in my case, which is what I am setting the redis pool size to in my sidekiq.yaml
Does anything in my config look glaringly wrong?