I have something going on with sidekiq and i'm not sure how to solve it.
My api is monitoring folders on syncplicity (its similar to sharefile), looking for new folders that are being created. It then will record the information about the files, check them against an xml file that should be accompanying them, and then move the folder where it belongs. All this is handled with several jobs that run in the background.
What's happening is, there are a lot of folders being added, so there is a lot of work to be done.
And the number of jobs being processed has dropped to a very low amount. Maybe 1 ever couple of minutes.
I increased the number of threads from 10 to 30. I have a connection pool of 50 on my database.
This is what it looked like before I increased the number of threads:
And this is what it looks like after:
Here's a screen capture of top running on the server. It's not using very much in the way of resources at all.
Here's my sidekiq config:
staging: :concurrency: 30production: :concurrency: 30:queues: - [default, 1] - [event_importer, 1] - [import_birth_record, 3] - [import_images, 5] - [move_folder_worker, 4] - [cart_cleaner, 1]What can I do to increase throughput? Adding more workers has resulted in me seeing more of these errors come up:
Faraday::ConnectionFailed: Failed to open TCP connection to localhost:9200 (Connection refused - connect(2) for 127.0.0.1:9200)