I am using an API request and this API has a limit of 2 calls a second and 250 records per request. That's the gist.
I have created this background job that also has the option of a second background job. This may be overkill.
Flow:
- Order webhooks on creates from shopify
- Cron job once per day for that days orders in case webhooks fail.
Goal for request:
If there are >= 250
records/orders in the first API request, to then create a second background job in a new worker to fetch page 2 in about 3 minutes, and if page 2 has >= 250
, to then create a new background job in the same worker has page 2 (after 2 completes) 3 minutes after page 2 jobs started fetch page 3, and so on.. I use n
for the page and add 1
to n
if the 250 statement is true.
Cron Job:
shops = Shop.all
shops.map do |shop|
if shop.company.present?
ShopifyOrderUpdatesWorker.perform_later(shop)
end
end
Background job 1: (for first API call)
def perform(shop)
n = 1
orders = ShopifyAPI::Order.find(:all, params: {created_at_min: 1.day.ago.beginning_of_day.iso8601}, limit: 250, page: n )
while (orders.count >= 250) || n == 1
unless n =< 1
while n > 1 && orders.count >= 250
orders = ShopifyAPI::Order.find(:all, params: {created_at_min: 1.day.ago.beginning_of_day.iso8601 }, limit: 250, page: n)
#while orders.count >= 250 || n == 2
t = 3
ShopifyOrderUpdatesLimitWorker.delay_for(t.minutes).perform_later(orders)
n += 1 #add page to API call request
t += +3 #add 3 minutes for each loop to buffer the api call queue to avoid api limits to be safe
#end
end
end
if n == 1
orders.map do |order|
#code here
end
end
n += 1
end
end
Background job 2: (for any API call after the first)
def perform(orders)
orders.map do |order|
#code here
end
end
This way, all shops can update "quickly" without being in queue behind other shops. Shops that have a lot of orders will wait the same time they would in either case of doing all of this in one action or in 2.
Is this overkill? Done right for code
In reality, it is probably very rare a webhook will fail so the chances of the second background job being called is slim.
Any possible improvements or suggestions for the code?
This may not be the right place to ask this question, but if anyone has experience with shopify or similar api situations, what are you doing?