Using Rails 6. I have a long time Sidekiq worker, that can take up to 4-6 hours depending on the load it is given. But if the process stops due to an error or daily restarts, I end up loosing the progress made so far. I have been trying to implement a good caching mechanism because its basically a big loop and I can keep track and resume if I have the value of the loop when execution was stopped. Before I was only saving it the loop value in a variable and after say 200 updates, transfer it to Postgres database. Note: each loop iteration can take up to half a second on average. But I can still end up loosing the track if execution is stopped between the 200 iterations. I want to keep track of every single iteration. Should I use Redis to cache the loop iterator to prevent from loosing track? I can catch any error using exception handling and update the iterator value in PG. But can the SIGTERM signal from dyno restart/update be caught somehow? Or should I rely on Redis? Maybe a better way?
↧