We have a fairly typical Ruby on Rails 6.1 app with a Redis cache store which we are also using for our session store. We are using Devise and an single-page app with lots of graphql requests.
Recently memory has been gradually increasing on the servers until hitting 95%. We're working on resolving...
What's strange is that the responses to the users seem to be that they are getting logged out. Does this make sense? I would expect slow or failing responses but not user sessions being terminated. They can refresh and re-login and usually keep working (perhaps hitting a different web server, currently running with ~6-8 and have been for a long time). Normally I would expect to see errors on the server (via Sentry) that we're having memory allocation issues, but instead we just see users getting logged out and no real errors in Sentry.
The Redis server does not seem to have any elevated metrics (CPU sits are 3%, memory around 25%). The only major change recently has been that we upgraded PostgreSQL from 11 to 15.
Any thoughts on what would be causing users to be logged out?
Generally when the servers hit high memory usage we restart them and that resolves the issue.
To fix the memory issues we're looking at jemalloc and resizing the servers.