TestOps is experiencing degraded performance

Incident Report for Katalon

Resolved

TestOps is experiencing degraded performance due to Cache transfer was taking longer than expected due to a high volume of data within the system, which cause a portion of our fleet to stuck in a crash loop when booting up after a hotfix deployment earlier today. Due to this crash loop issue, we were operating with a lower than usual capacity (hence service degradation from 10am to 1:30pm) while the team trying to find a way to allow a huge cache transfer to complete. After exhausting all the available option, without success, we had to issue a total service recycle (took 20s between 1:29pm to 1:30pm) to free up the cache and allow out fleet to start up with a clean state, which bring the service back to its normal operation at 1:31pm.

Posted Nov 02, 2023 - 03:30 UTC