Queues 90% cleared. Performance back to normal. Unpausing outlier repos.
Posted May 24, 2022 - 20:09 PDT
Job queues being stubborn. Scaling infrastructure further to get past 67% cleared.
Posted May 24, 2022 - 16:32 PDT
Queues cleared by 65%. Continuing to clear them and monitor performance.
Posted May 24, 2022 - 13:53 PDT
Jobs in backed up queues have come down by half. We paused ~10 repos, which we'll unpause when we've cleared by 80%.
Posted May 24, 2022 - 11:57 PDT
We are continuing to monitor for any further issues.
Posted May 24, 2022 - 10:16 PDT
We are experiencing slower than normal performance right now due to an unusually high number of background jobs stuck in queue. This is a result of increased traffic or excessive posts by one or more repos. We are working to clear the queues right now, which may involve pausing processing on outlier repos. If you think your repo may have been paused, email us to find out at firstname.lastname@example.org.