This incident has been resolved. All repos unpaused.
Posted May 26, 2022 - 15:00 PDT
Queues cleared by 25%. Continuing to monitor.
Posted May 26, 2022 - 12:14 PDT
A fix has been implemented and we are monitoring the results.
Posted May 26, 2022 - 11:39 PDT
We are again experiencing slower performance due to backed up job queues, which are normally due to a spike in general usage. However, instead of higher traffic in general, the usage seems to be coming from a few outlier repos. We are pausing repos with 200 or more jobs currently in queue. (We have a few repos with more than 1K jobs in queue at this moment.) If you think your repo was affected, you can reach out to us at firstname.lastname@example.org. We will un-pause those repos as soon as we've cleared the queues by 80% for all other users and get performance back down to targets.