Delayed jobs are 50% cleared. We are continuing to monitor.
Posted Nov 17, 2021 - 17:27 PST
Update
Background jobs have cleared by 15%. We have temporarily increased processing resources and are continuing to monitor. We expect most previously delayed jobs to be cleared in the next 1-2 hrs.
Posted Nov 17, 2021 - 14:18 PST
Update
We are continuing to monitor for any further issues.
Posted Nov 17, 2021 - 13:19 PST
Monitoring
A fix has been implemented and we are monitoring the results.
Note: If your repo had more than 200 jobs in queue we may have paused processing on it until the bulk of jobs in shared queues are cleared. To find out if you have 200+ jobs in queue, multiply your number of unfinished builds for today by the number of parallel jobs in each build. (Example: 20 builds X 10 parallel jobs each.) So far we have identified <10 repos with this scenario (among hundreds of thousands).
Posted Nov 17, 2021 - 12:11 PST
Identified
The issue has been identified and a fix is being implemented.