We are keeping this issue open as we continue to monitor as we un-pause paused repos. Without further issues, we expect to resume normal operation by 6pm.
Posted Nov 30, 2022 - 17:13 PST
We have paused several repos from the group mentioned above as we determine the nature of their impact on other system users. We will unpause all repos by EOD today. If you think you may have been paused and want to confirm, email us at email@example.com.
Posted Nov 30, 2022 - 14:56 PST
We experienced some general performance issues last night that continued into early morning, but, this morning, should only be affecting a small group of repos:
The group of affected repos should be limited to ~20-25 repos and the defining characteristics of these repos are that they have:
- More than 100 jobs in one day, and/or - More than 40 jobs in one hour
Upon hitting those thresholds, these repos are being routed to a job processing queue that has lower precedence than queues that process jobs for the general population of repos.
If you think you may be affected by this issue because your repo may be exceeding these thresholds, feel free to contact us at firstname.lastname@example.org and we will confirm or advise.
If you believe you are experiencing slow performance (slow build times), but don't think your repo is exceeding the above thresholds, also please contact us at email@example.com, as we can verify that and also see if there's another issue at play for your repo.