Backed up job queues cleared by 90%. We are unpausing outlier repos. Will scale back down before EOD. Let us know if you have any further performance issues after today.
Posted Jun 15, 2022 - 15:48 PDT
Monitoring
We have manually scaled our infrastructure, have paused five (5) outlier repos, and are monitoring progress, which currently stands at about 40% cleared.
Posted Jun 15, 2022 - 14:58 PDT
Investigating
We are investigating recent slow builds as reported by a customer. Primary behavior is slow PR comments.
Dequeue times have spiked a couple of times in the last 24-hrs, but we haven't received notifications about performance based on traffic. Number of background jobs are normal for day and time, but we will be scaling infrastructure preventatively and may pause some outlier repos (>200 jobs in queue) until performance stabilizes. If you think your repo has been paused, check in with us at support@coveralls.io.