The single backed-up queue on our troubled server is now clear, so all jobs related to slow builds are no longer pending, but are now in-progress (being processed by Coveralls).
Affected builds aside, performance should be back to normal for all users.
Again, if you think you were affected by, or are still affected by this incident, please reach out to firstname.lastname@example.org with your build URLs and we'll look into them for you.
Posted Oct 18, 2023 - 17:25 PDT
We are continuing to clear jobs from the backed-up queue and are at about 80%. We'll continue to monitor and update, but we expect most stalled jobs to start processing again in the next 15-min.
Posted Oct 18, 2023 - 17:04 PDT
We've received reports of slow build times from some users associated with some repos. We have identified one server in OOM hang that is servicing one queue, so customers in this queue are most likely the ones affected.
We will be clearing that queue as quickly as possible.
If you are experiencing slow builds, feel free to reach out to email@example.com and we can find out if any of your recent builds are involved.