Postmortem -
Read details
Feb 17, 09:25 PST
Resolved -
We are closing this issue but will continue to monitor as we clear the remaining queues of background jobs from yesterday. If you believe any of your recent builds are still affected (incomplete), or if you are having any issues uploading coverage reports, please reach out to us at support@coveralls.io.
Feb 13, 12:41 PST
Update -
We are out of maintenance mode and monitoring live transactions.
Feb 13, 09:40 PST
Update -
Our ETA for reverting maintenance mode is within the next 30 minutes.
Feb 13, 09:15 PST
Update -
We are in maintenance mode as we perform some database tasks to improve our performance in clearing background jobs still stuck-in-queue. ETA: 2 hrs. But we may need to update this as we monitor progress.
Feb 13, 05:00 PST
Update -
We are not clearing background jobs fast enough to recover by morning US PST, so we will be putting the site into read-only mode for about 2 hrs from (4:30a-6:30a US PST) in order to perform some database operations.
Feb 13, 04:16 PST
Update -
We are continuing to monitor for any further issues.
Feb 12, 21:33 PST
Update -
We have resolved the partial outages and are monitoring.
Feb 12, 14:25 PST
Update -
We are still experiencing partial outages as we try to deploy across an extended fleet of servers. We are all hands and working to resolve asap.
Feb 12, 14:13 PST
Update -
Partial outages. We are working to resolve asap.
Feb 12, 14:00 PST
Update -
We are deploying extra servers to help clear backed up jobs. This will entail a rolling reboot, which may cause some users to lose their current connection to Coveralls.io. Your connection should be restored momentarily, so please try again in 30-sec to 1-min.
Feb 12, 13:43 PST
Update -
While we have applied a fix and are monitoring for any further issues, we are clearing backlogged jobs for some accounts. If you are waiting on some recent builds to complete, please give them at least another 20 minutes to clear. If you are not seeing your builds clear after that, please reach out to us with your org/subscription and repo name(s) at support@coveralls.io.
Feb 12, 13:20 PST
Monitoring -
A fix has been implemented and we are monitoring the results.
Feb 12, 13:08 PST
Identified -
The issue has been identified and a fix is being implemented.
Feb 12, 13:07 PST
Investigating -
We are currently investigating reports of service disruptions for some users, possible related to specific subscriptions or repos.
Feb 12, 12:54 PST