Coveralls is in Read Only mode while we work on updating the system. Sorry for the inconvenience.

VIGILANCE! Check this page Any time you notice a problem with coveralls

All Systems Operational
Coveralls.io Web ? Operational
Coveralls.io API ? Operational
GitHub Operational
Travis CI API Operational
Pusher Presence channels Operational
Pusher WebSocket client API Operational
Stripe API Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Average Web/API Response Time
Fetching
Coverage Calculation Background Job Dequeue Time ?
Fetching
Past Incidents
Sep 28, 2021

No incidents reported today.

Sep 27, 2021

No incidents reported.

Sep 26, 2021

No incidents reported.

Sep 25, 2021

No incidents reported.

Sep 24, 2021

No incidents reported.

Sep 23, 2021

No incidents reported.

Sep 22, 2021

No incidents reported.

Sep 21, 2021
Resolved - We are closing the incident after normal performance throughout the day. If you are experiencing issues, including performance issues, let us know at support@coveralls.io.
Sep 21, 22:12 PDT
Update - We will keep this incident open for at least another hour as we monitor performance, which has remained steady for the past ~45-min.

Apologies for the inconvenience.

We will share our findings in our postmortem here, but in the meantime a summary for those interested:

AWS/RDS support notified us mid-week last week that we were headed for transaction ID wraparound in a few days, so we decided to put the site in unplanned maintenance mode on Fri, Sep 17 at 5pm US PDT, expecting to be out before Mon, Sep 20. However, as we worked through suggested steps to avoid wraparound we encountered error after error, as well as extremely long-running steps like vacuum/auto-vacuum and reindexing tasks that took days instead of minutes or hours, even in non-concurrent mode. (Several attempts at concurrent mode proved fatal, forcing us to throw the site back into maintenance mode several times through the weekend, extending through Mon.) To improve performance on some of these long-running tasks, we tried upgrading Postgres to 13 (not available) and increasing our instance size (not available from RDS), so these attempts just added time to our efforts.
Sep 21, 09:29 PDT
Update - We believe this incident is resolved. We have taken the site out of maintenance mode and are monitoring performance, which has been normal and steady for 15-minutes.
Sep 21, 08:46 PDT
Update - Database index has finished rebuilding, but queries to the table are failing. Investigating new errors.
Sep 21, 06:48 PDT
Update - We are still monitoring progress on rebuild of a database index.
Sep 20, 20:13 PDT
Update - Continuing to monitor progress of database reindex.
Sep 20, 11:16 PDT
Update - We are continuing to monitor progress on database reindex.
Sep 20, 11:14 PDT
Update - We are continuing to monitor for any further issues.
Sep 20, 09:48 PDT
Monitoring - We are currently performing emergency database maintenance. We had completed to 95% by 3am and closed the previous maintenance window this morning when one index failed to rebuild and caused transactions to eventually slow and fail. We have taken the site back into maintenance mode to allow the remaining index to rebuild, and will take it back out of maintenance as soon as possible. We will provide progress updates when available.
Sep 20, 09:13 PDT
Sep 20, 2021
Resolved - Original incident resolved and maintenance window closed at 3:30am US PDT. Re-opened after subsequent slowdowns and transactions errors. We are addressing the remaining issues in the current maintenance window.
Sep 20, 03:31 PDT
Update - Upgrade complete. Rebuilding indexes now.
Sep 18, 22:18 PDT
Update - Vacuuming complete. We are now upgrading Postgres.
Sep 18, 12:04 PDT
Monitoring - We are currently performing emergency database maintenance to avoid transaction wraparound and will provide progress updates when available.
Sep 17, 19:50 PDT
Sep 19, 2021

No incidents reported.

Sep 18, 2021
Sep 17, 2021
Sep 16, 2021

No incidents reported.

Sep 15, 2021

No incidents reported.

Sep 14, 2021

No incidents reported.