Coveralls is in Read Only mode while we work on updating the system. Sorry for the inconvenience.

VIGILANCE! Check this page Any time you notice a problem with coveralls

All Systems Operational

Coveralls.io Web Operational

Coveralls.io API Operational

GitHub Operational

Travis CI API Operational

Pusher Presence channels Operational

Pusher WebSocket client API Operational

Stripe API Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

System Metrics Month Week Day

Average Web/API Response Time

Fetching

Coverage Calculation Background Job Dequeue Time

Fetching

Past Incidents

Jul 4, 2022

No incidents reported today.

Jul 3, 2022

No incidents reported.

Jul 2, 2022

No incidents reported.

Jul 1, 2022

No incidents reported.

Jun 30, 2022

Production database maintenance

Completed - We have completed our maintenance.
Jun 30, 23:30 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jun 30, 22:15 PDT

Scheduled - We'll be undergoing required maintenance from 10p US PST to 2a US PST.
Jun 30, 22:00 PDT

Degraded web performance & slower than normal builds

Resolved - Resolving this incident as we enter our maintenance window from 10p US PST to 2a US PST.
Jun 30, 22:00 PDT

Monitoring - We have a performance issue rooted in our database, which, to resolve, will take several full maintenance activities with downtime. We tried to do so last night without luck due to time needed. We have scaled infrastructure to its maximum useful size to manage the current issue, but we will need to complete the full maintenance to return performance to normal. That will require additional planned maintenance window tonight from 10p-2a US PST.

This incident will remain open all of today as we monitor the situation until the next maintenance window.
Jun 30, 09:00 PDT

Unplanned database maintenance

Completed - We were unable to complete planned maintenance. We've scaled our infrastructure as much as possible for today's traffic. We will try again tomorrow to make these improvements.
Jun 30, 04:00 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jun 30, 01:00 PDT

Scheduled - In order to resolve our recent performance issues, we are forced to undergo unplanned maintenance, currently scheduled for Jun 29 at 10p US PST to Jun 30 at 2a US PST.
Jun 29, 22:00 PDT

Jun 29, 2022

Slow performance reported

Resolved - We are closing this incident, not because it is resolved but because in order to resolve it, we must undergo unplanned maintenance. We will close the incident and open the maintenance window, from approximately 10p US PST to 2a US PST.
Jun 29, 22:00 PDT

Update - We have scaled infrastructure to address performance. We have not paused any outlier repos. Continuing to monitor.
Jun 29, 13:53 PDT

Monitoring - We've identified the cause of current performance issues and have addressed it. We'll now implement normal scaling procedure to drain background queues more quickly. Look for performance to start improving immediately, with normal performance likely to return in 1-3 hrs.
Jun 29, 12:24 PDT

Investigating - We are currently investigating reports of slow performance (slow build times) reported by several customers and borne out by their repo stats. We aren't seeing metrics that indicate a performance slowdown for expected reasons, so we are investigating. We will potentially pause outlier repos and/or manually perform DB maintenance tasks. During such time coveralls.io may be in maintenance mode for 5-10 minutes. If longer wait times are expected, we will put coveralls.io into read-only mode.

If you think your repo may be an outlier, you can email us at support@coevralls.io to verify.
Jun 29, 10:12 PDT

Jun 28, 2022

No incidents reported.

Jun 27, 2022

No incidents reported.

Jun 26, 2022

No incidents reported.

Jun 25, 2022

No incidents reported.

Jun 24, 2022

No incidents reported.

Jun 23, 2022

No incidents reported.

Jun 22, 2022

No incidents reported.

Jun 21, 2022

Slow builds

Resolved - This incident has been resolved.
Jun 21, 18:00 PDT

Update - Original queue traffic cleared by 50%. Continuing to monitor.
Jun 21, 14:23 PDT

Monitoring - Traffic in one queue extremely backed up. Has been cleared by 25% so far by scale-up. One outlier repo with ~2,000 jobs in queue, paused. If you think it may be yours, you can email us at support@coveralls.io.
Jun 21, 13:28 PDT

Investigating - We are getting reports of slow builds from some customers. We're investigating while we scale up infrastructure.
Jun 21, 13:12 PDT

Jun 20, 2022

No incidents reported.