Coveralls is in Read Only mode while we work on updating the system. Sorry for the inconvenience.
VIGILANCE! Check this page any time you notice a problem with coveralls
Elevated Latency
Incident
Report for Coveralls
Resolved
We’re now closing this incident, several hours after restoring full system stability. Over the past 4 hours, we’ve continued to monitor key requests and queries closely. During that time, we identified a number of previously long-running queries that we’ve either:
- Optimized immediately, based on new platform characteristics; or
- Added to a short-term optimization backlog for tuning over the next few days.
These efforts are part of our ongoing work to adapt all app queries to the updated infrastructure context.
- Optimized immediately, based on new platform characteristics; or
- Added to a short-term optimization backlog for tuning over the next few days.
These efforts are part of our ongoing work to adapt all app queries to the updated infrastructure context.
Update
The site remains fully operational, and performance for all new builds is normal. We’re continuing to monitor request and query times closely to identify any long-running queries that may have contributed to recent job processing delays or latency spikes.
Update
Performance has been restored to standard, and site is fully operational, but we will continue to clear any previously blocked (or retry) jobs we discover in background job queues and monitor performance stats as they clear.
Update
Monitoring for further issues. Performance for new builds is normal. Waiting for dequeue metrics to fall below 50% normal before we lift "degraded performance" rating.
Update
We believe the issue is resolved. We are scaling infrastructure to clear any delayed background jobs, and monitoring to ensure latency stays within normal range.
Update
We are continuing to monitor for any further issues.
Monitoring
A fix has been implemented and we are monitoring the results.
Identified
The issue has been identified and a fix is being implemented.
Investigating
We are investigating elevated latency in our background jobs system. Some users have also reported receiving Timeout errors while trying to load web pages.
This incident affected: Coveralls.io Web and Coveralls.io API.