Coveralls is in Read Only mode while we work on updating the system. Sorry for the inconvenience.
VIGILANCE! Check this page any time you notice a problem with coveralls
Increased latency for Large Repos (5k+ source files)
Incident
Report for Coveralls
Postmortem
Update: Major Performance Optimization Released for Heavy Repo Builds
Triggered by this latest incident, we've released a major optimization to our code that processes large (aka. “heavy”) repos (>=5K source files).
Tests over the weekend of Fri, Oct 9-Sun, Oct 11 show a near 20x improvement in performance per average heavy repo build at lower, weekend volumes. While we expect the optimization to continue to drastically improve build times for these repos (and reduce incidents of increased latency for them), we won’t know what the aggregate improvement is until we are at full volume this week.
We are monitoring to get a sense of where the performance improvement lands, but are hopeful we’ve crossed a significant milestone in how efficiently these builds are processed given the unique challenges they pose.
Resolved
This incident has been resolved.
Update
The root cause of issues with large repos has been fully addressed. We are continuing to clear some backlogged background jobs. We will continue to monitor and report updates here.
Monitoring
Users in Europe and US east coast with large repos (5k+ repos) experienced slow build completion, starting around 2am PST / 5am EST.
Behavior: PRs opened between 2am-6am PST may have slow or missing Coveralls PR comments or status checks.
We have identified the issue and allocated resources to clear a backlog of background jobs. Jobs should clear and builds should complete in the next ~1hr.
Behavior: PRs opened between 2am-6am PST may have slow or missing Coveralls PR comments or status checks.
We have identified the issue and allocated resources to clear a backlog of background jobs. Jobs should clear and builds should complete in the next ~1hr.
This incident affected: Coveralls.io Web and Coveralls.io API.