Coveralls is in Read Only mode while we work on updating the system. Sorry for the inconvenience.
VIGILANCE! Check this page any time you notice a problem with coveralls
Elevated Latency
Incident
Report for Coveralls
Resolved
We have attempted to improve latency this morning for new jobs from users in EU and US time zones, which has meant offloading older jobs to specialized queues with scaled up resources, which, at this point, have already drained.
System-wide, latency continues improving for all users and should reach normal in 15-30 minutes for most users.
If you are still experiencing elevated latency after that, or have any jobs from CI builds run in the last 24 hrs that have not completed, please reach out to us at support@coveralls.io and we'll investigate to determine whether your builds were caught up in this incident, or if they have a different cause.
NOTE: Missing data points in the "DEQUEUE" Graph on our main Status Page does not indicate that processing stopped during that period, just that the server that reports our stats lost comms during that period. As stated elsewhere, to avoid this confusion in the future it's one of our goals to spread stats collection across all servers.
System-wide, latency continues improving for all users and should reach normal in 15-30 minutes for most users.
If you are still experiencing elevated latency after that, or have any jobs from CI builds run in the last 24 hrs that have not completed, please reach out to us at support@coveralls.io and we'll investigate to determine whether your builds were caught up in this incident, or if they have a different cause.
NOTE: Missing data points in the "DEQUEUE" Graph on our main Status Page does not indicate that processing stopped during that period, just that the server that reports our stats lost comms during that period. As stated elsewhere, to avoid this confusion in the future it's one of our goals to spread stats collection across all servers.
Update
We have confirmed elevated latency for global users in ASIA, AU and EU starting on Mon, Nov 10 around 09:15a UTC (01:15a PDT).
We have resolved the cause but will be performing RCA in the next 24-48 hrs to understand the triggering conditions and avoid further incidents.
In the meantime, we will be monitoring latency and will likely manage workloads to improve latency for new jobs (from currently active time zones), routing jobs from inactive time zones to new resources to clear the backlog.
We will upgrade performance from Degraded to (Fully) Operational when complete.
We have resolved the cause but will be performing RCA in the next 24-48 hrs to understand the triggering conditions and avoid further incidents.
In the meantime, we will be monitoring latency and will likely manage workloads to improve latency for new jobs (from currently active time zones), routing jobs from inactive time zones to new resources to clear the backlog.
We will upgrade performance from Degraded to (Fully) Operational when complete.
Identified
The issue has been identified and a fix is being implemented.
Update
We are continuing to investigate this issue.
Investigating
We are currently investigating this issue.
This incident affected: Coveralls.io Web and Coveralls.io API.