A postmortem for this incident and its related incidents has been posted here:

• Postmortem: Reports of “Website under heavy load” errors

Related incidents

1. Aug 13: Intermittent request rejections
2. Aug 14: Service unavailable with HTML error page or 500 errors
3. Aug 18: Reports of "Website under heavy load" errors
4. Aug 19 (Today): Reports of "Website under heavy load" errors

‌

Previous postmortem, prior to final resolution:

We received customer reports early this morning PDT, Thu, Aug 14, indicating that the Coveralls.io service was unavailable for a substantial period overnight.

While our Dequeue Graph appears unresponsive since 18:00 UTC yesterday Aug 13 / 11:00 PDT yesterday, Aug 13, this does not seem to be related to the unavailable period, which appears to have started around 02:00 UTC Aug 14 / 22:00 PDT Aug 13, because we were in service all day and afternoon yesterday, Wed, Aug 13, through EOD PDT.

Our uptime ping did not catch this because Phusion Passenger was returning an HTML error page. Some customers report receiving a 500 error in that message; others report simply receiving HTML as their API response instead of JSON.

While we have not seen production bugs specific to the issue, we suspect it’s because the issue occurred at the Passenger/Nginx layer, which simply prevented new traffic since occurring. As far as previously received coverage report uploads, we see successful background processing of jobs in queue all through the evening into this morning.

We suspect as root cause a regression from a deploy around 5pm yesterday, due mainly to the fact that rolling back to a previous deploy has, so far, resolved the issue.

We will investigate further to obtain a full understanding of the root cause, but in the meantime, we’ll continue monitoring.

Please reach out if you are still experiencing any form of error page in Web requests, or any non-200 API response from Coveralls.io.

Our apologies for the inconvenience.

Note: Recommended Best Practice: continue-on-error: true

As a general best practice, we recommend adding the input option continue-on-error: true (for Coveralls GitHub Action), or continue_on_error: true (for Coveralls Orb for CircleCI); or the --no-fail flag (for Coveralls Universal Coverage Reporter), which will keep Coveralls errors from breaking your CI workflows in the future.