Expired certificate on ucarecdn.com
Incident Report for Uploadcare
Postmortem

What happened

On 19th July 2019 at 15:30 UTC certificate used to serve traffic from ucarecdn.com and www.ucarecdn.com has expired. All HTTPS traffic to these domains effectively stopped.

We were able to quickly (within minutes) redirect www.ucarecdn.com traffic to alternative CDN provider with proper certificate installed.

It took us approximately 105 minutes since start of the incident to fix the issue with ucarecdn.com.

Why that happened

  1. Our CDN partner's certificate management system failed to renew the certificate in question and needed an input from our team.
  2. It did sent an email 5 days prior to expiration requesting manual input.
  3. This email wasn't noticed, because it was sent to a team member that was off the grid on vacation.

What we should do to improve

  1. Change notification system settings, so any issues with certificates with our CDN partners are sent to a team, not a person [done]
  2. Use 3rd party service to monitor certificate expiration and other settings [done]
  3. Use CDN partner's APIs to actively and automatically monitor certificates [in progress]
  4. Create an service that could be used to quickly redirect ucarecdn.com traffic to backup CDNs in case of any issues with the main one (not only due to certificates)
Posted 21 days ago. Aug 01, 2019 - 14:14 UTC

Resolved
This incident has been resolved.
Posted about 1 month ago. Jul 19, 2019 - 19:35 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted about 1 month ago. Jul 19, 2019 - 18:03 UTC
Update
We are continuing to work on a fix for this issue.
Posted about 1 month ago. Jul 19, 2019 - 17:50 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted about 1 month ago. Jul 19, 2019 - 15:59 UTC
This incident affected: CDN.