What happened

On 19th July 2019 at 15:30 UTC certificate used to serve traffic from ucarecdn.com and www.ucarecdn.com has expired. All HTTPS traffic to these domains effectively stopped.

We were able to quickly (within minutes) redirect www.ucarecdn.com traffic to alternative CDN provider with proper certificate installed.

It took us approximately 105 minutes since start of the incident to fix the issue with ucarecdn.com.

Why that happened

1. Our CDN partner's certificate management system failed to renew the certificate in question and needed an input from our team.
2. It did sent an email 5 days prior to expiration requesting manual input.
3. This email wasn't noticed, because it was sent to a team member that was off the grid on vacation.

What we should do to improve

‌

1. Change notification system settings, so any issues with certificates with our CDN partners are sent to a team, not a person [done]
2. Use 3rd party service to monitor certificate expiration and other settings [done]
3. Use CDN partner's APIs to actively and automatically monitor certificates [in progress]
4. Create an service that could be used to quickly redirect ucarecdn.com traffic to backup CDNs in case of any issues with the main one (not only due to certificates)