-/json/ and -/main_colors/ operations returning 400 status

Date: 2023-11-21

Authors: Alyosha Gusev

Status: Complete, action items in progress

Summary: From 2023-11-21 17:40 to 2023-11-22 18:40 -/json/ and -/main_colors/ operations started to return 400 status

Root Causes: HTTP rewrites misconfiguration

Trigger: Deploy of the new functionality that enables our customers to save more money on traffic, and end users to experience lower load times (improvements to automatic image optimisation)

Resolution: Bugfix deploy

Detection: Our automatic tests detected the issue

Action Items:

‌

Lessons Learned

What went well

• Problem was detected by automatic tests
• Bugfix was deployed immediately

What went wrong

• We didn’t found out that tests were failing immediately

Timeline

All times UTC

• 2023-11-21 17:40: SERVICE DEGRADATION BEGINS. Deploy of the new functionality
• 2023-11-22 16:18: Team notices failing tests
• 2023-11-22 16:19: Issue localised, Incident response team is formed
• 2023-11-22 16:19: Team starts bugfix implementation
• 2023-11-22 17:40: Fix deployed to production
• 2023-11-22 18:40: Last cached error response expired
• 2023-11-22 18:40: SERVICE DEGRADATION ENDS