Minor URL API degradation
Incident Report for Uploadcare
Postmortem

-/json/ and -/main_colors/ operations returning 400 status

Date: 2023-11-21

Authors: Alyosha Gusev

Status: Complete, action items in progress

Summary: From 2023-11-21 17:40 to 2023-11-22 18:40 -/json/ and -/main_colors/ operations started to return 400 status

Root Causes: HTTP rewrites misconfiguration

Trigger: Deploy of the new functionality that enables our customers to save more money on traffic, and end users to experience lower load times (improvements to automatic image optimisation)

Resolution: Bugfix deploy

Detection: Our automatic tests detected the issue

Action Items:

Action Item Type Status
Fix rewrites mitigate DONE
Improve visibility of failing tests notification prevent DONE

Lessons Learned

What went well

  • Problem was detected by automatic tests
  • Bugfix was deployed immediately

What went wrong

  • We didn’t found out that tests were failing immediately

Timeline

All times UTC

  • 2023-11-21 17:40: SERVICE DEGRADATION BEGINS. Deploy of the new functionality
  • 2023-11-22 16:18: Team notices failing tests
  • 2023-11-22 16:19: Issue localised, Incident response team is formed
  • 2023-11-22 16:19: Team starts bugfix implementation
  • 2023-11-22 17:40: Fix deployed to production
  • 2023-11-22 18:40: Last cached error response expired
  • 2023-11-22 18:40: SERVICE DEGRADATION ENDS
Posted Nov 27, 2023 - 10:23 UTC

Resolved
2023-11-21 17:40 -/json/ and -/main_colors/ operations returning 400 status
Posted Nov 21, 2023 - 12:00 UTC